On 05/19/2014 09:52 AM, Timothy Stitt wrote:
Hi all,
I’ve been trying to convert a large (40GB) ASCII text file (composed of
multiple records of 4 line ASCII strings about 90 characters long) into
NetCDF format. My plan was to rewrite the original serial code to use
parallel NetCDF to have many MPI processes concurrently read records and
process them in parallel.
I was able to write some code to convert the ASCII records into
[unlimited][4][90] NetCDF NC_CHAR arrays, which I was able to read
concurrently via parallel NetCDF routines. My question is related to the
size of the converted NetCDF file.
I notice that the converted NetCDF file is always double the size of the
ASCII file whereas I was hoping for it be to much reduced. I was
therefore wondering if this is expected or is more due to my bad
representation in NetCDF of the ASCII records? I am using
nc_put_vara_text() to write my records. Maybe I need to introduce
compression that I’m not doing already?
Are you using the classic file format or the NetCDF-4 file format?
Can you provide an ncdump -h of the new file?
==rob
Thanks in advance for any advice you can provide.
Regards,
Tim.
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
