Hi all,
I’ve been trying to convert a large (40GB) ASCII text file (composed of
multiple records of 4 line ASCII strings about 90 characters long) into NetCDF
format. My plan was to rewrite the original serial code to use parallel NetCDF
to have many MPI processes concurrently read records and process them in
parallel.
I was able to write some code to convert the ASCII records into
[unlimited][4][90] NetCDF NC_CHAR arrays, which I was able to read concurrently
via parallel NetCDF routines. My question is related to the size of the
converted NetCDF file.
I notice that the converted NetCDF file is always double the size of the ASCII
file whereas I was hoping for it be to much reduced. I was therefore wondering
if this is expected or is more due to my bad representation in NetCDF of the
ASCII records? I am using nc_put_vara_text() to write my records. Maybe I need
to introduce compression that I’m not doing already?
Thanks in advance for any advice you can provide.
Regards,
Tim.
