Roy,
> Thanks. But the reason I don't want to use nccopy, but want to use =
> ncgen from the start, is that the netcdf3 version of the file will be =
> about 4Gb. What I did is create a small version of the file, used NCO =
> to convert to netcdf4 using settings I commonly used, and then using =
> ncdump -s was able to look at the special attributes created, added them =
> to my cdl, used ncgen and it worked quite nicely, especially since I am =
> creating an empty array that will be filled as the output of my model =
> runs, so the default netcdf4 files is miniscule.
>
> Mainly I don't know enough about the different settings and what they do =
> to the file to know what options to put. I know these settings are =
> more from HDF5 then you guys, but is there an idiot's guide to this for =
> those of us who know little about it but are finding it to be of great =
> use. I know Rich Signell has done a little bit of timings on the =
> tradeoffs between size and read time for the various settings, do you =
> know of others who have? We are pretty much interesting in reads, and =
> using a lot of separate files aggregated in THREDDS, where the user will =
> want a time series or time series from a region. Using NCO notation =
> if we go "ncks -4 -L 1 -O " then we get really good compression but =
> relatively slow reads, and if we go "ncks -O -4 -L 1 --cnk_dmn lat,16 =
> --cnk_dmn lon,16" the reads are right up there but the compression is =
> so-so. Are there any good guidelines for this, or just trial and error.
We've been learning more about tradeoffs among size, chunk shapes, chunk
caches, deflation level, and data characteristics, but haven't distilled
the experience yet into guidance for how to make the best use of
chunking and compression. If you want to follow progress or comment on
needs for such guidance, see the following two open issues:
Documentation: improve guidance for use of chunking and compression
https://www.unidata.ucar.edu/jira/browse/NCF-73
Make use of "chunking policies" for a high-level specification on netCDF
chunking
https://www.unidata.ucar.edu/jira/browse/NCF-153
The best we can offer currently are the following online sections of last
summer's netCDF training workshops:
Chunking and Deflating Data with NetCDF-4
http://www.unidata.ucar.edu/netcdf/workshops/2011/nc4chunking/
Example of the Importance of the Chunk Cache
http://www.unidata.ucar.edu/netcdf/workshops/2011/chunk_cache/
--Russ
> This is becoming a very important issue for us (assuming we are still =
> around next year) because we are starting to serve out a lot of 1km =
> data. Storing in netcdf4 saves us a huge amount of space, but we don't =
> want a huge speed hit on the reads.
>
> Thanks.
>
> -Roy
