2012 Unidata NetCDF Workshop > Chunking and Deflating Data with NetCDF-4
19.3 The Cache in NetCDF-4
The chunk cache is used when reading or writing data. Set the cache size
correctly for each variable to improve performance.
- The cache size for data transfers in HDF5 is 1 MB, by
default. This is small for many real applications, so netCDF-4 uses
a default cache size of 32 MB.
- The default cache size for netCDF-4 can be changed at configure
time.
- NetCDF allows user to specify default variable chunk cache size
for the file at runtime with nc_set_chunk_cache().
- NetCDF allows user to specify variable-specific chunk cache size
for the file at runtime with nc_set_var_chunk_cache().
- Set the cache size so that it can hold multiple chunks of data. If
your chunk size is less than your cache size, performance will
suffer.
- This does not get set permanently for the variable, it only lasts
as long as the file is open.
- Due to the interaction of various layers of hardward and software
buffers in your system, it is not always easy to know what is going
on, and how to best set the cache.
- The nccopy utility can rechunk data to specified chunk sizes,
and you can set the chunk cache while copying to see the effect on performance.
2012 Unidata NetCDF Workshop > Chunking and Deflating Data with NetCDF-4