2009 Unidata NetCDF Workshop for Developers and Data Providers > Formats and Performance
13.6 Using Less Time for Data Access
If netCDF access time is a bottleneck, these
techniques may help.
- Change dimension order for variables so that you
read data in the same order it was written, with most frequently varying dimension
last. In particular, avoid writing or reading one value per record
(per index of the unlimited dimension), as this will typically access
a new disk block for every data value.
- Each seek to a new location in a file causes buffer
flushes, so reading randomly negates most of the advantages of buffering.
- If data must be written in a different order than it is read,
consider transposing the data to the most efficient read order once
if it will be read many times.
- If you will always write all data values for each variable,
consider turning off "fill mode" that pre-fills data on the disk with
fill values. This can save 50% of write time, but it eliminates the
possibility of later
detecting unwritten values.
- For writing a netCDF file from multiple processors when a
parallel file system is available, use parallel
netCDF
NetCDF-4 note:
Supports per-variable chunking, compression, parallel I/O,
and control of endianness, which can all improve I/O performance.
2009 Unidata NetCDF Workshop for Developers and Data Providers > Formats and Performance