2010 Unidata NetCDF Workshop > Chunking and Deflating Data with NetCDF-4
18.1 Choosing Chunksizes
How do you pick chunksizes?
- Choosing good chunksizes depends on the access patterns of your
data. Are you trying to optimize writing, reading, or both? What are
the access patterns at I/O bottlenecks?
- Choose chunksizes so that the subsets of data you are accessing
fit into a chunk. That is, the chunks should be as large, or larger
than, the subsets you are reading/writing.
- The cache size must also be adjusted for good performance. The
cache must be large enough to hold at least one chunk.
- Setting a larger cache (for example, big enough to hold tens or
hundreds of chunks) will pay off only if the access patterns support
it.
- On today's high-performance systems, large amounts of RAM are
available (both to the user and as internal hardware caching.) This
suggests that chunks and caches should be large, and programs should
take large sips of data.
2010 Unidata NetCDF Workshop > Chunking and Deflating Data with NetCDF-4