2011 Unidata NetCDF Workshop > Example of the Importance of the Chunk Cache
25.0 The Problem
Chunks good for writing data can be bad for reading it. This example
illustrates how bad.
A user has a large 3D variable, about 17.5 GB, stored with time
varying most slowly:
netcdf cvx3 {
dimensions:
latitude = 1617 ;
longitude = 1596 ;
time = 1698 ;
variables:
...
float var(time, latitude, longitude) ;
...
}
- Accessing all data for a specific time (10.3 MB) is fast (0.25 sec)
- Accessing all data for a specific location (6792 bytes) is very slow: (10.3
sec)
- Users want to access data both ways
- Data currently stored as netCDF classic file contiguously.
- Possible solutions:
- Keep multiple copies of data for different forms of access.
Takes too much space, can lead to update anomalies.
2011 Unidata NetCDF Workshop > Example of the Importance of the Chunk Cache