Re: requirements for handling chunking...

• To: netcdf-hdf@xxxxxxxxxxxxxxxx
• Subject: Re: requirements for handling chunking...
• From: Quincey Koziol <koziol@xxxxxxxxxxxxx>
• Date: Wed, 17 Dec 2003 13:22:27 -0600 (CST)

Hi Ed,
    Seems reasonable.  (We _really_ need to increase the default chunk cache
size... :-)

        Quincey

> Howdy all!
> 
> Based on recent emails and discussions, here's an attempt at
> requirements relating to the setting of chunksizes. This is from the
> requirements at
> http://my.unidata.ucar.edu/content/software/netcdf/netcdf-4/reqs.html. 
> 
>     * Chunking is required in any dataset with one or more unlimited
>       dimension in HDF5. NetCDF-4 supports setting chunk parameters at
>       variable creation with the following new function:
> 
> int nc_def_var_x(int ncid, const char *name, nc_type xtype, int ndims, 
>         const int *dimidsp, int *varidp, int chunkalg, int *chunksizes);
> 
>       Where chunksize is a pointer to an array of size ndims, with the
>       chunksize in each dimension. If chunksizes is NULL, the user can
>       select a chunking algorithm by setting chunkalg to NC_CHUNK_SEQ
>       (to optimize for sequential access), NC_CHUNK_SUB (for chunk
>       sizes set to favor equally subsetting in any dimension.
> 
>       When the (netcdf-3) function nc_def_var is used, a sequential
>       chunking algorithm will be used. (Just as if the var had been
>       created with NC_CHUNK_SEQ).
> 
>       The sequential chunking algorithm sets a chunksize of 1 all
>       unlimited dimensions, and all other chunksizes to the size of
>       that dimension, unless the resulting chunksize is greater than
>       250 KB, in which case subsequent dimensions will be set to 1
>       until the chunksize is less than 250 KB (one quarter of the
>       default chunk cache size).
> 
>       The subsetting chunking algorithm sets the chunksize in each
>       dimension to the nth root of (desired chunksize/product of n
>       dimsizes).
> 
>