> On Fri, Feb 22, 2013 at 11:32 AM, Russ Rew <russ@xxxxxxxxxxxxxxxx> wrote:
>
> > It means that if you have a variable with an unlimited dimension, such
> > as
> >
> > float var(time, lon, lat)
> >
> > where time is unlimited, then the default chunks will be of shape
> >
> > 1 x clon x clat
> >
> > values (not bytes), for integers clon, clat computed to be smaller
> > than but proportioanl to the sizes of the lon and lat dimensions
>
>
> Let me see if I understand. Here's the setup of my file:
>
> ----------------
> dimensions:
> step = UNLIMITED ;
> cell = 100 ;
> variables:
> float temperature(step, cell) ;
> float flux(step, cell) ;
> float energy(step, cell) ;
> byte steadyState(step) ;
> -----------------
>
> So -- by default -- temperature, flux, and energy will have a chunk size of
> -- "smaller than but proportioanl to", meaning smaller than? -- 100 floats
> (so 400 bytes)?
> ... Likewise, by default, steadyState will have a chunk size of
> 1 byte? (Since it can't be smaller.)
>
> If that's true, that does sound terribly inefficient.
Yes, it's true. You can verify this by using ncgen, nccopy, and ncdump
to see the default chunk shapes, assuming your file is described in
foo.cdl:
$ ncgen -b foo.cdl # make foo.nc, a netCDF classic file
$ nccopy -d1 foo.nc foo4.nc # make foo4.nc, a compressed (hence chunked) file
$ ncdump -s -h foo4.nc > foo4.cdl # show "special" attributes
$ grep _ChunkSizes foo4.cdl # show chunk shapes, in units of elements
temperature:_ChunkSizes = 1, 100 ;
flux:_ChunkSizes = 1, 100 ;
energy:_ChunkSizes = 1, 100 ;
steadyState:_ChunkSizes = 1 ;
> And I override these defaults by providing a chunksizes array when calling
> NF90_DEF_VAR?
Not quite. You have to call nc_def_var_chunking(..., chunkshape) for
each variable *before* calling nc_enddef(). Or you could use nccopy to
specify chunk shape components for each dimension in a copy of the file,
as in:
nccopy -c step/20,cell/100 foo.nc foo4.nc
to make chunk sizes be 10 x 100 elements, or 4KB for all variables of
type float and 1KB for byte variables.
As I noted, the current chunk shape defaults need to be improved ...
--Russ
> --e89a8fb1ef2e16ed2104d655f7b5
> Content-Type: text/html; charset=UTF-8
> Content-Transfer-Encoding: quoted-printable
>
> On Fri, Feb 22, 2013 at 11:32 AM, Russ Rew <span dir=3D"ltr"><<a href=3D=
> "mailto:russ@xxxxxxxxxxxxxxxx"; target=3D"_blank">russ@xxxxxxxxxxxxxxxx</a>&=
> gt;</span> wrote:<br><div class=3D"gmail_quote"><blockquote class=3D"gmail_=
> quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1=
> ex">
> It means that if you have a variable with an unlimited dimension, such<br>
> as<br>
> <br>
> =C2=A0 =C2=A0float var(time, lon, lat)<br>
> <br>
> where time is unlimited, then the default chunks will be of shape<br>
> <br>
> =C2=A0 =C2=A01 x clon x clat<br>
> <br>
> values (not bytes), for integers clon, clat computed to be smaller<br>
> than but proportioanl to the sizes of the lon and lat dimensions</blockquot=
> e><div><br></div><div>Let me see if I understand. Here's the setup of m=
> y file:</div><div><br></div><div>----------------</div><div>dimensions:</di=
> v>
> <div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 step =3D UNLIMITED ;</div><div>=C2=A0 =C2=
> =A0 =C2=A0 =C2=A0 cell =3D 100 ;</div><div>variables:</div><div>=C2=A0 =C2=
> =A0 =C2=A0 =C2=A0 float temperature(step, cell) ;</div><div>=C2=A0 =C2=A0 =
> =C2=A0 =C2=A0 float flux(step, cell) ;</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=
> =A0 float energy(step, cell) ;</div>
> <div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 byte steadyState(step) ;</div><div>-------=
> ----------</div><div><br></div><div>So -- by default -- temperature, flux, =
> and energy will have a chunk size of -- "smaller than but proportioanl=
> to", meaning smaller than? -- 100 floats (so 400 bytes)? Likewise, by=
> default, steadyState will have a chunk size of 1 byte? (Since it can't=
> be smaller.)</div>
> <div><br></div><div>If that's true, that does sound=C2=A0terribly=C2=A0=
> inefficient.</div><div><br></div><div>And I override these defaults=C2=A0by=
> providing a chunksizes array when calling NF90_DEF_VAR?</div><div>-Leon</d=
> iv></div>
>
> --e89a8fb1ef2e16ed2104d655f7b5--
