On Thu, Jan 9, 2014 at 4:22 PM, Charlie Zender <zender@xxxxxxx> wrote:
> We use record and unlimited interchangeably.
> A record dimenison is an unlimited dimension.
got it -- and that explains the lack of references to unlimited...
> yes, the rd1 map is not good for access speeds on 1D vars.
> use the --cnk_dmn=time,sz option to override this. or use lfp map...
got it.
> > Not so clear from the amount of time I've spent reading that, but what
> > would be the default chunking for a 1-d unlimited variable? or a 2-d,
> > with one dimension very small (Nx3, for instance)?
>
> for lfp map these are [N] and [N/3,3] where N is total chunksize, as
> you can see with these commands:
> ncks -O -4 --cnk_plc=all --cnk_map=lfp ~/nco/data/in.nc ~/foo.nc
> ncks -m --cdl --hdn ~/foo.nc
>
> We may change the NCO default map from rd1 to lfp or something
> similar in 4.4.1.
>
I think that's a good idea --I'm sure most users don't "get" chunking -- I
sure didn't until recently. So good default are key.
Note that in our case, the default (chunk size 1) writing a bunch of big
1-d arrays nto only was slow, but it made for huge files, and ended up
crashing out our model. Changing that to 1024 chunk made a huge difference.
So I think there should be a default minimum chunk size for any record
variable -- maybe about 1024, but not sure about that.
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@xxxxxxxx
