On Apr 14, 2014, at 6:31 PM, Kent Yang <myang6@xxxxxxxxxxxx> wrote:
I don’t think your programs cause this. For unlimited dimensions, the
HDF5 chunking storage is used. There are space overheads for chunks,
especially when the chunk size is small.
Exactly -- very small chunk sizes can make for huge files. But this seems
like a lot of overhead to me.
Did you use the defaults for chunk sizes? I would have thought they'd do
pretty well for this case, but it's certainly worth trying setting them by
hand to see what you get.
Some googling will find you some good advice, but this is a good start:
http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_why_it_matters
Chris
One way to mitigate this issue is to increase the chunk size and then to
use compression.
Kent
*From:* netcdfgroup-bounces@xxxxxxxxxxxxxxxx [mailto:
netcdfgroup-bounces@xxxxxxxxxxxxxxxx] *On Behalf Of *Wei Huang
*Sent:* Monday, April 14, 2014 4:38 PM
*To:* NetCDF Mailing List
*Subject:* [netcdfgroup] NetCDF-4 filesize question.
Hello group,
I am doing some NetCDF-4 test, and find when I use fixed, and unlimited
dimensions cause file size
changed dramatically.
Below is a list of files (with dimension 5 x 10 x 73 x 144, with few group
names):
1. has one unlimited dimension (the most left one)
2. has two unlimited dimensions (the left two)
3. has fixed dimensions.
-rw-r--r-- 1 huangwei CIT\Domain Users 6312828 Apr 14 15:25
NCLcreatedNC4.nc.1unlimited
-rw-r--r-- 1 huangwei CIT\Domain Users 10508612 Apr 14 15:24
NCLcreatedNC4.nc.2unlimited
-rw-r--r-- 1 huangwei CIT\Domain Users 2112758 Apr 14 15:26
NCLcreatedNC4.nc.fixed
My question is: does our program (has issues which) cause the file size
difference,
or, is it NetCDF4 which needs the extra space (for unlimited dimension)?
Thanks,
Wei Huang
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
