Thanks to both Kent and Chris.
I'll play with chunk size a little bit.
I asked this question is that I want to know if this is the overhead from
NetCDF4 (HDF5),
or it is a bug in our own program. Now, your response, which we can explain to
our users in case they have the similar issue.
Again, Thanks!
Wei Huang
On Apr 15, 2014, at 8:43 AM, Chris Barker - NOAA Federal
<chris.barker@xxxxxxxx> wrote:
> On Apr 14, 2014, at 6:31 PM, Kent Yang <myang6@xxxxxxxxxxxx> wrote:
>> I don’t think your programs cause this. For unlimited dimensions, the HDF5
>> chunking storage is used. There are space overheads for chunks, especially
>> when the chunk size is small.
>
> Exactly -- very small chunk sizes can make for huge files. But this seems
> like a lot of overhead to me.
>
> Did you use the defaults for chunk sizes? I would have thought they'd do
> pretty well for this case, but it's certainly worth trying setting them by
> hand to see what you get.
>
> Some googling will find you some good advice, but this is a good start:
>
> http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_why_it_matters
>
> Chris
>
>> One way to mitigate this issue is to increase the chunk size and then to
>> use compression.
>>
>> Kent
>>
>> From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx
>> [mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Wei Huang
>> Sent: Monday, April 14, 2014 4:38 PM
>> To: NetCDF Mailing List
>> Subject: [netcdfgroup] NetCDF-4 filesize question.
>>
>> Hello group,
>>
>> I am doing some NetCDF-4 test, and find when I use fixed, and unlimited
>> dimensions cause file size
>> changed dramatically.
>>
>> Below is a list of files (with dimension 5 x 10 x 73 x 144, with few group
>> names):
>>
>> 1. has one unlimited dimension (the most left one)
>> 2. has two unlimited dimensions (the left two)
>> 3. has fixed dimensions.
>>
>> -rw-r--r-- 1 huangwei CIT\Domain Users 6312828 Apr 14 15:25
>> NCLcreatedNC4.nc.1unlimited
>> -rw-r--r-- 1 huangwei CIT\Domain Users 10508612 Apr 14 15:24
>> NCLcreatedNC4.nc.2unlimited
>> -rw-r--r-- 1 huangwei CIT\Domain Users 2112758 Apr 14 15:26
>> NCLcreatedNC4.nc.fixed
>>
>>
>> My question is: does our program (has issues which) cause the file size
>> difference,
>> or, is it NetCDF4 which needs the extra space (for unlimited dimension)?
>>
>> Thanks,
>>
>>
>> Wei Huang
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
