[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[THREDDS #KXL-349288]: Problem subsetting netcdf files

Hi Rostislav,

Sorry for the delayed response. This is indeed a problem in the hdf-java code. 
Specifically, it fails when a variable has a very large chunk size. Thanks for 
the report!

At this time, the only way to work around the problem is to rechunk the file to 
use smaller chunks, which is something you might want to do anyway. Currently, 
you have:
                cnc_POLLEN_BIRCH_m22:_ChunkSizes = 97, 8, 315, 467 ;
Or, in other words, the chunk is as large as the entire variable. Such a large 
chunk size blunts the potential performance gains that chunking was meant to 
provide.
Furthermore, HDF clients must read the entire chunk into memory before they can 
access even a small piece of it. That's actually what's causing netCDF-Java to 
fail when chunks are very large.
By rechunking your dataset to use smaller chunks, I was able to read it 
successfully:
    nccopy -k nc4 -d 1 -c time/13,y/40,x/59,height/1 
SILAM-POLLEN-europe20160402.nc4 SILAM-POLLEN-europe20160402_rechunked.nc4

Again, there is still a definite problem in netCDF-Java that needs to be 
addressed, but rechunking may be something you want to consider anyway. [1] and 
[2] are
good primers on the matter.

Cheers,
Christian

[1] 
http://www.unidata.ucar.edu/blogs/developer/entry/chunking_data_why_it_matters
[2] 
http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_choosing_shapes

Ticket Details
===================
Ticket ID: KXL-349288
Department: Support THREDDS
Priority: High
Status: Closed