[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #AWT-862217]: nccopy chunking argument

Hi Mark,

> This issue is closely related to Support Ticket  #CUV-251255 and Jira
> issues NCF-85 and NCF-73.
> 
> I am trying to rechunk a large (43GB) NetCDF 4 file into a form where
> the chunks are optimsed for the read patterns that I am using. The file
> structure is
> 
> dimensions:
> lat = 720 ;
> lon = 1440 ;
> date = 5186 ;
> variables:
> int date(date) ;
> date:_Storage = "chunked" ;
> date:_ChunkSizes = 1 ;
> date:_Endianness = "little" ;
> float lat(lat) ;
> lat:long_name = "latitude" ;
> lat:units = "degrees_north" ;
> lat:axis = "Y" ;
> lat:_Storage = "contiguous" ;
> float lon(lon) ;
> lon:long_name = "longitude" ;
> lon:units = "degrees_east" ;
> lon:axis = "X" ;
> lon:_Storage = "contiguous" ;
> float CHL1_mean(date, lat, lon) ;
> CHL1_mean:long_name = "Chlorophyll-a concentration. Case 1 water - Mean of 
> the binned pixels" ;
> CHL1_mean:_FillValue = -999.f ;
> CHL1_mean:units = "mg/m3" ;
> CHL1_mean:pct_characterised_error = 53.04f ;
> CHL1_mean:_Storage = "chunked" ;
> CHL1_mean:_ChunkSizes = 1, 720, 1440 ;
> 
> I would like to rechunk the variables so that instead of being stored as
> date/1,lat/720/lon1440 that they are stored as date/1,lat/30,lon/30. I
> am using a version of nccopy compiled from the nightly builds with the
> following command:
> 
> nccopy -m 1G -h 2G -e 10000 -c lon/30,lat/30 step1.nc step2.nc
> 
> This command, however hangs - it writes about 20GB of the 40GB to step2.nc
> within the first 5 minutes, but after that the file size remains static
> for 13 hours (even though nccopy is still active and occupying RAM). At
> this point I loose patience.
> 
> However, I believe I have found the cause of the error. I was able to
> read the header of the (broken) 20GB file produced, and it showed that
> the chunks that were being created were in fact date/5186,lon/30,lat/30
> rather than the date/1,lon/30,lat/30 that I was aiming for. I would guess
> that the (unintended) change in the date chunking is coming from nccopy.
> 
> So, the questions are:
> 
> 1. why does nccopy hang when trying to create chunks that are
> 5186 x 30 x 30 ?

I'm not sure, and would need to duplicate that behavior here to see what's 
happening.
Is it possible to supply me with a test file or a small program that generates 
a test
file that demonstrates the problem?

> 2. is it desired behaviour that nccopy changes the chunking of dimensions
> that are not-specified in the re-chunking command?

No, I think that's a bug.  Does it work when you explicitly specify 
"-c date/1,lon/30,lat/30" ?

--Russ

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: AWT-862217
Department: Support netCDF
Priority: Normal
Status: Closed