Simon,
The underlying format of netcdf-4 is supported by the HDF5 library and
other support libraries. It is quite possible that the read timing
differences are coming from changes in the HDF5 layer, not the netcdf-C
library code itself. In your binary search, please keep track of which
HDF5 library versions are in use.
You might also pay attention to the libz version. I will be surprised if
it turns out you found a problem in libz.
--Dave
On Tue, Dec 13, 2016 at 5:55 PM, Simon (Vsevolod) Ilyushchenko <
simonf@xxxxxxxxxx> wrote:
> Sure. The rest is the same. I'll do binary search next.
>
> First:
>
> netcdf a {
> dimensions:
> depth = 2 ;
> lat = 2001 ;
> lon = 4500 ;
> time = UNLIMITED ; // (1 currently)
> variables:
>
<snip>
short water_u(time, depth, lat, lon) ;
> water_u:long_name = "Eastward Water Velocity" ;
> water_u:standard_name = "eastward_sea_water_velocity" ;
> water_u:units = "m/s" ;
> water_u:_FillValue = -30000s ;
> water_u:missing_value = -30000s ;
> water_u:scale_factor = 0.001f ;
> water_u:add_offset = 0.f ;
> water_u:NAVO_code = 17 ;
> water_u:_Storage = "chunked" ;
> water_u:_ChunkSizes = 1, 1, 976, 2196 ;
> water_u:_DeflateLevel = 9 ;
> water_u:_Endianness = "little" ;
>
<snip>
:history = "Fri Nov 20 10:01:58 2015: ncks -4 --deflate 9
> -v water_u,water_v -d depth,1,2,1 hycom_glb_911_2015110200_t000_uv3z.nc
> hycom_glb_911_2015110200_t000_uv3z-test.nc\narchv2ncdf3z" ;
> :NCO = "4.4.2" ;
> :_Format = "netCDF-4" ;
>
<snip>
On Tue, Dec 13, 2016 at 10:04 AM, Ed Hartnett <edwardjameshartnett@xxxxxxxxx
> > wrote:
>
>> Can you do an ncdump with the -s option, so we can see the chunksizes?
>>
>> On Tue, Dec 13, 2016 at 10:54 AM, Chris Barker <chris.barker@xxxxxxxx>
>> wrote:
>>
>>> if I understand the question, this is read times -- is this in fact with
>>> exactly the same files? In which case the chunking is already set.
>>>
>>> less performant decompression??
>>>
>>> sounds like the binary search is needed.
>>>
>>> Is EVERYTHING else the same? disk system, OS, etc?
>>>
>>> -CHB
>>>
>>>
>>> On Tue, Dec 13, 2016 at 9:21 AM, Charlie Zender <zender@xxxxxxx> wrote:
>>>
>>>> Hello Simon,
>>>>
>>>> Since both files are netCDF4 compressed that
>>>> means they use chunking. My wild guess is that
>>>> different chunking defaults cause the observed
>>>> change in dumping time. You can see the
>>>> chunk sizes employed with ncdump -s or ncks --hdn,
>>>> and you can play with the chunk sizes/policy
>>>> with either.
>>>>
>>>> Charlie
>>>>
>>>
