Simon, thanks for the small test file. That helped.
I reproduced slow reads with netcdf-C 4.4.1 and HDF5 1.8.x on both Mac OS
and Linux. The important thing is that I used ncdump as my test reader,
like you did. When I substituted a simple fortran program to read the full
1.8 Mb data array, read time dropped to under 0.4 seconds. Program is
attached.
So I think you have a read cacheing failure, due to interaction between the
ncdump read pattern, and your chunking scheme. I think the chunking scheme
is perfectly reasonable. However, ncdump may be reading values across full
rows, meaning that it potentially jumps between several file chunks for
each data row. This should work fine, *unless read chunk cacheing is not
working right* for one reason or another. Inadequate chunk cache size
might be the cause.
netcdf qualityFlags {
dimensions:
columns = 4865 ;
rows = 3682 ;
variables:
uint quality_flags(rows, columns) ;
quality_flags:_ChunkSizes = 891, 1177 ;
quality_flags:_DeflateLevel = 2 ;
:_Format = "netCDF-4" ;
A sampling tool found that ncdump was spending more than 96% of its time
inside an HDF5 chunk reader with decompression. Every time an HDF5 chunk
is physically read from disk, the *entire* chunk must be decompressed, even
to access a single value. You see why chunk cacheing is important.
IMO ncdump should not be sluggish for a small test file, and your complaint
about change in behavior is valid. But look in ncdump and chunk cacheing
for the possible cause, not in the underlying libraries. I need to leave
it to those with better understanding of chunk cacheing for the final
explanation and bug fix.
--Dave
On Wed, Dec 14, 2016 at 6:42 PM, Simon (Vsevolod) Ilyushchenko <
simonf@xxxxxxxxxx> wrote:
> The issue is present in 4.3.1-rc5, but not in 4.3.1-rc4. The only relevant
> release note I see is "When opening a netCDF-4 file, streamline the
> iteration over objects in the underlying HDF5 file."
>
> I found a 2M file that shows the same problem - see
> https://drive.google.com/file/d/0B_ijCK6xm1R1MW1nU2dsWnN
> XbE0/view?usp=sharing
>
> 3 second ncdump with rc4 vs almost 2 minute ncdump with rc5.
>
> Simon
>
Attachment:
simple_xy_nc4_rd.qf-demo.f90
Description: Binary data
netcdfgroup archives: