Dear Christopher,
HDF5 does provide a chunk cache, but I presume in this case it is
simply too small to fit.
You can imagine if you increase the number of columns further at some
point the 2D chunks will exceed (any) sized cache.
This is however, suboptimal in that case as ncdump outputs row after row.
One way to fix this would be to adjust ncdump to increase the chunk
cache to a reasonable amount of main memory, potentially offering
users a way to influence this value from command line.
See https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetChunkCache
There is quite a good doku here about general aspects of Chunking:
https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/
Julian
2016-12-15 20:17 GMT+01:00 Chris Barker <chris.barker@xxxxxxxx>:
> On Wed, Dec 14, 2016 at 9:13 PM, Dave Allured - NOAA Affiliate
> <dave.allured@xxxxxxxx> wrote:
>
>>
>> So I think you have a read cacheing failure, due to interaction between
>> the ncdump read pattern, and your chunking scheme.
>
> ...
>>
>> A sampling tool found that ncdump was spending more than 96% of its time
>> inside an HDF5 chunk reader with decompression. Every time an HDF5 chunk is
>> physically read from disk, the *entire* chunk must be decompressed, even to
>> access a single value. You see why chunk cacheing is important.
>
>
> Does HDF5 not do any chunk caching itself? or for that matter, netcdf4? Is
> is really up to the application level to manage the caching? that seems like
> handling it at the wrong level to me.
>
> -CHB
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
>
> Chris.Barker@xxxxxxxx
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
--
http://wr.informatik.uni-hamburg.de/people/julian_kunkel
