Two notes:
1. Adding this feature to ncdump also requires adding
it to the netcdf-c library API. But providing some means
for client programs to pass thru parameter settings to the hdf5 lib
seems like a good idea.
2. Have you tried using nccopy to see if that is slow as well?
=Dennis Heimbigner
Unidata
On 12/15/2016 1:03 PM, Chris Barker wrote:
On Thu, Dec 15, 2016 at 11:30 AM, Julian Kunkel
<juliankunkel@xxxxxxxxxxxxxx <mailto:juliankunkel@xxxxxxxxxxxxxx>> wrote:
HDF5 does provide a chunk cache, but I presume in this case it is
simply too small to fit.
You can imagine if you increase the number of columns further at some
point the 2D chunks will exceed (any) sized cache.
This is however, suboptimal in that case as ncdump outputs row after
row.
so to see if I have this right -- libhdf5 manages chunk caching for you.
But it's up to the client application to set an appropriate size for the
chunk cache.
I suspect that the defaults in HDF are pretty small, to be conservative.
One way to fix this would be to adjust ncdump to increase the chunk
cache to a reasonable amount of main memory,
I think this is a good idea -- ncdump is usually (always?) used in a
one-off -- t will red the file and then the program closes -- so no real
worry it hanging on to a lot of memory.
potentially offering
users a way to influence this value from command line.
sure -- though that's kind of a low-level feature -- no sure if anyone
using ncdump is liley to use it.
But there is still the question as to the regression in performance?
what changed in netcdf or ncdump in this regard???
Also -- the OP states they first noticed this with GDAL, not ncdump --
so maybe GDAL needs to be smarter about setting the cache size -- but
I'm thinking that netcdf, or even hdf, may want to increase the default
chunk cache size.
I see this in the HDF docs: "The default size is 1 MB" -- 1MB was a lot
not that long ago, but with laptops sporting many GB of memory -- it
would make sense to make the default a lot bigger (10% of system memory?
even 1% of system memory would be more these days.
Of course, another reminder that you really don't want to use large (or
highly non-square) chunk sizes unless you really know your access patterns.
I did some experiments with this a while back, because netcdf's default
chunk sizes had a bug of sorts (1 element chunk sizes for 1-d arrays!)
-- in that one use case, with that particular system, the writing
performance was about the same once chunk sizes got about about 1k or
so. I can't recall if I tested read performance.
The point being -- really large chunks are not helpful, and can kill you
if you are using an incompatible access pattern
-CHB
See
https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetChunkCache
<https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetChunkCache>
There is quite a good doku here about general aspects of Chunking:
https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/
<https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/>
Julian
2016-12-15 20:17 GMT+01:00 Chris Barker <chris.barker@xxxxxxxx
<mailto:chris.barker@xxxxxxxx>>:
> On Wed, Dec 14, 2016 at 9:13 PM, Dave Allured - NOAA Affiliate
> <dave.allured@xxxxxxxx <mailto:dave.allured@xxxxxxxx>> wrote:
>
>>
>> So I think you have a read cacheing failure, due to interaction
between
>> the ncdump read pattern, and your chunking scheme.
>
> ...
>>
>> A sampling tool found that ncdump was spending more than 96% of
its time
>> inside an HDF5 chunk reader with decompression. Every time an
HDF5 chunk is
>> physically read from disk, the *entire* chunk must be
decompressed, even to
>> access a single value. You see why chunk cacheing is important.
>
>
> Does HDF5 not do any chunk caching itself? or for that matter,
netcdf4? Is
> is really up to the application level to manage the caching? that
seems like
> handling it at the wrong level to me.
>
> -CHB
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959
<tel:%28206%29%20526-6959> voice
> 7600 Sand Point Way NE (206) 526-6329
<tel:%28206%29%20526-6329> fax
> Seattle, WA 98115 (206) 526-6317
<tel:%28206%29%20526-6317> main reception
>
> Chris.Barker@xxxxxxxx <mailto:Chris.Barker@xxxxxxxx>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
<http://www.unidata.ucar.edu/mailing_lists/>
--
http://wr.informatik.uni-hamburg.de/people/julian_kunkel
<http://wr.informatik.uni-hamburg.de/people/julian_kunkel>
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@xxxxxxxx <mailto:Chris.Barker@xxxxxxxx>
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
