On Fri, Dec 6, 2019 at 6:45 AM Amr, Mahmoud <mahmoud.amr@xxxxxxxxxxxxxxxxx>
wrote:
> Maybe you can shed some light on how to set the chunk cache size and how
> to pick the chunk size? We used information provided by this article which
> suggests the chunk size should be small to get good average performance in
> all dimensions:
>
>
>
>
> https://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_why_it_matters
>
Chunking is tricky -- if you want to support all access patterns, you want
your chunks to be "square", not small.
squareness means that accessing any axis is equivalent.
But the optimum size of the chunks has to do with other things, like the
size of the disk cache, block size, etc.
"When a user or a program makes a request for data, the hard drive pulls
that data and the data around it on the platter and stores it all in the
buffer." (https://www.techjunkie.com/what-is-hard-drive-cache/):
So even if your program asks for only one byte, the disk is reading a much
larger "chunk" of data into cache anyway -- ideally your HDF chunk size
matches the chuck that the disk is putting in cache.
Which is hard, because you probably don't know the nature of the hardware
that will be used to access your files :-(
However, in limited experimentation, I found that performance wasn't very
sensitive to chunk sizes within a "reasonable" range:
- very small chunks absolutely kill performance (and make the files larger
than needed)
- very large chunks also kill performance for non-optimum access patterns.
The only thing to do (that I know of) is experiment with your "typical"
hardware and access patterns.
Another thought: do you really need access to ALL axes equally? maybe yes,
maybe no.
NOTE:
I'm not really up on hardware, but 64 x 64 x 64 is 256k -- that is pretty
darn small by today's standards. I'd start with at least 640x640x640
HTH,
-CHB
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@xxxxxxxx
