On 10/27/2014 11:30 PM, Chris Barker - NOAA Federal wrote:
On Oct 27, 2014, at 8:46 PM,
However, you don't want chunks to be very small or very large just
for the sake of efficiency. The ideal size and shape for a chunk
is dictated by what access patterns you anticipate to be common,
to make each access one chunk.
Indeed, but I suspect that you'd hardly notice the difference between
large and really large chunk sizes. For instance a chunk larger that
your disk cache would probably buy you nothing. What's a typical disk
cache these days? I have no idea.
But I'll guess you won't see a noticeable drop in performance for a
multi-gigabyte read if you use a 10MB or so chunk size.
close. Consider Mira, our large Blue Gene with GPFS. 16 MiB is a good
size for "go as fast as you can" -- on that file system the block size
is 8 MiB.
Some (small) multiple of the underlying file system block size is
probably as far as you need to go. Certainly less than 100 MiB in 2014
(though that number is likely to creep upwards over time)
==rob
If you do any testing, please post your results here.
-Chris
--Russ
On Mon, Oct 27, 2014 at 12:13 AM, Samrat Rao <samrat.rao@xxxxxxxxx
<mailto:samrat.rao@xxxxxxxxx>> wrote:
Hi,
I am using netCDF-fortran-4.1.1 with netCDF-4.3.2. I get an
error 'Bad chunk sizes' when the 3D data size is more than a
certain number.
I am unable to understand the cause of this error.
My OS is CentOS 6.6. The errors are the same even if is use
netCDF (v 4.1.1) supplied by the CentOS community.
i am attaching a minimal example. Please let me know if there
are any errors from my side.
Thanks,
Samrat.
--
Samrat Rao
Research Associate
Engineering Mechanics Unit
Jawaharlal Centre for Advanced Scientific Research
Bangalore - 560064, India
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
Samrat Rao
Research Associate
Engineering Mechanics Unit
Jawaharlal Centre for Advanced Scientific Research
Bangalore - 560064, India
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
