Hi Russ,
Thanks for the info. On reading your reply I too recollected reading
somewhere that the maximum chunk size is 2Gb.
I'll try to break down the chunk sizes --- i also did not know that the
entire chunk is to be read into memory.
---
Samrat.
On Mon, Oct 27, 2014 at 11:53 PM, Russ Rew <russ@xxxxxxxxxxxxxxxx> wrote:
> Hi Samrat,
>
> You're requesting chunks over 17GB in size. If you were to read even a
> single value from a chunk of that size, it would require reading the full
> chunk into memory first, since chunks are the atomic units of data access
> for chunked datasets.
>
> The theoretical limit on the size of chunks is how much contiguous memory
> can be allocated in one call, which is 2^31 on 32-bit systems and 2^63 on
> 64-bit platforms. However, the practical limit is how much memory it's
> possible to use and you're willing to use for an I/O buffer. I have a
> recollection that the HDF5 library used as the storage layer for netCDF-4
> limits chunks to be smaller than 2 GB, but I can't find where that's
> documented. Looking at the netCDF library code, I can't see an enforced
> maximum size, other than the maximum value of the size_t type, but we
> should add that to the documentation if it's not there.
>
> However, you don't want chunks to be very small or very large just for the
> sake of efficiency. The ideal size and shape for a chunk is dictated by
> what access patterns you anticipate to be common, to make each access one
> chunk.
>
> --Russ
>
> On Mon, Oct 27, 2014 at 12:13 AM, Samrat Rao <samrat.rao@xxxxxxxxx> wrote:
>
>>
>> Hi,
>>
>> I am using netCDF-fortran-4.1.1 with netCDF-4.3.2. I get an error 'Bad
>> chunk sizes' when the 3D data size is more than a certain number.
>>
>> I am unable to understand the cause of this error.
>>
>> My OS is CentOS 6.6. The errors are the same even if is use netCDF (v
>> 4.1.1) supplied by the CentOS community.
>>
>> i am attaching a minimal example. Please let me know if there are any
>> errors from my side.
>>
>> Thanks,
>> Samrat.
>>
>> --
>>
>> Samrat Rao
>> Research Associate
>> Engineering Mechanics Unit
>> Jawaharlal Centre for Advanced Scientific Research
>> Bangalore - 560064, India
>>
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>
>
--
Samrat Rao
Research Associate
Engineering Mechanics Unit
Jawaharlal Centre for Advanced Scientific Research
Bangalore - 560064, India
