Re: [netcdfgroup] Error with large filesizes

On Oct 27, 2014, at 8:46 PM,

However, you don't want chunks to be very small or very large just for the
> sake of efficiency.  The ideal size and shape for a chunk is dictated by
> what access patterns you anticipate to be common, to make each access one
> chunk.
>

Indeed, but I suspect that you'd hardly notice the difference between large
and really large chunk sizes. For instance a chunk larger that your disk
cache would probably buy you nothing. What's a typical disk cache these
days? I have no idea.

But I'll guess you won't see a noticeable drop in performance for a
multi-gigabyte read if you use a 10MB or so chunk size.

If you do any testing, please post your results here.

-Chris



> --Russ
>
> On Mon, Oct 27, 2014 at 12:13 AM, Samrat Rao <samrat.rao@xxxxxxxxx> wrote:
>
>>
>> Hi,
>>
>> I am using netCDF-fortran-4.1.1 with netCDF-4.3.2. I get an error 'Bad
>> chunk sizes' when the 3D data size is more than a certain number.
>>
>> I am unable to understand the cause of this error.
>>
>> My OS is CentOS 6.6. The errors are the same even if is use netCDF (v
>> 4.1.1) supplied by the CentOS community.
>>
>> i am attaching a minimal example. Please let me know if there are any
>> errors from my side.
>>
>> Thanks,
>> Samrat.
>>
>> --
>>
>> Samrat Rao
>> Research Associate
>> Engineering Mechanics Unit
>> Jawaharlal Centre for Advanced Scientific Research
>> Bangalore - 560064, India
>>
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>
>


-- 

Samrat Rao
Research Associate
Engineering Mechanics Unit
Jawaharlal Centre for Advanced Scientific Research
Bangalore - 560064, India

_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit:
http://www.unidata.ucar.edu/mailing_lists/
  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: