As a quick answer to the question, we (Sandia Labs) use netcdf
underneath our exodusII
file format for storing finite element results data.
If the mesh contains #nodes nodes and #elements elements, then there
will be a dataset of the size #elements*8*4 (assuming a hex element with
8 nodes, 4 bytes/int) to store the nodal connectivity of each hex
element in a group of elements (element block). Assuming 4GiB, this
limits us to ~134 Million elements per element block (using CDF-2) which
is large, but not enough to give us more than a few months breathing
room. Using CDF-1 format, we top out at about 30 million elements or
less which is hit routinely.
There is a pdf file at
http://endo.sandia.gov/SEACAS/Documentation/exodusII.pdf that shows
(starting at page 177) how we map exodusII onto netcdf. There have been
some changes since the report was written to reduce some of the dataset
sizes. For example, we split the "coord" dataset into 3 separate
datasets now and we also split the vals_nod_var into a single dataset
per nodal variable.
--Greg
John Caron wrote:
> Hi Rob:
>
> Could you give use case(s) where the limits are being hit?
> I'd be interested in actual dimension sizes, number of variables,
> whether you are using a record dimension, etc.
>
> Robert Latham wrote:
>> Hi
>>
>> Over in Parallel-NetCDF land we're running into users who find even
>> the CDF-2 file format limitations, well, limiting.
>> http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/NetCDF-64-bit-Offset-Format-Limitations.html
>>
>>
>> http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#Large%20File%20Support10
>>
>>
>> If we worked up a CDF-3 file format for parallel-netcdf (off the top
>> of my head, maybe a 64 bit integer instead of an unsigned 32 bit
>> integer could be used to describe variables), would the serial netcdf
>> folks be interested, or are you looking to the new netcdf-4 format to
>> take care of these limits?
>>
>> Thanks
>> ==rob
>>
>
> ==============================================================================
>
> To unsubscribe netcdfgroup, visit:
> http://www.unidata.ucar.edu/mailing-list-delete-form.html
> ==============================================================================
>
>
>
==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================
