> Our model outputs are typically about 2 Gbytes in size. We have an in-house
> direct access format that permits us to break this up into multiple files and
> a strategy that allows a "data set" (an associated group of files) to be the
> equivalent of a netCDF hyperslab such that the data set still shares the grid
> coordinates and indices of the full model output. This permits us in most
> cases to avoid working with the full multi-gigabyte data set.
>
> At present we have adapted the hyperslab strategy to netCDF files (using a
> handfull of netCDF attributes) but we have not yet implemented the ability to
> split the netCDF data set into multiple files. Because of this we havn't
> been using the netCDF format for our HUGE files yet - order 10-20 Mbytes,
> only so far. But we will likely be facing similar performance issues to
> your own in the future.
[stuff deleted]
> | NOAA/PMEL | ph. (206) 526-6080
> Steve Hankin | 7600 Sand Point Way NE | FAX (206) 526-6744
> | Seattle, WA 98115-0070 | hankin@xxxxxxxxxxxx
At University of Illinois we are working on a data management strategy
for netCDF files similar to yours. We add a layer on top of the
netCDF library and group related variables and dimensions in the files
together as a "project." Variables which span multiple files can then
be accessed as single entities. We have not tested our approach on a
data set of that size, yet. I would be very interested in hearing how
your strategy for defining and using a "dataset" works.
This work is part of our Envision software project, which will support
visualization and manipulation of geophysical data sets. The first
release of Envision is scheduled for the first of the year and the
code, including source, will be made freely available. I would be
happy to provide additional information for anyone who is interested.
--------------------------------------------------------------------------
Keith Searight, Research Programmer keith@xxxxxxxxxxxxxxxxxxxx
Univ. of Illinois, Atmospheric Sciences Ph. (217) 333-8132
105 S. Gregory Ave., Urbana, IL 61801 Fax (217) 244-4393
