Thanks for taking on oceanographic conventions for NetCDF.
LANL plans to generate Terabytes of data from an Ocean
Model executing on a Connection Machine (CM-5). We are interested
in conventions so that we may utilize as many off-the-shelf
products as possible for post-processing.
Many of the issues brought up are of interest, but the biggest
concern currently is
>19) Huge Data Sets / Multiple Files
We wish to put data generated from the Ocean model directly to
tape, perhaps without metadata in the file. Your proposal:
>Alternative 1: a "parent" netCDF file with variables and
> attributes suitably defined to point to "child" files.
sounds like the direction that we are interested in pursuing.
We would like a centralized (or parent) metadata file (that would
reside on disk) which would contain NetCDF metadata and
enough information to be able to request that a hyperslab be
read from one or more tapes (associated with a file) and cached
to disk for access.
It is desired that the files on tape have as few limitations as
possible, in other words:
- they can contain metadata for the specific tape file (the
metadata will be ignored for the parent metadata; this will
make it difficult to append data to the end of the tape file) OR
have no metadata in the tape file (this would allow tape files to
easily have new data appended)
- they can consist of differet amounts of array sets (in other words
data from new model runs can be appended at any time)
Another concern is a modification required to NetCDF to perform
Connection Machine read and writes, rather than standard Unix I/O.
I want to write data using CMIO commands and write the metadata
using UNIX I/O and impact the code as little as possible.
Any one else have this problem?
> Compression
Echoing previous comments, compression that is automatic and happens
inside the NetCDF routines would be wonderful. It would mean, however,
based on the comments above, that NetCDF routines would be able to
uncompress, on the fly, a compressed hyperslab cached to disk from tape.
christina mercier
cwm@xxxxxxxx
los alamos, new mexico
