I am trying to figure out how to store lots of sequence-like data in .nc
files for efficient access via OPeNDAP. In particular, I am trying to
determine if actual OPeNDAP Sequences (Structures with an unlimited
dimension in the .nc file) is not appropriate for our purposes.
Yes, I could store the data in a file on the computer where the program
needing access is running, and not have to access it via OPeNDAP, so
that network transmission time would be minimized. But this project is
partly an experiment in dealing with remotely accessed data. So I am
trying to design a solution where the data is accessed from another
computer via OPeNDAP.
Here's an example. Let's say I want to store all NDBC buoy data in a .nc
file. There are over 100 buoys. For each buoy, there are readings for
some time period (e.g., just 1989, or from 1990 to the present). The
readings are an hour apart. Several variables (e.g., WindSpeed and
WindDirection) are measured at each time point. Since we work with
real-time data, I plan to update this file frequently (every day, but
ideally every hour).
The problem is, I need to have *quick* access via OPeNDAP:
* Across all buoys at a specific time point, e.g., What is the wind
speed at all buoys at 2004-12-14T09:00Z?
* Or, for all time points available, what is the wind speed, for
example, at a specific buoy?
Regarding the first requirement, from what I understand, if I use
sequences, there is no way to get the data for a given time point
without reading either the whole file up to that time point, or without
reading a whole variable. Either of which would seem to take too long
if I want the values for 100 buoys (given that I am using OPeNDAP to
connect to a remote computer and want the response quickly for my
CoastWatch Browser program, which graphs the data for on-line users who
want a quick response).
Since the time range of available data for each buoy varies greatly, it
seems grossly wasteful of space to have a common Time dimension for all
buoys. Doing so would probably force me over the 2GB file size, which is
generally trouble. So I am thinking about either:
* A time dimension for each buoy (e.g., time14978 for buoy 14978) and a
several variables which use that dimension to store the data for that
buoy (e.g., windSpeed14978, windDirection14978, etc.). This setup would
be replicated for each buoy.
* Or, a Group for each buoy, again with a time dimension and several
variables in each group to store the data for each buoy. (If this is a
new .nc feature, does OPeNDAP deal with this yet?)
* Or, an ArrayObject.1D of variables, each element of which is an
ArrayObject.1D of the variables for a given buoy. (I'm not sure if this
can be done.)
* Or, an ArrayObject.2D of variables, with buoys as one dimension and
the various variables (e.g., WindSpeed, WindDirection) on the other
dimension. (I'm not sure if this can be done.)
I plan to solve the updating problem by leaving rows of missing values
at the end of the data for each active buoy. As new data comes in, I
will replace the missing values with actual data. Then, I only have to
rewrite the file (to add more rows of missing values) once in a while,
not every time.
Which approach sounds best? Is there another approach? Do you have any
advice?
Are sequences the wrong way to go? Of course, that could change if one
could efficiently access specific ranges from variables in a
Sequence/Structure. But it my understanding that that is not currently
possible.
Although I gave this specific example, we store a lot of sequence-like
data where I work. Whatever .nc file structure is appropriate for the
buoys will likely be appropriate for much of this other data. So I want
to get it right.
Thank you.
Sincerely,
Bob Simons
Satellite Data Product Manager
Environmental Research Division
NOAA Southwest Fisheries Science Center
1352 Lighthouse Ave
Pacific Grove, CA 93950-2079
(831)658-3205
bob.simons@xxxxxxxx
<>< <>< <>< <>< <>< <>< <>< <>< <><
