Bob Simons wrote:
John Caron wrote:
Hi Bob:
We've been working on these kinds of problems, and thinking about
how/whether to use sequences or not. So im just going to work through
your use case below:
Bob Simons wrote:
I am trying to figure out how to store lots of sequence-like data in
.nc files for efficient access via OPeNDAP. In particular, I am
trying to determine if actual OPeNDAP Sequences (Structures with an
unlimited dimension in the .nc file) is not appropriate for our
purposes.
Yes, I could store the data in a file on the computer where the
program needing access is running, and not have to access it via
OPeNDAP, so that network transmission time would be minimized. But
this project is partly an experiment in dealing with remotely
accessed data. So I am trying to design a solution where the data is
accessed from another computer via OPeNDAP.
Here's an example. Let's say I want to store all NDBC buoy data in a
.nc file. There are over 100 buoys. For each buoy, there are readings
for some time period (e.g., just 1989, or from 1990 to the present).
The readings are an hour apart. Several variables (e.g., WindSpeed
and WindDirection) are measured at each time point. Since we work
with real-time data, I plan to update this file frequently (every
day, but ideally every hour).
How large do you expect the file to get (total number of readings ) ?
reading == record == one structure in the sequence.
Approximately:
There are 400 buoys * ~8 years of data * 8760 hours/year = ~28,000,000
records.
Given the great variation to time ranges for each buoy, I will probably
arrange it as 400 sequences (one per buoy), with an average of 8 * 8760
= 70,080 records per sequence.
Each record has 1 double and 15 floats; hence
68 bytes/record * 28,000,000 records = ~1.9GB
The problem is, I need to have *quick* access via OPeNDAP:
* Across all buoys at a specific time point, e.g., What is the wind
speed at all buoys at 2004-12-14T09:00Z?
* Or, for all time points available, what is the wind speed, for
example, at a specific buoy?
Do you need other queries, like "find all readings with wind speed >
30 mph" ??
In general, no.
The most common variant of the first request above is: restrict the
request to all buoys in a rectangular geographic region, .... But I
can separately manage and subset the geographic locations of the buoys,
if needed.
Regarding the first requirement, from what I understand, if I use
sequences, there is no way to get the data for a given time point
without reading either the whole file up to that time point, or
without reading a whole variable. Either of which would seem to take
too long if I want the values for 100 buoys (given that I am using
OPeNDAP to connect to a remote computer and want the response quickly
for my CoastWatch Browser program, which graphs the data for on-line
users who want a quick response).
im not sure how your specific dods server works, but theres a good
change that the server has to read the entire file to answer your
query. We need to find that out, if we are going to figure out how to
scale this.
I definitely want to avoid having the server read the entire file for
each query. That's why I ask (below) about avoiding structures and just
using lots of variables.
in the opendap world, you can in fact put a CE (constraint expression)
on the sequence, eg your first query would be something like "time =
2004-12-14T09:00Z", and your second "buoy=2309" (ill have to check the
exacct syntax). Now we dont yet properly support that in the nj22
library, but I think it may not be that hard to do. The hard problem
is probably on the server, if it has to read the entire file to answer
it.
I'm not sure why you say it has to read the entire file. If I set up the
file in certain ways, can't that be avoided?
it depends on the server.
Since the time range of available data for each buoy varies greatly,
it seems grossly wasteful of space to have a common Time dimension
for all buoys. Doing so would probably force me over the 2GB file
size, which is generally trouble. So I am thinking about either:
* A time dimension for each buoy (e.g., time14978 for buoy 14978) and
a several variables which use that dimension to store the data for
that buoy (e.g., windSpeed14978, windDirection14978, etc.). This
setup would be replicated for each buoy.
I am leaning toward this. It is easy to understand. It doesn't use any
special features of .nc or OPeNDAP, so should work will different
servers and clients.
* Or, a Group for each buoy, again with a time dimension and several
variables in each group to store the data for each buoy. (If this is
a new .nc feature, does OPeNDAP deal with this yet?)
* Or, an ArrayObject.1D of variables, each element of which is an
ArrayObject.1D of the variables for a given buoy. (I'm not sure if
this can be done.)
* Or, an ArrayObject.2D of variables, with buoys as one dimension and
the various variables (e.g., WindSpeed, WindDirection) on the other
dimension. (I'm not sure if this can be done.)
Our current thinking on how to write netcdf files for "observation
data" is written up at:
http://www.unidata.ucar.edu/software/netcdf-java/formats/UnidataObsConvention.html
in particular, appending records using backwards-linked lists seems
like a good solution, and its what we are currently doing with the
realtime metar data on motherlode.
I really don't like linked lists. They force each query to go through
the all the rows (and read all of the data for each row).
I think separate variables are the way to go. They can be made
expandable in the way that Java's ArrayList is expandable: have a
backing array, and keep track of how many elements are currently in use
(size). If you need more capacity, make a new larger array and copy the
values to it. Then you can get random access to any value in any
variable. Further, if you need to do constraints, you only need to read
the constraint variables, and even then you can minimize the reads. For
example, if I sort an buoy's records by time and have a query like "time
>= t1 && time <= t2 && windSpeed > 30", I don't even have to read the
windSpeed variable until I find a record in the correct time range. And
I never have to read the other variables until I know the constraint
expression is satisfied.
In fact, part of my raising this question was to try to figure out
why/when structures/sequences are a good approach. I feel like I'm
missing something. It looks like they are implemented as linked lists.
If so, they don't look like a good data structure to me, because they
force all file accesses to go through the whole file. They are efficient
when appending data (which you do infrequently and when you care less
about speed), but inefficient for searches (both sequential searches of
one or a few variables, or random access to any datum) (which you do
frequently and when you really care about speed). And there are other
data structures (in the style of Java's ArrayList) which are efficient
for writing (random access or appending) and reading (sequential or
random access). Comments?
Im a bit confused if were talking about the server or the client?
Im assuming that you want to write a netcdf file that a opendap server can serve? I dont know of any that would automatically serve sequences. We are looking at adding that to the THREDDS data server (TDS), but havent yet.
Netcdf-3 files can only be expanded along one dimension. Netcdf-4 wont have
that limitation, but they arent ready yet. In the UnidataObsConvention, we
write along the record dimension as the data comes in, there's a seperate
linked list for each station. As long as you keep the files moderately small,
this isnt a bad solution for modest datasets. We may add an external indexing
capability in the TDS for large datasets.
To optimize the search, I would probably not use the linked list, but the "contiguous list" option (see the UnidataObsConvention.html). You definitetly want to use the record dimension, by the way, you could see a factor of 100 times slower without it. You could have the current data come into a daily file (using linked lists?), then add the daily file to the archive, with periodic rewriting of the file.
But none of these are options unless the server can deal with it. So these are
just some ideas that I am considering for the TDS server.
Im afraid Ive run out of time to go into more depth. We'll have to continue this after the holidays.
Happy Holidays!
I plan to solve the updating problem by leaving rows of missing
values at the end of the data for each active buoy. As new data comes
in, I will replace the missing values with actual data. Then, I only
have to rewrite the file (to add more rows of missing values) once in
a while, not every time.
the above approach, if it works for you, probably obviates this.
Which approach sounds best? Is there another approach? Do you have
any advice?
Are sequences the wrong way to go? Of course, that could change if
one could efficiently access specific ranges from variables in a
Sequence/Structure. But it my understanding that that is not
currently possible.
The DAP 2 spec currently does not allow this. But the whole point of
sequences is to allow you to subset using a query (i think its called
a selection), and then only return the data needed, so you dont need
index subsetting.
But if the server has to go through the whole file, it will never be fast.
Although I gave this specific example, we store a lot of
sequence-like data where I work. Whatever .nc file structure is
appropriate for the buoys will likely be appropriate for much of this
other data. So I want to get it right.
Right now, Id say that it depends on what server you are using.
Sequences are elegant, but they are a different animal from indexed
access that is the bread and butter of netcdf files.
The critical things to answer first:
1. How many records will you serve? What about in the future?
Approximately:
400 buoys * ~8 years of data * 8760 hours/year = ~28,000,000 records
Each record has 1 double and 15 floats; hence
68 bytes/record * 28,000,000 records = ~1.9GB
More than half of the buoys are active so it will grow by about:
300 buoys * 8760 hours/year = 2,628,000 records / year
(almost 200MB/year)
Given that it is close to 2GB, I may separate it into a file for
inactive buoys and a file for active buoys.
2. What queries do you need to support?
3. What response time is acceptable ?
I would like 1 second search time on the server + whatever the network
transmission time is for OPeNDAP to send me the results. Note that the
results are often/usually < 100 KB of data. I am willing to do a lot to
get that response time, e.g., store the buoy locations and time ranges
in memory. Buoy readings are every hour, but with gaps. So perhaps I
would constrain the times for each buoy's reading to be regularly spaced
(e.g., missing data would appear as rows of missing values in the file),
so that I can very quickly calculate the relevant row(s) of data based
on time constraints.
4. What clients do you want to support? Just your own, or more general?
I guess I only care about my client. But it seems like if I do this
right, it will be useful to any client that works with a given server.
For me, OPeNDAP is here now and available for no effort on my part. So,
any OPeNDAP client can use the .nc file. Presumable, other servers (LAS,
THREDDS) could use the file, too, in the future.
5. What server do you want to use? Does it matter?
It doesn't matter to me, except for ease of use. So I'm strongly
inclined to use one of the OPeNDAP servers which is already administered
here (by someone else). I'll make the file. They'll serve it.
Sincerely,
Bob Simons
Satellite Data Product Manager
Environmental Research Division
NOAA Southwest Fisheries Science Center
1352 Lighthouse Ave
Pacific Grove, CA 93950-2079
(831)658-3205
bob.simons@xxxxxxxx
<>< <>< <>< <>< <>< <>< <>< <>< <><
