Gerry Creager N5JXS wrote:
OK, so I'm catching up.
Bib, what you describe is very much what I do with a relational database
(spatially aware). Admittedly most of the sites we work with are fixed
geographically, but the time series "stuff" is consistent with what
you're looking to do. In fact, we're now working with a lot of th buoys...
We maintain a station file with geoposition data (spatially indexed,
too), and a data table (or 'n' data tables in the normalization we're
attempting now). The two tables are linked by an indexing relation.
I've been able to request (quickly) all site data within a rectangle
defined in lat/lon and for a time series, ordered sequentially by
date/time stamp.
Is that what you're trying to achieve? I'm not aware of open source
tools that will take, say, a PostGIS dataset and turn it into a netCDF
file. I suspect that can be engineered, though.
Am I going down the line you're interested in? If so, I can expand a
bit. We've been doing this for surface met data for a while.
It sounds like we are doing similar things, but you are querying a
relational database and I want to query OPeNDAP (which will get the data
from a .nc file). I want to use OPeNDAP because I want to access the
data from other computers.
The big question for me has boiled down to: are NetCDF Structures (in
the specific sense of the word) only accessible by reading
structure-by-structure through the file (because they are stored as a
linked list), or, can they be accessed individually in an *efficient*
way (e.g., give me structure number (a.k.a. record number) 4537 from the
file)? I understand that OPeNDAP may be able to give me structure
#4537 either way, but I want to know if OPeNDAP can do it *efficiently*
(or if it has to read though record #0 - 4536 to get to #4537).
Stated another way: does using Structures preclude *efficient* indexed
access to individual Structures in a file?
Roy Mendelssohn pointed me to this clue: The Unidata Observation Dataset
Conventions document indicates that Structures can be stored as a linked
list or a contiguous list. But it is unclear if this section of the
document is relevant (is it just related to dealing with multiple
parents?) and in any case, I don't understand how to set up the .nc file
to use one approach or the other. (I understand linked lists; I just
don't understand how you are suggesting that I implement them.) And it
is unclear to me if the contiguous list is really storing all of the
Structures together on the disk for fast, indexed access to each
individual Structure from a given parent, or if it is just storing all
the Structures for each parent together (a contiguous list where each
element points to a linked list).
I'm still confused.
gerry
John Caron wrote:
Bob Simons wrote:
John Caron wrote:
Hi Bob:
We've been working on these kinds of problems, and thinking about
how/whether to use sequences or not. So im just going to work
through your use case below:
Bob Simons wrote:
I am trying to figure out how to store lots of sequence-like data
in .nc files for efficient access via OPeNDAP. In particular, I am
trying to determine if actual OPeNDAP Sequences (Structures with an
unlimited dimension in the .nc file) is not appropriate for our
purposes.
Yes, I could store the data in a file on the computer where the
program needing access is running, and not have to access it via
OPeNDAP, so that network transmission time would be minimized. But
this project is partly an experiment in dealing with remotely
accessed data. So I am trying to design a solution where the data
is accessed from another computer via OPeNDAP.
Here's an example. Let's say I want to store all NDBC buoy data in
a .nc file. There are over 100 buoys. For each buoy, there are
readings for some time period (e.g., just 1989, or from 1990 to the
present). The readings are an hour apart. Several variables (e.g.,
WindSpeed and WindDirection) are measured at each time point. Since
we work with real-time data, I plan to update this file frequently
(every day, but ideally every hour).
How large do you expect the file to get (total number of readings ) ?
reading == record == one structure in the sequence.
Approximately:
There are 400 buoys * ~8 years of data * 8760 hours/year =
~28,000,000 records.
Given the great variation to time ranges for each buoy, I will
probably arrange it as 400 sequences (one per buoy), with an average
of 8 * 8760 = 70,080 records per sequence.
Each record has 1 double and 15 floats; hence
68 bytes/record * 28,000,000 records = ~1.9GB
The problem is, I need to have *quick* access via OPeNDAP:
* Across all buoys at a specific time point, e.g., What is the wind
speed at all buoys at 2004-12-14T09:00Z?
* Or, for all time points available, what is the wind speed, for
example, at a specific buoy?
Do you need other queries, like "find all readings with wind speed >
30 mph" ??
In general, no.
The most common variant of the first request above is: restrict the
request to all buoys in a rectangular geographic region, .... But I
can separately manage and subset the geographic locations of the
buoys, if needed.
Regarding the first requirement, from what I understand, if I use
sequences, there is no way to get the data for a given time point
without reading either the whole file up to that time point, or
without reading a whole variable. Either of which would seem to
take too long if I want the values for 100 buoys (given that I am
using OPeNDAP to connect to a remote computer and want the response
quickly for my CoastWatch Browser program, which graphs the data
for on-line users who want a quick response).
im not sure how your specific dods server works, but theres a good
change that the server has to read the entire file to answer your
query. We need to find that out, if we are going to figure out how
to scale this.
I definitely want to avoid having the server read the entire file for
each query. That's why I ask (below) about avoiding structures and
just using lots of variables.
in the opendap world, you can in fact put a CE (constraint
expression) on the sequence, eg your first query would be something
like "time = 2004-12-14T09:00Z", and your second "buoy=2309" (ill
have to check the exacct syntax). Now we dont yet properly support
that in the nj22 library, but I think it may not be that hard to do.
The hard problem is probably on the server, if it has to read the
entire file to answer it.
I'm not sure why you say it has to read the entire file. If I set up
the file in certain ways, can't that be avoided?
it depends on the server.
Since the time range of available data for each buoy varies
greatly, it seems grossly wasteful of space to have a common Time
dimension for all buoys. Doing so would probably force me over the
2GB file size, which is generally trouble. So I am thinking about
either:
* A time dimension for each buoy (e.g., time14978 for buoy 14978)
and a several variables which use that dimension to store the data
for that buoy (e.g., windSpeed14978, windDirection14978, etc.).
This setup would be replicated for each buoy.
I am leaning toward this. It is easy to understand. It doesn't use
any special features of .nc or OPeNDAP, so should work will different
servers and clients.
* Or, a Group for each buoy, again with a time dimension and
several variables in each group to store the data for each buoy.
(If this is a new .nc feature, does OPeNDAP deal with this yet?)
* Or, an ArrayObject.1D of variables, each element of which is an
ArrayObject.1D of the variables for a given buoy. (I'm not sure if
this can be done.)
* Or, an ArrayObject.2D of variables, with buoys as one dimension
and the various variables (e.g., WindSpeed, WindDirection) on the
other dimension. (I'm not sure if this can be done.)
Our current thinking on how to write netcdf files for "observation
data" is written up at:
http://www.unidata.ucar.edu/software/netcdf-java/formats/UnidataObsConvention.html
in particular, appending records using backwards-linked lists seems
like a good solution, and its what we are currently doing with the
realtime metar data on motherlode.
I really don't like linked lists. They force each query to go through
the all the rows (and read all of the data for each row).
I think separate variables are the way to go. They can be made
expandable in the way that Java's ArrayList is expandable: have a
backing array, and keep track of how many elements are currently in
use (size). If you need more capacity, make a new larger array and
copy the values to it. Then you can get random access to any value
in any variable. Further, if you need to do constraints, you only
need to read the constraint variables, and even then you can minimize
the reads. For example, if I sort an buoy's records by time and have
a query like "time >= t1 && time <= t2 && windSpeed > 30", I don't
even have to read the windSpeed variable until I find a record in the
correct time range. And I never have to read the other variables
until I know the constraint expression is satisfied.
In fact, part of my raising this question was to try to figure out
why/when structures/sequences are a good approach. I feel like I'm
missing something. It looks like they are implemented as linked
lists. If so, they don't look like a good data structure to me,
because they force all file accesses to go through the whole file.
They are efficient when appending data (which you do infrequently and
when you care less about speed), but inefficient for searches (both
sequential searches of one or a few variables, or random access to
any datum) (which you do frequently and when you really care about
speed). And there are other data structures (in the style of Java's
ArrayList) which are efficient for writing (random access or
appending) and reading (sequential or random access). Comments?
Im a bit confused if were talking about the server or the client?
Im assuming that you want to write a netcdf file that a opendap server
can serve? I dont know of any that would automatically serve
sequences. We are looking at adding that to the THREDDS data server
(TDS), but havent yet.
Netcdf-3 files can only be expanded along one dimension. Netcdf-4 wont
have that limitation, but they arent ready yet. In the
UnidataObsConvention, we write along the record dimension as the data
comes in, there's a seperate linked list for each station. As long as
you keep the files moderately small, this isnt a bad solution for
modest datasets. We may add an external indexing capability in the TDS
for large datasets.
To optimize the search, I would probably not use the linked list, but
the "contiguous list" option (see the UnidataObsConvention.html). You
definitetly want to use the record dimension, by the way, you could
see a factor of 100 times slower without it. You could have the
current data come into a daily file (using linked lists?), then add
the daily file to the archive, with periodic rewriting of the file.
But none of these are options unless the server can deal with it. So
these are just some ideas that I am considering for the TDS server.
Im afraid Ive run out of time to go into more depth. We'll have to
continue this after the holidays.
Happy Holidays!
I plan to solve the updating problem by leaving rows of missing
values at the end of the data for each active buoy. As new data
comes in, I will replace the missing values with actual data. Then,
I only have to rewrite the file (to add more rows of missing
values) once in a while, not every time.
the above approach, if it works for you, probably obviates this.
Which approach sounds best? Is there another approach? Do you have
any advice?
Are sequences the wrong way to go? Of course, that could change if
one could efficiently access specific ranges from variables in a
Sequence/Structure. But it my understanding that that is not
currently possible.
The DAP 2 spec currently does not allow this. But the whole point of
sequences is to allow you to subset using a query (i think its
called a selection), and then only return the data needed, so you
dont need index subsetting.
But if the server has to go through the whole file, it will never be
fast.
Although I gave this specific example, we store a lot of
sequence-like data where I work. Whatever .nc file structure is
appropriate for the buoys will likely be appropriate for much of
this other data. So I want to get it right.
Right now, Id say that it depends on what server you are using.
Sequences are elegant, but they are a different animal from indexed
access that is the bread and butter of netcdf files.
The critical things to answer first:
1. How many records will you serve? What about in the future?
Approximately:
400 buoys * ~8 years of data * 8760 hours/year = ~28,000,000 records
Each record has 1 double and 15 floats; hence
68 bytes/record * 28,000,000 records = ~1.9GB
More than half of the buoys are active so it will grow by about:
300 buoys * 8760 hours/year = 2,628,000 records / year
(almost 200MB/year)
Given that it is close to 2GB, I may separate it into a file for
inactive buoys and a file for active buoys.
2. What queries do you need to support?
3. What response time is acceptable ?
I would like 1 second search time on the server + whatever the
network transmission time is for OPeNDAP to send me the results. Note
that the results are often/usually < 100 KB of data. I am willing to
do a lot to get that response time, e.g., store the buoy locations
and time ranges in memory. Buoy readings are every hour, but with
gaps. So perhaps I would constrain the times for each buoy's reading
to be regularly spaced (e.g., missing data would appear as rows of
missing values in the file), so that I can very quickly calculate the
relevant row(s) of data based on time constraints.
4. What clients do you want to support? Just your own, or more
general?
I guess I only care about my client. But it seems like if I do this
right, it will be useful to any client that works with a given
server. For me, OPeNDAP is here now and available for no effort on my
part. So, any OPeNDAP client can use the .nc file. Presumable, other
servers (LAS, THREDDS) could use the file, too, in the future.
5. What server do you want to use? Does it matter?
It doesn't matter to me, except for ease of use. So I'm strongly
inclined to use one of the OPeNDAP servers which is already
administered here (by someone else). I'll make the file. They'll
serve it.
Sincerely,
Bob Simons
Satellite Data Product Manager
Environmental Research Division
NOAA Southwest Fisheries Science Center
1352 Lighthouse Ave
Pacific Grove, CA 93950-2079
(831)658-3205
bob.simons@xxxxxxxx
<>< <>< <>< <>< <>< <>< <>< <>< <><
--
Sincerely,
Bob Simons
Satellite Data Product Manager
Environmental Research Division
NOAA Southwest Fisheries Science Center
1352 Lighthouse Ave
Pacific Grove, CA 93950-2079
(831)658-3205
bob.simons@xxxxxxxx
<>< <>< <>< <>< <>< <>< <>< <>< <><
