Steve Hankin wrote:
John Caron wrote:
Thank you!
BTW, I'll be looking at your case next, in the "new types of
aggregation".
Hi John,
I'll take that as an invitation to elaborate a little on the needs for
forecast data set collections. To reiterate, the challenge is to expose
three views of a collection of successive (operational) forecasts:
1. the "integration foreward in time" axes (one virtual data set per
model run)
2. the "sequence of successive forecasts" axis (one virtual dataset
per forecast time increment)
and, if possible
3. the "successively shorter forecast of the same endpoint" axes (one
virtual dataset per end time)
just to make sure i understand, i am going to reword this:
a "ForecastModelRun" is a collection of all the forecasts for one model run. we
make this into a single netcdf file.
a "ForecastModelRun collection" is a collection of ForecastModelRun files for
the same model.
so
#1 = is a virtual dataset consisting of all the time=0 slices in each
modelForecastRun, except the last, which uses all of the time steps.
#2 = is all of the time steps in a ForecastModelRun
#3 = the set of time steps with constant forecast time
Assuming this is correct, then we already have #2. #1 is probably
straightforward to do as a special kind of aggregation on the server.
#3 is harder because there is a whole collection of these, one for each
forecast time. Im not sure whether to make seperate datasets for each, or make
a more complex object that is parameterized by forecast time (I assume thats
what you mean by a 5-D dataset).
Any of these aggregatations could efficiently be done on a client, and i might
be tempted to do it on the client in the nj22 library. But then only java apps
could use it.
I will take your advice and investigate the vanilla server approach, ie making
seperate (virtual) datasets on the server that can be accessed by any dods
client.
The parentheses indicate where a special challenge is -- namely, that
the collection proliferates into a number of discrete virtual datasets.
This of course is a reflection of reality; each time the forecast model
is run, it truly does generate a new dataset.
Here's what I want to add to the discussion: As a software developer --
striving for simplicity, compactness, elegance -- it may be tempting to
solve this by using a more-than-4-dimensional array. I'd like to lobby
that while of course it is fine (and fun) to explore new solution
spaces, it is also vital to include in the solution an approach that
makes the aggregations look like "standard" (CF) datasets -- i.e.
4-dimensional. Else, standard CF clients will not be able to read the
datasets.
I suspect that this will involve something like the synthesis of a
virtual "file system" in the URLs
http://IP/server/forecast_collection ...
/by_start_time...
/2006-01-01
/2006-01-02
/2006-01-03
...
/by_forecast_increment ...
/z-0
/z-4hrs
/z-8hrs
...
/by_end_time
/2006-01-03
/2006-01-04
/2006-01-05
...
Just initial thoughts. A unique problem, eh?
- steve
Steve Hankin wrote:
Yahoo! Congratulations.
- steve
John Caron wrote:
A new stable release of the THREDDS Data Server 3.4 is now ready at
http://www.unidata.ucar.edu/projects/THREDDS/tech/index.html#TDS
Among other feaures, this version now has all the functionality of
the older OpenDAP Aggregation Server (AS), which is no longer
supported. The AS allowed aggregation of static sets of files, using
"Union", "Join Existing" and "Join New" types. The TDS also adds
automatic scans of directories, plus manipulation of the dataset
metadata using NcML.
The configuration of the TDS is significantly different than the AS,
since the TDS uses NcML Aggregation. See:
http://motherlode.ucar.edu:8080/thredds/docs/NcML.htm
http://www.unidata.ucar.edu/software/netcdf/ncml/
You are encouraged to upgrade to new stable releases as they become
available, since various bugs are fixed. A stable release means that
no further features will be added, and only bug fixes will be made.
New development is now being done on the 3.5 version. This version
will be adding more features, especially to deal with dynamically
changing datasets, new types of aggregation, and some of the
sophisticated features of TDS catalog generation (filter, sort,
name) will be pushed into NcML aggregation. Use the development
version only if you need the new features and are willing to bleed a
little bit.
--
--
Steve Hankin, NOAA/PMEL -- Steven.C.Hankin@xxxxxxxx
7600 Sand Point Way NE, Seattle, WA 98115-0070
ph. (206) 526-6080, FAX (206) 526-6744
