Hi Steve,
I am not sure if I get what is ERDDAP exactly is, but I think, in the
oceanotron approach, the ERDDAP protocol for in-situ data would be a
point feature opendap front-desk (ie interface for the users).
We already have developed internal transformation which can transform
2-levels features (profiles, pointSeries, trajectories) into simple
points feature as disseminated in ERDDAP. So this could be easy to add
the ERDDAP front-desk to Oceanotron and disseminate any supported
dataset through this interface.
For the inverse way, I am not sure if this would be efficient or useful
to re-assemble the 2 level features from ERDDAP 1 level feature. We
would rather take them as point features and re-disseminate it as-is
(point features) in other protocols if required.
I fully agree with your idea of focusing on point features to quickly
reach a given interoperability level, as :
1. this point feature structure is enough for many users (from what I
know : satellite calibration/validation, numerical model assimilation),
2. this allows a mid-term range work on 2 level features standardization
(profile, timeSeries, trajectories) for more advanced users.
3. this approach does not mean the in-situ data managers/providers
should store or manage their in-situ data as points but only disseminate
them as point.
I'll be very happy to be able to reach interoperability with your latest
tools, and oceanotron has been designed to allow extension to various
protocols (and rather focus on the standardization observation data
models). The hard point is now to manage (with time or funding, or both)
the numerous interesting protocols to be implemented for in-situ datasets !
Best regards,
Thomas
PS. ncStream looks interesting as well.
On 12/07/2010 07:05 PM, Steve hankin wrote:
On 12/7/2010 9:47 AM, Roy Mendelssohn wrote:
Hi Thomas:
Thanks. Your timing couldn't have been better. There is a lot of
discussion in NOAA right now on how to serve observational data. I
am certain this will be given a good look.
Hi Thomas,
I agree on the spirit of Roy's remark, though not entirely on the
details. It would have been nice if we had had "oceanotron" on our
radar during the UAF meeting a few weeks ago. We did discuss DAPPER,
but not in great detail. So lets open an email discussion how best to
pull the threads (no pun) together.
Our UAF group consensus was to focus on the simpler, non-nested
ERDDAP protocol, (and the ASCII equivalent, as well) rather than the
DAPPER protocol. One straight-forward direction to move from here is
to utilize the ERDDAP system as a gateway from your DAPPER offerings
into the ERDDAP protocol(s). What about the inverse? Can Oceanotron
read the single-level ERDDAP OPeNDAP protocol and re-assemble the
2-level nesting in order to serve as a gateway into DAPPER? Granted,
there are some inefficiencies in taking this approach, but there are
potentially important interoperability gains.
And of course there are interoperability considerations (and implicit
development tasks) regarding ncStream, as well.
- Steve
=======================
Regards,
-Roy
On Dec 7, 2010, at 9:34 AM, Thomas LOUBRIEU wrote:
Dear Roy, dear all,
We've just published our in-situ data server at Ifremer. It is fully
compliant with what has been done with DAPPER (and Dchart can
request our interface).
It disseminates vertical profiles datasets and soon it will
disseminate pointSeries (e.g. mooring datasets) and trajectories
(e.g. drifting buoys or ferry box).
The URL is : http://www.ifremer.fr/oceanotron/OPENDAP/
The request-able criteria are of course, x,y,z,t coordinates and id,
but also feature's metadata (platform code, institution) and
phenomenons values (e.g. temperatures). A session/pagination system
is also implemented (with OPeNDAP indices) to get big data subsets
results with several requests each of them not exceeding a quota.
The software is open-source and can be extended with modules to read
datasets (storage units), to transform datasets (transform unit) and
disseminate the datasets (front-desk). The communication between
these modules, especially observations features, is standardized
with Value Objects (their description is available at :
http://www.ifremer.fr/isi/oceanotron/umlVOReport/oceanotron_uml.html)
You can get the software documentation and source from our wiki :
https://gforge.ifremer.fr/wiki/doku.php?id=oceanotron:start
If you have any question or comment, please do not hesitate to send it.
Best regards,
Thomas
On 12/06/2010 05:53 PM, Roy Mendelssohn wrote:
Agreed. I said similar things at the meeting. But I was starting
from the discussions at the meeting, which was the CDM as
implemented in THREDDS. So I want to get clear in my mind what TDS
will be able to do for observational data first before looking for
alternatives.
-Roy
On Dec 2, 2010, at 2:42 PM, Upendra Dadi wrote:
Roy,
With regard WODB, the users want to be able to search not just
using physical coordinates like lat-lon bounding box, but also
using other attributes like institution, project, platform etc
which are usually not represented using coordinate variables. So
grouping the data based on physical coordinates might cause
performance issues when searching using other criteria. Would it
be possible to have some additional structures, something like a
database index, but without a full-fledge DBMS? Does TDS allow
such structures?
Upendra
Roy Mendelssohn wrote:
Hi John:
Since the UAF meeting in Seattle I have been giving some thought
about how to serve some large, important datasets, such as the
raw ICOADS observations or the WODB observations. While reading
over the PointObservation Conventions proposal on the CF site,
while the proposal makes it clear how I might put data into a
netcdf file, it doesn't make clear what the interplay might be
with a service in TDS, and how a possible service might be
affected by a very large dataset without further structure.
So it seems pretty clear that the ICOADS would be points. From
the example:
dimensions:
obs = 1234 ;
variables:
double time(obs) ;
time:long_name = "time of measurement" ;
time:units = "days since 1970-01-01 00:00:00" ;
float lon(obs) ;
lon:long_name = "longitude of the observation";
lon:units = "degrees_east";
float lat(obs) ;
lat:long_name = "latitude of the observation" ;
lat:units = "degrees_north" ;
float alt(obs) ;
alt:long_name = "vertical distance above the surface" ;
alt:standard_name = "height" ;
alt:units = "m";
alt:positive = "up";
alt:axis = "Z";
float humidity(obs) ;
humidity:long_name = "specific humidity" ;
humidity:coordinates = "time lat lon alt" ;
float temp(obs) ;
temp:long_name = "temperature" ;
temp:units = "Celsius" ;
temp:coordinates = "time lat lon alt" ;
attributes:
:CF\:featureType = "point";
Now I am assuming that in a TDS implementation of a service, I
will be able to select on the coordinate variables, is that
correct? Even so, for something like ICOADS, obs is quite large
and that extract could be quite slow unless either there is
additional structure or the TDS pre-fetches the coordinate
variables much as the present Dapper server does.
Other options would be to say have a file for each 10-degree
block, and then have TDS aggregate over the files - would this be
possible. Then the search would a lot faster when people want
time series in a region as opposed to more synoptic extractions.
Would the TDS service be supporting such an option? Or, as
netcdf-4 supports groups, to have 10-degree groups with 2-degree
subgroups, which would work as far as netcdf-4 is concerned, but
that is not the same as TDS knowing what to do with the hierarchy
or to take advantage of the structure.
My questions for Profiles (that is for the WODB) are pretty much
the same. I assume that the TDS service will be able to search
on the coordinate variables, is that correct? And I have the
issue with the fact that the profile dimension variable will get
quite large and without further structure the search could be
very slow. Adding the same types of structures mentioned above
would provide possible solutions, but only if TDS, as opposed to
netcdf4, supported them.
As you may have guessed, these are not theoretical questions - I
would really like to see ICOADS and WODB served as part of the
year 2 UAF effort. So now is a good time to start thinking about
how to do it correctly and what the service will be able to do.
Thoughts?
Thanks,
-Roy
**********************
"The contents of this message do not reflect any position of the
U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097
e-mail:Roy.Mendelssohn@xxxxxxxx<mailto:Roy.Mendelssohn@xxxxxxxx> (Note
new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www:http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
------------------------------------------------------------------------
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,
visit:http://www.unidata.ucar.edu/mailing_lists/
**********************
"The contents of this message do not reflect any position of the
U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097
e-mail:Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www:http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,
visit:http://www.unidata.ucar.edu/mailing_lists/
--
-------------------------------------------------------------
Thomas LOUBRIEU
IFREMER IDM/ISI
BP70
29280 Plouzane
FRANCE
email:Thomas.Loubrieu@xxxxxxxxxx
WWW :http://www.coriolis.eu.org/cdc
Tel.: (+33) (0)2 98 22 48 53
Fax: (+33) (0)2 98 22 46 44
-------------------------------------------------------------
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097
e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
--
-------------------------------------------------------------
Thomas LOUBRIEU
IFREMER IDM/ISI
BP70
29280 Plouzane
FRANCE
email: Thomas.Loubrieu@xxxxxxxxxx
WWW : http://www.coriolis.eu.org/cdc
Tel.: (+33) (0)2 98 22 48 53
Fax: (+33) (0)2 98 22 46 44
-------------------------------------------------------------
