Glenn:
I see observations data as time series as one kind of service, and weather
model, remote sensing and Nexrad data as another. For the weather and
climate "field" information, some variant of what John Caron is doing seems
appropriate. For the observations data, I think we have a fairly good scheme
figured out in the work we are doing with CUAHSI. I can see the netCDF
style of things for the weather and climate fields. I think the time series
stuff
could be handled that way but is easier and perhaps more appropriate just to
do it in XML via SOAP or REST.
What I would like to see developed is a "Weather and Climate Server" so that
the data users can go to one system and get into the various information sources
without having to go separately to NCDC, Unidata, NCAR, NWS, etc. This is
like WaterOneFlow that we are attempting to do in CUAHSI for water observation
data.
David
________________________________
From: Glenn.Rutledge@xxxxxxxx [mailto:Glenn.Rutledge@xxxxxxxx]
Sent: Fri 8/18/2006 6:39 PM
To: David Maidment
Subject: Re: RE: NCDC Web Services
Hi David,
I agree. Here's the note I sent Sharon earlier this week after your
request. I want to plan for the long term and work toward the benefit
of all. We are meeting on Monday. Don't worry about the lack of the
attachment I reference for the moment. Regards, Glenn
__________________________________
Sharon et al,
I think we need to have a set of meetings to define our plan for
exposing NCDC services. I'm not aware of the status of this kind of
activity already, but there are several efforts underway within the
Center and we're already on a similar track - we just need to solidify
our goals- and maybe generate a requirements document from which to work
toward. btw- NOAA's GEO-IDE would benefit from just such an activity.
Rich's efforts in OGC compliant (GIS) services are the way to go, and
NOMADS team have implemented a THREDDS Data Server (TDS) for OGC WCS
services and provides for OPeDNAP enabled requests handled under TDS.
Of course SteveD's group has advanced even higher levels of standards
using SOAP for more generalized services descriptions. There are
limitations to generalities with such standards, and not community based
standards such as OPeNDAP at the present time.
I'm co-PI on a NASA, Unidata, OPeNDAP and GMU effort called ACCESS that
has defined and are building just such a "Gateway" using NetCDF files
initially- using the COARDS and CF compliant OGC Catalog Services for
the Web (CSW). This cross- community effort has many players including
OGC, CEOS, GALEON, GO-ESSP and many others. It builds on the work of
the Common Data Model and TDS work at Unidata and OGC and International
services. There is work to do on my end: Grib/grib2/BUFR issues but
we've already made significant headway in this regard and participated
in the last GEO demo last May in Beijing. NCDC will be the benefactor of
this effort and RSAD will be installing this capability (in test mode)
when completed later this year.
The attached could be a staring point for these Center-wide discussions,
and I suggest that the attached be reviewed and considered for expansion
into the center when fully developed. The reason that I'm co-PI on the
effort is that NOMADS would install and provide this service for models.
It applies to other datasets as well but would need to be adapted for
Neal's/Steve's data. Comments? Glenn
What David Maidmont seeks of course is a unified NCDC web presence to
all of NCDC's resources.
----- Original Message -----
From: David Maidment <maidment@xxxxxxxxxxxxxxx>
Date: Friday, August 18, 2006 7:28 pm
Subject: RE: NCDC Web Services
Rich:
There are a lot of questions here that require some reflection.
What I want to do
in the short term is stand up a web service to NCDC that is as
simple as possible
that shows that we have a serious commitment to work together.
That is why CRN
is attractive to me -- its high quality data thus good for
research, its not very many
stations, so not too difficult to build the catalog even if we do
it station by station
using the web services that you created (do they access CRN?), and
hopefullyyour services will work onto the archive reasonably well.
I am trying to create a path that points the way to the future in a
methodicalmanner that step by step we figure out what comes next
and implement it
and we start with some fairly straightforward, confidence building
steps that
solidify the relationships which already exist on a technical level
betweenyou and your colleagues and us. We need to learn how you
think about
data at NCDC. We need to understand what databases you maintain, how
many services you actually offer and which ones we should focus on
first, which
ones leave to later.
Our first priority is access to historical NCDC data archives not
near real-time
data.
I want to understand what datasets you are publishing in netCDF-
accessibleform and how those dovetail with what is available from
NCAR and Unidata.
I don't know enough about NIDIS yet to comment on it in an informed
way.
Thanks for your feedback.
David
________________________________
From: Rich.Baldwin [mailto:Rich.Baldwin@xxxxxxxx]
Sent: Fri 8/18/2006 8:32 AM
To: David Maidment
Cc: Cedric David; Ilya Zaslavsky; Glenn Rutledge; Sharon Le Duc;
Neal Lott
Subject: Re: NCDC Web Services
Hello David and Ilya,
I was glad to meet you all last week. I had a productive meeting with
your teams at UTexas and SDSC. It's good to be back from the
conferenceand vacation. I have a conflict for the scheduled
Tuesday meeting, but
I'll see if I can break away. Just in case I can't make it, let
me try
to clear away some debris.
There are 3 main issues to deal with
1) Data Access
I appreciate your concern regarding the potential server impact
resulting from use of our web services (particularly in light of the
NWIS problems). Accessing the ASOS stations within ISD for the last
days data through our current web services shouldn't produce an undue
impact on our servers. I have made some preliminary tests which bare
this out. As I mentioned in our meeting last week, NCDC has begun a
major effort to assemble an element inventory (data catalog) for all
datasets. The initial inventory will be from the COOP (NWS
cooperativeobserver network) dataset; available early 2007. An
inventory/catalogfor ISD will follow that effort. There are 3
scenarios that present
themselves.
a) The existing web services point to a dataset (global hourly)
which is
updated daily with the last days observations. You all could use
theseweb services to extract current data and build a
catalog/inventory of
observations. Basic station observations of temperature, wind, dew
point, pressure will be recorded and observations like precipitation
will be present when observed. Among the additional station
observations, there will be a good deal of variance day to day,
stationto station. This would give you a current archive, and the
finalmodifications will be done well before the 10/30 target. A
historicalrecord could be built from our planned web service
inventory/catalogavailable next year.
b) You all could wait for us to build element inventories (catalog)
fordatasets of interest and then use web services to those tables
to update
information. This would probably take around 6 months.
c) You could access the archived data through files on our ftp server,
create your own catalog/inventory. Then plug into our web services
catalog/inventory for updates.
During last weeks meeting, I discussed our current global hourly data
inventory from which I can implement a web service for access to an
inventory of the number of observations for each station by year and
month. This would take a day to put together. Note that this doesn't
make a distinction of what was observed only that something was
observed.
2) NIDIS
As I mentioned last week, NCDC is heavily involved in the NIDIS effort
(National Integrated Drought Information System), a collaborative
effortof serveral agencies and institutions. The scope and
requirements for
the NIDIS portal are still being sorted out so in that regard... What
requirements might CUAHSI find useful in a drought portal? What web
service products might NCDC be able to consume from CUAHSI? At this
point, I don't believe the pieces are in place for something like
watershed flux analysis, but this and similar products would be
the goals.
3) NetCDF
The resource links you sent were very useful; I have already begun
on a
mock up for implementation. At our meeting last week, you were
satisfied w/ the time line which I laid out concerning netcdf
development/implementation (post 10/30). If that has changed, please
let me know.
Take care, Rich
PS. My laptop which has the slides from my presentation is getting a
s/w and h/w update (potential exploding battery), I'll send the
slides ASAP.
David Maidment wrote:
Rich:
Thanks so much for spending time with us on Thursday at SDSC. I
appreciate you presenting
to us the work that you've done with web services and I
especially appreciate that you have tried
to make them CUAHSI style. As you know, I have asked Cedric
David to work with you to test your
services and keep me informed as to which databases they provide
access to and how we can best
use what you have created. Could you please send me a copy of
the slides that you used last Thursday.
I want to study the nomenclature that you employed since its
reflective of how things are called at NCDC
and I need to understand that.
Cedric -- what we need to do is to use these services to create
an observations catalog for each
of the NCDC databases that NCDC is exposing using these services.
I am not sure if this should
be done at Texas or at SDSC.
Ilya -- what do you think about the above?
Rich --- the website with the tutorial on how to access the
Unidata NetCDFServer is at:
http://www.crwr.utexas.edu/gis/gishydro06/SpaceAndTime/NetCDF/Animating%20netCDF%20Data%20in%20ArcMap.htm>
The actual server itself is at:
http://motherlode.ucar.edu:8080/thredds/idd/nam_model.html>
This is really hard to find if you just search the Unidata web site.
I'd like to discuss with you and your colleagues how best to
develop a "weather and climate server"
that uses time series web services and also netCDF services.
Perhaps that is something we can
discuss with Sharon LeDuc.
David
H
