Glenn:
OK, good to get this cleared up. I am glad this is not some inherent issue
with OpenDAP,
which I had earlier suspected it might be.
David
________________________________
From: Glenn.Rutledge [mailto:Glenn.Rutledge@xxxxxxxx]
Sent: Thu 9/7/2006 9:01 AM
To: David Maidment
Cc: John Caron; support-thredds; Ilya Zaslavsky
Subject: Re: NCDC Web Services
Dave-
Yes- you are indeed correct. SOme organizations (NSF I guess), require a
standard pot 80 and block all others unless special arrangement have been made.
Because your's is not the first complaint (maybe the 3rd), NOMADS is going to
set up a Tomcat/Apache interface where we will present standard port to users.
We had delayed that because we are doing a complete overhaul of our front end
(to a load balanced arrangement). Anyway- the hardware got delayed and so to
did the action to standardize that access for you. So- you should have access
within several weeks..
Sorry for the hassle.... Glenn
David Maidment wrote the following on 9/6/2006 8:56 AM:
Glenn:
I think that what the information below means from John Caron is that
OpenDAP
services need to be accessible through Port 80 even if that is not the
default
with the Tomcat server or otherwise access to them will be blocked by
firewalls
in some instances. Can you check to see whether in publishing NARR
data
access through OpenDAP you have used Port 80 or not. This might be why
we and others are having trouble with firewalls when accessing NARR
from within
some organizations, but not with others.
David
________________________________
From: John Caron [mailto:caron@xxxxxxxxxxxxxxxx]
Sent: Sun 9/3/2006 7:25 PM
To: David Maidment
Cc: Glenn.Rutledge; support-thredds; Ilya Zaslavsky
Subject: Re: NCDC Web Services
David Maidment wrote:
John:
I feel like Dr Watson when Sherlock Holmes says "Elementary my
dear Watson".
I am sure what you are telling me is elementary but it requires
more knowledge of
server technology than I possess.
First of all -- what is a port? How is a port defined or
established? By the server? If
so how? Can a server have multiple ports? Why would you
want to do that?
A port is just an abstraction to establish various communication
channels to the same host machine. I mention it because thats generally how
firewalls work.
Is Tomcat a server technology built with open source
components?
Yes, its entirely open source.
Is it the java equivalent of a Windows based server?
Its a jaav-based HTTP server.
What is the significance of the default port on Tomcat being
8080 and on Windows (?) servers 80?
Tomcat can run under port 80 if you want. Again, its only relevent
because standard firewall policy is to allow HTTP traffic on port 80. Often
other ports are not allowed. This doesnt particularly solve security problems,
but its what people do anyway.
Are SOAP/XML services similarly restricted by port numbers? It
seems like our
SOAP/XML services are not similarly restricted by firewalls as
what we've done with
OpenDAP, but that may be because we are mediating them through
SDSC that may
be using Port 80.
Most servers can use any port you tell them. But they have to use a
specific port.
Ah -- the glories of computer technology!
SOrry i cant give you a simpler answer.
David
________________________________
From: John Caron [mailto:caron@xxxxxxxxxxxxxxxx]
Sent: Wed 8/23/2006 5:12 PM
To: David Maidment
Cc: Glenn.Rutledge; 'support-thredds'
Subject: Re: NCDC Web Services
Im not sure what the issue is, but its not really a server
thing. Firewalls live on the client. Generally HTTP traffic on port 80 is let
through, and not much else. Tomcat default port is 8080, it can be changed to
port 80 is need be, but you cant expect THREDDS servers to do that for the
convenience of some site that has some restrictive firewall.
A proxy server can be a good solution, but i dont know what the
details of use are.
David Maidment wrote:
Glenn:
Yes, please find out what we have to do. This looks
like a significant
issue to me.
John -- could you look at the email trail below and
comment on the
issues of OpenDAP and
firewalls. Is there any difference if we use your
netCDF server?
David
------------------------------------------------------------------------
*From:* Glenn.Rutledge [mailto:Glenn.Rutledge@xxxxxxxx]
*Sent:* Wednesday, August 23, 2006 10:09 AM
*To:* David Maidment
*Subject:* Re: NCDC Web Services
Systems folks would do this...and as I understand it-
is not at all
uncommon. It's a representation of a service within a
firewall I am
out of my realm but I do plan to investigate further
since NASA had the
exact comment you did just last week.
I'll let you know more if you like. GR
David Maidment wrote the following on 8/23/2006 10:54
AM:
Glenn:
What is a proxy server? How would I or any
other regular user know
how to use one of these?
David
------------------------------------------------------------------------
*From:* Glenn.Rutledge
[mailto:Glenn.Rutledge@xxxxxxxx]
*Sent:* Tuesday, August 22, 2006 12:53 PM
*To:* David Maidment
*Subject:* Re: NCDC Web Services
Hi David-
A response from OPeNDAP:
our code (libdap, libnc-dap, our clients - with
the possible exception
of the ODC) can all be told to use a proxy
server. Open ports to that
machine and you're off and running. If someone
can use the Internet,
they can use our clients and read from your (or
any) server. Point
these folks toward the documentation on the
.dodsrc file. If they
still have questions (and I know some of our
docs are getting stale),
then have them contact support@xxxxxxxxxxxxxxx
with their questions
(or the newly renamed opendap or opendap-tech
emails lists - see
www.opendap.org <http://www.opendap.org/> and
look under support for information about those).
I'm glad you asked me, because this is an
important feature of the
client-side software!
Hope this helps--- Glenn
David Maidment wrote the following on 8/21/2006
9:13 PM:
Glenn:
I have some concerns about how useful
OpenDAP is. It seems sensitive
to being blocked by firewalls.
Our direct calls to NARR that relied on
using OpenDAP worked fine
>from here at UT but from inside of
NSF with all their firewalls in place,
the data were blocked, and the
same is true when trying to do this
>from within ESRI.
I think the question of how to do these
things is linked up with
computer security and that needs to be
factored into the debate.
David
------------------------------------------------------------------------
*From:* Glenn.Rutledge
[mailto:Glenn.Rutledge@xxxxxxxx]
*Sent:* Monday, August 21, 2006 9:13 AM
*To:* David Maidment
*Subject:* Re: NCDC Web Services
Thanks David-
I agree- but somewhat concerned whether
SOAP/REST will have
capabilities across the other data
forms you cite - although based on
your individual needs- it doesn't seem
to matter. But for the larger
whole- we need to test this out. Glenn
David Maidment wrote the following on
8/21/2006 9:47 AM:
Glenn:
I see observations data as time series
as one kind of service, and weather
model, remote sensing and Nexrad data
as another. For the weather and
climate "field" information, some
variant of what John Caron is doing seems
appropriate. For the observations
data, I think we have a fairly good scheme
figured out in the work we are doing
with CUAHSI. I can see the netCDF
style of things for the weather and
climate fields. I think the time series stuff
could be handled that way but is easier
and perhaps more appropriate just to
do it in XML via SOAP or REST.
What I would like to see developed is a
"Weather and Climate Server" so that
the data users can go to one system and
get into the various information sources
without having to go separately to
NCDC, Unidata, NCAR, NWS, etc. This is
like WaterOneFlow that we are
attempting to do in CUAHSI for water observation
data.
David
________________________________
From: Glenn.Rutledge@xxxxxxxx
[mailto:Glenn.Rutledge@xxxxxxxx]
Sent: Fri 8/18/2006 6:39 PM
To: David Maidment
Subject: Re: RE: NCDC Web Services
Hi David,
I agree. Here's the note I sent Sharon
earlier this week after your
request. I want to plan for the long
term and work toward the benefit
of all. We are meeting on Monday.
Don't worry about the lack of the
attachment I reference for the moment.
Regards, Glenn
__________________________________
Sharon et al,
I think we need to have a set of
meetings to define our plan for
exposing NCDC services. I'm not aware
of the status of this kind of
activity already, but there are several
efforts underway within the
Center and we're already on a similar
track - we just need to solidify
our goals- and maybe generate a
requirements document from which to work
toward. btw- NOAA's GEO-IDE would
benefit from just such an activity.
Rich's efforts in OGC compliant (GIS)
services are the way to go, and
NOMADS team have implemented a THREDDS
Data Server (TDS) for OGC WCS
services and provides for OPeDNAP
enabled requests handled under TDS.
Of course SteveD's group has advanced
even higher levels of standards
using SOAP for more generalized
services descriptions. There are
limitations to generalities with such
standards, and not community based
standards such as OPeNDAP at the
present time.
I'm co-PI on a NASA, Unidata, OPeNDAP
and GMU effort called ACCESS that
has defined and are building just such
a "Gateway" using NetCDF files
initially- using the COARDS and CF
compliant OGC Catalog Services for
the Web (CSW). This cross- community
effort has many players including
OGC, CEOS, GALEON, GO-ESSP and many
others. It builds on the work of
the Common Data Model and TDS work at
Unidata and OGC and International
services. There is work to do on my
end: Grib/grib2/BUFR issues but
we've already made significant headway
in this regard and participated
in the last GEO demo last May in
Beijing. NCDC will be the benefactor of
this effort and RSAD will be installing
this capability (in test mode)
when completed later this year.
The attached could be a staring point
for these Center-wide discussions,
and I suggest that the attached be
reviewed and considered for expansion
into the center when fully developed.
The reason that I'm co-PI on the
effort is that NOMADS would install and
provide this service for models.
It applies to other datasets as well
but would need to be adapted for
Neal's/Steve's data. Comments? Glenn
What David Maidmont seeks of course is
a unified NCDC web presence to
all of NCDC's resources.
----- Original Message -----
From: David Maidment
<maidment@xxxxxxxxxxxxxxx> <mailto:maidment@xxxxxxxxxxxxxxx>
Date: Friday, August 18, 2006 7:28 pm
Subject: RE: NCDC Web Services
Rich:
There are a lot of questions here that
require some reflection.
What I want to do
in the short term is stand up a web
service to NCDC that is as
simple as possible
that shows that we have a serious
commitment to work together.
That is why CRN
is attractive to me -- its high quality
data thus good for
research, its not very many
stations, so not too difficult to build
the catalog even if we do
it station by station
using the web services that you created
(do they access CRN?), and
hopefullyyour services will work onto
the archive reasonably well.
I am trying to create a path that
points the way to the future in a
methodicalmanner that step by step we
figure out what comes next
and implement it
and we start with some fairly
straightforward, confidence building
steps that
solidify the relationships which
already exist on a technical level
betweenyou and your colleagues and us.
We need to learn how you
think about
data at NCDC. We need to understand
what databases you maintain, how
many services you actually offer and
which ones we should focus on
first, which
ones leave to later.
Our first priority is access to
historical NCDC data archives not
near real-time
data.
I want to understand what datasets you
are publishing in netCDF-
accessibleform and how those dovetail
with what is available from
NCAR and Unidata.
I don't know enough about NIDIS yet to
comment on it in an informed
way.
Thanks for your feedback.
David
________________________________
From: Rich.Baldwin
[mailto:Rich.Baldwin@xxxxxxxx]
Sent: Fri 8/18/2006 8:32 AM
To: David Maidment
Cc: Cedric David; Ilya Zaslavsky; Glenn
Rutledge; Sharon Le Duc;
Neal Lott
Subject: Re: NCDC Web Services
Hello David and Ilya,
I was glad to meet you all last week.
I had a productive meeting with
your teams at UTexas and SDSC. It's
good to be back from the
conferenceand vacation. I have a
conflict for the scheduled
Tuesday meeting, but
I'll see if I can break away. Just in
case I can't make it, let
me try
to clear away some debris.
There are 3 main issues to deal with
1) Data Access
I appreciate your concern regarding the
potential server impact
resulting from use of our web services
(particularly in light of the
NWIS problems). Accessing the ASOS
stations within ISD for the last
days data through our current web
services shouldn't produce an undue
impact on our servers. I have made
some preliminary tests which bare
this out. As I mentioned in our
meeting last week, NCDC has begun a
major effort to assemble an element
inventory (data catalog) for all
datasets. The initial inventory will
be from the COOP (NWS
cooperativeobserver network) dataset;
available early 2007. An
inventory/catalogfor ISD will follow
that effort. There are 3
scenarios that present
themselves.
a) The existing web services point to a
dataset (global hourly)
which is
updated daily with the last days
observations. You all could use
theseweb services to extract current
data and build a
catalog/inventory of
observations. Basic station
observations of temperature, wind, dew
point, pressure will be recorded and
observations like precipitation
will be present when observed. Among
the additional station
observations, there will be a good deal
of variance day to day,
stationto station. This would give you
a current archive, and the
finalmodifications will be done well
before the 10/30 target. A
historicalrecord could be built from
our planned web service
inventory/catalogavailable next year.
b) You all could wait for us to build
element inventories (catalog)
fordatasets of interest and then use
web services to those tables
to update
information. This would probably take
around 6 months.
c) You could access the archived data
through files on our ftp server,
create your own catalog/inventory.
Then plug into our web services
catalog/inventory for updates.
During last weeks meeting, I discussed
our current global hourly data
inventory from which I can implement a
web service for access to an
inventory of the number of observations
for each station by year and
month. This would take a day to put
together. Note that this doesn't
make a distinction of what was observed
only that something was
observed.
2) NIDIS
As I mentioned last week, NCDC is
heavily involved in the NIDIS effort
(National Integrated Drought
Information System), a collaborative
effortof serveral agencies and
institutions. The scope and
requirements for
the NIDIS portal are still being sorted
out so in that regard... What
requirements might CUAHSI find useful
in a drought portal? What web
service products might NCDC be able to
consume from CUAHSI? At this
point, I don't believe the pieces are
in place for something like
watershed flux analysis, but this and
similar products would be
the goals.
3) NetCDF
The resource links you sent were very
useful; I have already begun
on a
mock up for implementation. At our
meeting last week, you were
satisfied w/ the time line which I laid
out concerning netcdf
development/implementation (post
10/30). If that has changed, please
let me know.
Take care, Rich
PS. My laptop which has the slides
from my presentation is getting a
s/w and h/w update (potential exploding
battery), I'll send the
slides ASAP.
David Maidment wrote:
Rich:
Thanks so much for spending time with
us on Thursday at SDSC. I
appreciate you presenting
to us the work that you've done with
web services and I
especially appreciate that you have
tried
to make them CUAHSI style. As you
know, I have asked Cedric
David to work with you to test your
services and keep me informed as to
which databases they provide
access to and how we can best
use what you have created. Could you
please send me a copy of
the slides that you used last Thursday.
I want to study the nomenclature that
you employed since its
reflective of how things are called at
NCDC
and I need to understand that.
Cedric -- what we need to do is to use
these services to create
an observations catalog for each
of the NCDC databases that NCDC is
exposing using these services.
I am not sure if this should
be done at Texas or at SDSC.
Ilya -- what do you think about the
above?
Rich --- the website with the tutorial
on how to access the
Unidata NetCDFServer is at:
http://www.crwr.utexas.edu/gis/gishydro06/SpaceAndTime/NetCDF/Animating%20netCDF%20Data%20in%20ArcMap.htm>
The actual server itself is at:
http://motherlode.ucar.edu:8080/thredds/idd/nam_model.html>
This is really hard to find if you just
search the Unidata web site.
I'd like to discuss with you and your
colleagues how best to
develop a "weather and climate server"
that uses time series web services and
also netCDF services.
Perhaps that is something we can
discuss with Sharon LeDuc.
David
H
--
Glenn K. Rutledge
Services Team Leader
Remote Sensing and Applications Division
NOMADS Project Manager
National Oceanic and Atmospheric
Administration
National Climatic Data Center
Asheville NC 28801
Phone: (828) 271-4097
Fax: (828) 271-4328
NOMADS: http://nomads.ncdc.noaa.gov/
--
Glenn K. Rutledge
Services Team Leader
Remote Sensing and Applications Division
NOMADS Project Manager
National Oceanic and Atmospheric Administration
National Climatic Data Center
Asheville NC 28801
Phone: (828) 271-4097
Fax: (828) 271-4328
NOMADS: http://nomads.ncdc.noaa.gov/
--
Glenn K. Rutledge
Services Team Leader
Remote Sensing and Applications Division
NOMADS Project Manager
National Oceanic and Atmospheric Administration
National Climatic Data Center
Asheville NC 28801
Phone: (828) 271-4097
Fax: (828) 271-4328
NOMADS: http://nomads.ncdc.noaa.gov/
--
Glenn K. Rutledge
Services Team Leader
Remote Sensing and Applications Division
NOMADS Project Manager
National Oceanic and Atmospheric Administration
National Climatic Data Center
Asheville NC 28801
Phone: (828) 271-4097
Fax: (828) 271-4328
NOMADS: http://nomads.ncdc.noaa.gov/
