[galeon] WCS-netCDF and OpenDAP

NOTE: The galeon mailing list is no longer active. The list archives are made available for historical reasons.

Thread renamed per Ben's suggestion.

Aaron

Steve Hankin wrote:



Aaron Braeckel wrote:

[...snip...]

Steve Hankin wrote:

My apologies, but I cannot see the logic of this argument.  NetCDF and
OPeNDAP in fact already are very closely coupled and have been so for
many years.  Basically (only a slight over-simplification) any
application that can read a netCDF file can read the same data from a
remote host through OPeNDAP (with a relinking of the code at most). Bringing netCDF and OPeNDAP together (under a WCS umbrella or 
elsewhere) does not mandate that both are used by any given
community.  If you have reasons to want to avoid the use of OPeNDAP in
your community you can achieve that goal by simply ignoring it.

I think it is worth examining the deeper issues of where data
interoperability barriers come from.  It is that we have too many,
narrowly defined standards that cannot talk to one another --
netCDF-CF, HDF-EOS, geoTIFF, shapefiles,  tab-delimited format
description or what have you.   Community stovepipes.  The heart and
sole of the interoperability effort has to be doing the work necessary
to make standards inter-operate.

The "core plus extensions" philosophy of WCS is fundamentally weak, in
that it solves only the problem of metadata interoperability, data
interoperability remains a collection of stovepipes.  Your concerns
above reflect an inevitable consequence that this weakness of WCS has
on clients -- client software must become more complex in order to
address interoperability.  Unfortunately this aspect of WCS is a given
-- it is out of our scope to change it.

The desire and need to read a NetCDF file does not imply interest in
using the OpenDAP protocol.  While in practice they have frequently been
used together, *reading a NetCDF file is different than making an OpenDAP
request*.

Hi Aaron,

A good opportunity for conversation on a Friday afternoon ....
There is something fundamentally confused in our communications here. Wires are crossed somewhere. Every application I am aware of that 
reads netCDF files does so through the netCDF API(s) as defined and
supported by Unidata.  No doubt there are exceptions to this (say, in
high performance applications), but my assumption is that you are
using the standard netCDF API in your applications.  If so, the only
difference your application would see between reading a local netCDF
file and the same file  from a remote server via OPeNDAP is the name
of the file:  for OPeNDAP the file path will begin with "http://...";;
for a local file access it will not.   In fact, if your code is
written in Java you already have this capability built into your
application with no modifications at all (no need to relink -- ready
to test it out).


I can tell you unequivocally that I will be reading NetCDF
files from a WCS server, but I will not be making OpenDAP requests.

If I may rephrase your sentence a tad:  you will be _downloading _a
netCDF file via a WCS service.  I suspect that your application will
then _read_ that file using the NetCDF API.   If so that means that
your application would hardly know the difference if it were reading
the same file hosted remotely via OPeNDAP, instead.

My argument is primarily against lumping the specification of a protocol
together with the specification of a file format.  While in *practice*
there is much common use, I don't think they belong together.  In
practice there will be much overlap, but when it comes to specifications
I would prefer to maintain a separation of concerns with regards to
documentation, libraries, dependencies, etc.

Why
make a specification that joins both?  OpenDAP is a protocol stack on
top of NetCDF.  I could see building an OpenDAP extension profile on top
of the NetCDF one, but to me the two are logically separate things. 
The effort that you are describing ("a specification that joins both")
was already accomplished 10 years ago and has been running as an
operational capability since then (with an occasional hiccup, of
course, like any system) and has an active community supporting it.


There are a host of NetCDF utilities that can read NetCDF files but
cannot deal with OpenDAP URLs.

Where this is true, it is a deliberate choice on the part of the
application developer to avoid the easily accessible OPeNDAP
capabilities.  In some applications the ability to access data across
the network is irrelevant.   (Rarely would this be the case for a
WCS-enabled application, but I would still defend your right to
exclude OPeNDAP if you so choose. ;-) )

[aside note:  In point of fact relinking c or FORTRAN source code with
OPeNDAP is less straightforward than one would like it to be, because
the relevant OPeNDAP libraries are written in c++, which combines
awkwardly with c and FORTRAN code.  However, a development effort is
already well underway to rewrite the OPeNDAP libraries in pure c. A
relatively minor issue in the big scheme of things, anyway ....]

I can see both sides of this argument to some degree, especially if one
is trying to minimize the work of standardization.  Perhaps it is from
my experience with the OGC standards process, but overall I do prefer to
segregate logically separate entities when generating specifications
rather than generating a bigger document and telling people to ignore
the parts that don't apply to them.

One of several fundamental confusions that occur when people compare
OPeNDAP and OGC solutions is that they are comparing apples and
oranges -- a running system (with the limitations implicit therein)
compared to a standards process (with the seemingly limitless
possibilities implicit therein).   Given the "core+extensions" nature
of WCS,  OPeNDAP is definitely "low hanging fruit" in the GALEON
project.  It comes almost for free with NetCDF -- like the icing on a
cake.  You can scrape the icing off if you have a particular dislike
of icing.  But it is hard to make a compelling argument that everyone
else ought to make the same choice (unless you argue from idealogical
grounds).

I am not arguing that OpenDAP should not be a part of the OGC
standardization process, only that they should be specified or
standardized separately.  At risk of repeating myself, there is little
burden to keep the specifications/encoding profiles separate and I think
it keeps two logically distinct entities properly separate.  Perhaps a
good test to push one way or the other would be whether the file format
and the protocol change completely in sync when new versions come out. When NetCDF 4 came about, were any relevant changes to OpenDAP released 
at the same time in the same specification/document?  How about vice
versa with any recent changes to OpenDAP?  Are they one specification,
or two linked specifications?

I am not an expert on OpenDAP so my search was likely not exhaustive,
but I just looked at both the NetCDF tutorial and the NetCDF user's
guide and I see three minor mentions of OpenDAP in references and
statements such as "OpenDAP support added in version 2.0...".  I did not
see any description of OpenDAP itself, how it fits in, or any real
indication that two specifications (NetCDF and OpenDAP) are linked in a
formal way.  Is there a joint specification I am missing?
  • 2008 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the galeon archives: