NOTE: The galeon mailing list is no longer active. The list archives are made available for historical reasons.
Hi All,This is a really excellent discussion. At the end of it we'd make real progress towards a shared sense of priorities if we could agree upon a summary of it in bullet form.
At the heart of the discussion is the blunt question that Ben raised: "abandon the CF-netCDF encoding spec. and ONLY be proposing CF-OPeNDAP"? I'm not sure how to answer that directly. But if we alter the question to be, "*_Should we consider the fusion of OPeNDAP and WCS as a fundamental objective of GALEON?_*", I would say emphatically YES!
Differences of opinion are so often due to different interpretations of terminology and fundamental assumptions that have not been exposed. Below I will try to spell out my own assumptions (presumably fairly similar to Jon Blower's and Roy Mendelssohn's) in hopes of allowing others to pin point where we may be starting from terminology confusions or differing assumptions.
Terminology:
1. "netCDF-CF": we often refer to netCDF-CF in a manner so loose
that it becomes a source of confusion. NetCDF-CF is a standard
for data on complex grids -- curvilinear in XY; sigma and
density-related in Z; climatological and artificial calendars in
T; and heading towards "tile mosaics" and 5D forecast ensembles in
the near future. (The current ArcGIS release, as I understand it,
does not yet even fully support the vastly simpler COARDS
specification. There is a chasm separating ArcGIS from netCDF-CF
datasets.)
2. "OPeNDAP": for purposes of this discussion I would argue that
OPeNDAP is simply an approach that allows applications to access
remote, time-aggregated collections of netCDF-CF files (virtual
datasets -- often terabyte sized) through the unaltered netCDF API
-- as if they were local netCDF files. (There are opportunities
for endless interesting technical discussions about protocols,
mime-types, streaming, etc., but I think they miss the simpler
essence needed for this discussion.)
Assumptions:
1. The fundamental goal of GALEON is to promote interoperability.
There are two complementary forms of interoperability that must be
considered:
1. "inter-community" (between the fluid earth science community
and others) -- characterized by making complex fluid earth
science data accessible to users with a very broad range of
outlooks and levels of (mathematical) sophistication
2. "intra-community" (within the fluid earth science community)
-- characterized by exchanges between users of high
mathematical sophistication; data on complex grids (see
"netCDF-CF" definition, above); the need for high numerical
fidelity; and the need to "slice" the data arbitrarily along
all principal planes and axes in 4D space -- e.g. to extract
a depth-time slice from a 4D dataset
2. The current intention of the GALEON project is to incorporate the
full semantic richness of netCDF-CF into the WCS framework. (If
not, then what level of CF semantics is targeted by GALEON? and
in order to serve what classes of users?)
3. In order to succeed at achieving interoperability the applications
that users rely upon (in their distinct user communities) must be
able to request and read data and information in the proposed
standards and protocols
From T1 I conclude that many (most) OGC-friendly applications in use
outside of fluid earth science will never support the complexity of
netCDF-CF datasets -- even applications that might benefit from
occasional access to native fluid earth science data. Working with full
blown netCDF-CF datasets is not easy. To succeed requires the
involvement of programmers with mathematical skills similar to what we
find in the fluid earth science community. (Imagine an application used
by farmers planning crop rotations of their fields -- unlikely to be
able to handle curvilinear model output climate forecast data in
netCDF-CF, though there would be potential uses for this data. Or an
application for yachtsman to perform on-board navigation -- same deal.)
The importance of getting the WCS-OPeNDAP fusion into our conversation is that we need to unify our community in order to promote interoperability. We need to break down the barriers that currently separate "camps": WCS versus OPeNDAP.
- Steve ===================================================== stefano nativi wrote:
Dear Jon, Thank you for the valuable and stimulating comments.I am in serious agreement with several of the points you touched. I particularly agree that "more clarity is needed about the use cases that WCS is thought to satisfy". As matter of fact, Ben and I are working on a position paper with the temporary title: "Which coverage access services do we really need ?".For me, the Web Coverage Service(s) is/are important for enabling cross-disciplinary interoperability.IMHO, this extension profile draft contributes to clarify this question. In fact, mapping CF-NetCDF to WCS data model should provide a useful indication of the complexity and rewards that we are going to face. In fact, this exercise was done to support interoperability and cross-disciplinary research/applications. This is important to enable the CF-netCDF use in multi-disciplinary contexts.From my perspective, WCS 1.1.x is a rather complex specification; it will be replaced by WCS 1.2 which is trying to solve the complexity problems. Thus, this extension specification must be considered as propedeutic for the WCS 1.2 CF-netCDF extension.I will try to provide more discussion points inline.I have always regarded the OGC standards as a way in which the met-ocean community can communicate its data to other communities. Personally I do not look to OGC standards for providing methods for exchanging data within our community. We already have well-established tools, standards, protocols, data models and software for doing this. When communicating outside ones own community, one often has to accept that a certain amount of information will be lost. I wonder if it is realistic to expect that we might use WCS in future to communicate our data in all its subtlety?Due to multiple Community specifications and conventions, I think that a certain amount of information is often lost even when communicating inside ones own community. In my opinion, the real question is about the interoperability level required by applications to run.For example, GEOSS is working on cross-disciplinary applications for serving many SBAs (Societal Benefit Areas); in this case, what is the exact meaning of "communicating outside ones Community"?My other concern is that the WCS world is changing extremely rapidly, with at least four versions of the specification in existence (1.0, 1.1, 1.2 and "plus"). This contrasts with the relatively small numbers of real WCS servers "out there" in the wild (GALEON has done a great job in encouraging people to stand up real systems but in general, uptake of WCS is rather low). The ISO19123 Coverage model is also likely to be revised as it is broken in places. Can we keep up with this evolution?I completely share your concerns. I made similar comments to the WCS RWG.In my opinion, ISO 19123 Coverage model is best instrument we have now to confront the different "disciplinary coverage models" and design interoperability specifications for coverages.Thirdly, the WCS1.2 "core plus extensions" model worries me a bit. I understand that the "core" is small, implying that the different community "extensions" will have little interoperability with each other. Effectively, we'll end up with a lot of mutually-incompatible versions of WCS that share some extremely limited features (perhaps only their terminology). The "core plus extensions" model implies that WCS has a desire to encompass all data, a scope that is arguably too wide to be useful or realistic.You raised an important issue that must be carefully addressed by the SWG for WCS 1.2.Actually, the WCS 1.2 "core" specification use case is very close to the one you describe in your following comment (i.e. a WCS specification considerably simplified, with a defined list of things that it's good for and things that it isn't good for).The question Ben and I are posing in our position paper, is about how to decide the "list of things that it's good for". In fact, that means to decide how "general purpose" the service will be to the detriment of its effectiveness (i.e. semantic meaningfulness). The latter was one of the main drawback outlined for the WCS 1.0.Personally I would like to see the WCS specification considerably simplified, with a defined list of things that it's good for and things that it isn't good for. For the sake of argument, let's imagine a WCS that only serves data as GeoTIFFs in a lat-lon coordinate reference system (or lat-lon-elevation-time in 4D). Sure, a lot of information would be lost across this interface (e.g. the original data's grid) and certain specialist use cases could not be satisfied (e.g. calculating heat transports). However, it would be much simpler to implement (client and server) and would still satisfy a large number of use cases in which met-ocean data is needed outside our community.This is very close to an enhanced WMS; in my opinion you presented a valuable use case. However, the coverage concept and its types are quite different from the map concept and its types. Thus, for the sake of cross-disciplinary interoperability we must still wonder whether this is the only service we need, or if we need "proper" coverage access service(s), too.I guess I'm saying that I don't think it's realistic or desirable to develop a WCS specification to serve all kinds of geographic data, which is the way things seem to be headed in the OGC world. The effort required to support a small number of specialist use cases (in the specification, in servers and in clients) is fearsome. I think WCS should aim to complement, not replace, technologies such as OPeNDAP by satisfying a different class of user. I think more clarity is needed about the use cases that WCS is thought to satisfy.In my opinion, to provide access to CF-netCDF data model and encoding formats through "non-traditional" protocols should be a valuable asset for the FES community.Best regards, Stefano
galeon archives: