NOTE: The galeon mailing list is no longer active. The list archives are made available for historical reasons.
How so? To write a parser for a binary file one needs to know the detailed structure of the file - and if that changes your code is now broken. With GML or any XML-based encoding the file structure is not known to your application. Where do you see the problems in parsing and understanding GML? R -----Original Message----- From: Gerry Creager [mailto:gerry.creager@xxxxxxxx] Sent: August 21, 2009 1:17 PM To: Ron Lake Cc: John Caron; Unidata GALEON; wcs-2.0.swg Subject: Re: [galeon] [WCS-2.0.swg] CF-netCDF standards initiatives I'm not so sure that's true. Ron Lake wrote: > Hi John: > > Surely the GML encoding is going to be simpler to parse and "understand" than > any equivalent binary encoding. > > R > > -----Original Message----- > From: galeon-bounces@xxxxxxxxxxxxxxxx > [mailto:galeon-bounces@xxxxxxxxxxxxxxxx] On Behalf Of John Caron > Sent: August 21, 2009 12:52 PM > Cc: Unidata GALEON; wcs-2.0.swg > Subject: Re: [galeon] [WCS-2.0.swg] CF-netCDF standards initiatives > > A few thoughts on these issues: > > The WCS/SWE/OGC are making major contributions in elaborating conceptual > models and defining remote access protocols based on them. More > problematic is managing complexity and adoption paths of existing > communities. One of the main weaknesses of WCS is the likely > proliferation of binary encoding formats. WMS has been lucky to be able > to use existing, universal formats like JPEG, and partly because of > that, has been more widely adopted. > > A central issue is the relation of the WCS data encoding format to its > GML envelope. Should the data be "just bits" and let the GML encode all > the semantics, or should a binary format also encode some/all of the > semantics? In the first case, any client must understand GML, which is a > big barrier to use by existing applications, plus the possibility exists > that some essential information is offline or unavailable, as Tom W > points out. In the second case, we have a proliferation of data encoding > formats and the possibility that the semantics are encoded differently > between the GML and the data. > > From Unidata's POV, namely that of supporting a large community of > users of netcdf and other scientific data formats, using netCDF-CF to > encode the data is obvious. However, this is not just a matter of one > more group advocating for their favorite bespoke format. IMHO, netCDF is > close to being a "maximally simple implementation" of a binary data > encoding that allows data to be more than "just numbers". Its BNF > grammer fits on one side of a page of paper. If one chooses to encode > semantics into the data format, and not require a client to know GML, > then you need something like netCDF, and you wont find an encoding > significantly simpler. The CF Conventions group has been slowly adding > the necessary semantics for 10 years or so. From that motivation we > decided to recommend netCDF as a WCS encoding format. > > The hard part is describing in a formal way the relationship of the > netCDF-CF binary response to the actual request and to any GML part of > the response. Stefano et al are doing the heroic work of describing the > mapping between the two models, but there is and perhaps should be a > loose coupling between the two. There are always edge cases: what > happens when the requested lat/lon bounding box crosses a discontinuity > of the projection plane? One might get different, but reasonable, files > back from different servers of the same dataset. So what can we > standardize? None of this is unique to netcdf-CF or binary encodings. > > Taking the first path of letting GML carry all the semantics has many > advantages also, especially for new development and GIS-centric clients, > and as new libraries are developed that can mitigate some of the > complexity of GML. In (our) scientific community, these are new > possibilities, to be experimented with and prototypes tested. Once there > are real services delivering "gotta have" data we will do whatever we > need to do to "get me some of that" ;^) > > John Caron > Unidata > > Steve Hankin wrote: >> >> Robin, Alexandre wrote: >>> Hi Steve, >>> >>> Just to clarify when I said NetCDF was a "NEW standard" I meant a new >>> standard in OGC. >>> >>> As I was telling Ben in an offline email, I am totally conscious of >>> its penetration and usefulness in certain communities. >>> >>> However, _I am not convinced that /having two standards doing the >>> same thing in OGC /is sending the right message and is the best way >>> to go for a standardization organization_. >>> >> Hi Robin, >> >> I confess that I was aware of using a cheap rhetorical device when I >> twisted your intended meaning of "NEW". (Begging your tolerance.) It >> was helpful in order to raise more fundamental questions. You have >> alluded to a key question just above. Is it really best to think of >> the target of OGC as a the development of a single, definitive >> standard? one that is more general and more powerful than all existing >> standards? Or is it better to think of OGC as a process, through which >> the forces of divergence in geospatial IT systems can be weakened >> leading towards convergence over time? The notion that there can be a >> single OGC solution is already patently an illusion. Which one would >> you pick? WFS? WCS? SOS with SWE Common? SOS with its many other XML >> schema? (Lets not even look into the profusion of WFS application >> schema.) I trust that we are not pinning our hopes on a future >> consolidation of all of these. There is little evidence to indicate >> that we can sustain the focus necessary to traverse that path. The >> underlying technology is not standing still. >> >> What Ben (and David Arctur and others) have proposed through seeking >> to put an OGC stamp of approval on netCDF-CF technology is similar to >> what OGC has achieved through putting its stamp on KML ("There are >> sound business and policy reasons for doing so.") It is to create a >> process -- a technical conversation if you will -- which will lead to >> interoperability pathways that bridge technologies and communities. >> Real-world interoperability. >>> There has been a lot of experimentation with SWE technologies as well >>> that you may not know about and in many communities, especially in >>> earth science. >>> >>> What I'm saying is that perhaps it is worth testing bridging NetCDF >>> to SWE before we go the way of stamping two 100% overlapping >>> standards as OGC compliant. >>> >> Complete agreement that this sort of testing ought to occur. And >> interest to hear more about what has been achieved. But great >> skepticism that there is this degree of overlap between the >> approaches. And disagreement that this testing ought to be a >> precondition to OGC recognition of a significant ,community-proven >> interoperability mechanism like netCDF. OGC standardization of netCDF >> will provide a forum for testing and experimentation to occur much >> more rapidly and for a 2-way transfer of the best ideas between >> approaches. NetCDF & co. (its API, data model, CF, DAP) have a great >> deal to offer to OGC. >> >> - Steve >>> Regards, >>> >>> ------------------------------------------------- >>> >>> **Alexandre Robin** >>> >>> Spot Image, Web and E-Business >>> >>> Tel: +33 (0)5 62 19 43 62 >>> >>> Fax: +33 (0)5 62 19 43 43 >>> >>> http://www.spotimage.com >>> >>> Before printing, think about the environment >>> >>> ------------------------------------------------------------------------ >>> >>> *De :* Steve Hankin [mailto:Steven.C.Hankin@xxxxxxxx] >>> *Envoyé :* jeudi 20 août 2009 20:58 >>> *À :* Tom Whittaker >>> *Cc :* Robin, Alexandre; Ben Domenico; Unidata GALEON; wcs-2.0.swg >>> *Objet :* Re: [galeon] [WCS-2.0.swg] CF-netCDF standards initiatives >>> >>> Hi Tom, >>> >>> I am grateful to you for opening the door to comments "from 10 >>> thousand feet" -- fundamental truths that we know from many years of >>> experience, but that we fear may be getting short shrift in >>> discussions of a new technology. I'd like to offer a comment of that >>> sort regarding the interplay of ideas today between Robin ("/I hope >>> we don't have to define a NEW standard .../") and Carl Reed ("/there >>> are other organizations interested in bringing legacy spatial >>> encodings into the OGC. There are sound business and policy reasons >>> for doing so./"). >>> >>> The NEW standard in this discussion is arguably SWE, rather than >>> netCDF. NetCDF has decades of practice behind it; huge bodies of data >>> based upon it; a wide range of applications capable of accessing it >>> (both locally and remotely); and communities that depend vitally upon >>> it. As Ben points out, netCDF also has its own de jure pedigree. >>> >>> A key peril shared by most IT standards committees -- a lesson that >>> has been learned, forgotten, relearned and forgotten again so many >>> times that it is clearly an issue of basic human behavior -- is that >>> they will try to innovate. Too-common committee behavior is to >>> propose, discuss and document new and intriguing technologies, and >>> then advance those documents through a de jure standards process, >>> despite an insufficient level of testing. The OGC testbed process >>> exists to address this, but we see continually how large the gap is >>> between the testbed process and the pace and complexity of >>> innovations emerging from committees. >>> >>> Excellent reading on this subject is the essay by Michi Henning, /The >>> Rise and Fall of CORBA/ (2006 -- >>> http://queue.acm.org/detail.cfm?id=1142044). Among the many insights >>> he offers is >>> >>> **'Standards consortia need iron-clad rules to ensure that they >>> standardize existing best practice.** There is no room for innovation >>> in standards_._ Throwing in "just that extra little feature" >>> inevitably causes unforeseen technical problems, despite the best >>> intentions.' >>> >>> While it adds weight to an argument to be able to quote from an >>> in-print source, this is a self-evident truth. We need only reflect >>> on the recent history of IT. What we need is to work together to find >>> ways to prevent ourselves from continually forgetting it. >>> >>> There is little question in my mind that putting an OGC stamp of >>> approval on netCDF is a win-win process -- for the met/ocean/climate >>> community and for the broader geospatial community. It will be a path >>> to greater interoperability in the long run and it deserves to go >>> forward. The merits of SWE (or GML) as an alternative approach to the >>> same functionality also deserve to be explored and tested in >>> situations of realistic complexity. But this exploration should be >>> understood initially as a process of R&D -- a required step before a >>> "standards process" is considered. If that exploration has already >>> been done it should be widely disseminated, discussed and evaluated. >>> >>> - Steve >>> >>> ================================== >>> >>> Tom Whittaker wrote: >>> >>> I may be ignorant about these issues, so please forgive me if I am >>> completely out-of-line....but when I looked at the examples, I got >>> very concerned since the metadata needed to interpret the data values >>> in the "data files" is apparently not actually in the file, but >>> somewhere else. We've been here before: One of the single biggest >>> mistakes that the meteorological community made in defining a >>> distribution format for realtime, streaming data was BUFR -- because >>> the "tables" needed to interpret the contents of the files are >>> somewhere else....and sometimes, end users cannot find them! >>> >>> NetCDF and ncML maintain the essential metadata within the files: >>> types, units, coordinates -- and I strongly urge you (or whomever) not >>> to make the "BUFR mistake" again -- put the metadata into the files! >>> Do not require the end user to have to have an internet connection to >>> simply "read" the data....many people download the files and then >>> "take them along" when traveling, for example. >>> >>> If I simply downloaded the file at >>> <http://schemas.opengis.net/om/1.0.0/examples/weatherObservation.xml> >>> I would not be able to read it. In fact, it looks like even if I also >>> got the "metadata" file at: >>> <http://schemas.opengis.net/om/1.0.0/examples/weatherRecord1.xml> >>> I would still not be able to read it, since it also refers to other >>> servers in the universe to obtain essential metadata. >>> >>> That is my 2 cents worth....and I hope I am wrong about what I saw in >>> the examples.... >>> >>> tom >>> >>> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> galeon mailing list >> galeon@xxxxxxxxxxxxxxxx >> For list information, to unsubscribe, visit: >> http://www.unidata.ucar.edu/mailing_lists/ > > _______________________________________________ > galeon mailing list > galeon@xxxxxxxxxxxxxxxx > For list information, to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ > > _______________________________________________ > galeon mailing list > galeon@xxxxxxxxxxxxxxxx > For list information, to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ -- Gerry Creager -- gerry.creager@xxxxxxxx Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843
galeon archives: