Dave,
Thanks for putting this in words.
I agree that building a DODS --> VisAD adapter from the netCDF adapter
should be no more work than building a DODS --> Java netCDF library. For
the reasons you cite, building an adapter for VisAD would have some
benefits (e.g., support for Sequences, which are used in DODS to
represent table data). There is work in DODS to add data type
translation to our servers, but there will always be limits on what it
can do (for example, some data sources are not capable of supplying
length information and so their sequence variables cannot be translated
to arrays).
At this point I don't have the resources to develop this. However, we
can work with someone who does the work. Our experience with the Java
class library for DODS is that it is simpler to use than the C++
library. I suspect a DODS <--> VisAD adapter would still take several
months (but I know very little about Steve's netCDF adapter or VisAD, so
take my statements about time with a grain of salt).
James
Dave Fulker wrote:
>
> Hi,
>
> I would anticipate that the portion of Steve's netCDF adapter which
> constructs the MathType--through intelligent guessing--could be
> reused regardless of whether the underlying access is built on DODS
> classes or netCDF classes.
>
> It might take more human effort to build on the DODS classes, but
> that solution makes more data accessible to VisAD users. For example,
> there are DODS servers that make data available only as "sequences"
> and "structures" in DODS parlance, and I think these data are
> essentially invisible through the (DODS-implemented) netCDF API. The
> VisAD data model is rich enough to make sense of these data types,
> but the netCDF API would block access to them (unless I am mistaken).
>
> Hence, I favor option two (building a VisAD adapter directly on the
> DODS classes) unless 1) I have misunderstood the limitations of the
> netCDF interface or 2) this path is substantially more expensive in
> human effort.
>
> Though John's message seems slightly tangential to Ethan's question
> of how best to link VisAD applications with DODS servers, the message
> is quite important to Unidata and DODS in a larger sense. I see
> John's view as consistent with the objective we articulated in the
> "Unidata 2003" proposal to NSF as striving to build a framework
> (http://www.unidata.ucar.edu/proposals/NSF2003/Geosif.html) having:
>
> "1. Objects that hold data and behave like sampled functions on a
> specified domain, with methods specialized for irregular and
> gridded data (with images included in the latter). Also, methods
> for storing and retrieving portable copies of these objects,
> partially or wholly compatible with netCDF. (These generalize
> netCDF, because an array may be viewed as a function whose domain
> is a set of index vectors. Sampled functions can represent any
> dataset of observations or state descriptors, with navigations
> and calibrations.)
>
> "2. Objects that represent the (potentially continuous and infinite)
> domains of sampled functions, including complex shapes as for
> representing terrain, ocean basins, rivers, etc. Also, an
> associated and extensible set of space-time coordinates,
> encompassing various models for the shape of the earth as well as
> state-dependant vertical coordinates, such as pressure, density,
> potential temperature, etc."
>
> I think this objective remains valid, and perhaps "pluggable semantic
> parsers" could help us get there, though I don't yet understand
> exactly what it means. I once proposed that Steve's netCDF adapter
> for VisAD could be augmented with a constructor whose signature
> included a text string making the MathType explicit, i.e., replacing
> Steve's intelligent guesswork with knowledge of specific netCDF files
> or conventions. Is the pluggable semantic parser idea more or less
> equivalent?
>
> Underlying any parser, I suspect, must be a data model that is rich
> enough to embrace the full complexity of the data representations and
> the data-analysis functions we face. Therefore, I propose that we
> consider the following question:
>
> Can we agree on the VisAD data model--perhaps accompanied with
> some concept of pluggable parsers (and a few examples for the
> most common netCDF and JGOFS conventions) to improve the
> understandability of the model--as a foundation for the higher
> level of semantics that now should be embedded in the data sets
> and data servers we deal with in Unidata and DODS?
>
> If this is not the right question to pose, I welcome alternatives.
>
> Dave
>
> --On Thursday, 13 July2000 22:36 -0600 John Caron
> <caron@xxxxxxxxxxxxxxxx> wrote:
>
> > tomw wrote:
> >>
> >> Ethan:
> >>
> >> > We've thought more about the first option than the second
> >> > because: option one would also be valuable outside the VisAD
> >> > community; and with option two, the use of attribute conventions
> >> > would have to be added instead of taking advantage of the netCDF
> >> > conventions already in the VisAD netCDF adaptor.
> >>
> >> I agree about the first option. As long as VisAD uses the Java
> >> NetCDF library, and DODS provides the mechanism to extend these
> >> classes with ones which would, when needed use DODS access, it
> >> would allow anyone with an app that uses NetCDF to use DODS --
> >> just like in the C and Fortran world, right? That would seem to
> >> be the all-around best solution.
> >>
> >> (One of the reasons I would not want to make a DODS VisAD adapter
> >> is that there is a lot of 'magic' (read: hard work) that Steve
> >> Emmerson did in the NetCDF adapter to turn the metadata into the
> >> proper MathTypes in VisAD and make Data objects that are directly
> >> usable. It would seem the same kind of work would be needed to
> >> convert a DODS data object, as well.)
> >>
> >> Does someone see this differently?
> >>
> >
> > well of course i see it differently :^}
> >
> > It seems to me that the problem is that the netcdf API is not high
> > level enough for what we want to acheive (eg create visad objects
> > with semantics that match the data). I suspect DODs isnt either,
> > but im not sure. i dont see how you can adequately represent the
> > data semantics unless you are parsing something like a netcdf
> > Convention, but perhaps I'm missing something. it seems like the
> > problem must be compounded if you try to map DODS to netcdf
> > (partly because the two APIs have some impedence mismatch).
> >
> > One solution that keeps coming up for me is to create pluggable
> > semantic parsers specific to various datasets and/or Conventions.
> > Also create C/ Fortran/Java data APIs for the data _writers_, and
> > get away from specifying things at the netcdf attribute level (we
> > will still need that, but most users would prefer a higher level
> > API).
> >
> > I plan on looking at DODS (and ADDE) more closely so I can
> > understand them better with respect to these issues. Any pointers
> > appreciated.
>
> =======================================
> Dave Fulker
> Unidata Program Director
> University Corp for Atmospheric Research
> ----------------------------------------
> Mail: PO Box 3000
> Boulder, CO 80307-3000 USA
> Email: fulker@xxxxxxxxxxxxxxxx
> Phone: 303-497-8650 Fax: 303-497-8690
> =======================================
--
__________________________________________________________________________
James Gallagher The Distributed Oceanographic Data System
jgallagher@xxxxxxxxxxx
http://unidata.ucar.edu/packages/dods
Voice: 541.757.7992 Fax: 541.758.3254
