Yes, this is certainly an interesting and familiar discussion. I applaud
the ideas. They are definitely worth pursuing for the community. However,
they have been around for a while. For example, we addressed them over a
decade ago in some of the work that we did here at IBM Research on
scientific data models. We then implemented these ideas in a visualization
package called Data Explorer (DX). That software has been available as
open source for a few years now (http://www.opendx.org,
http://www.research.ibm.com/dx) and is used in the netCDF community. Some
of the implementation details in the data model are discussed at
http://www.research.ibm.com/people/l/lloydt/dm/dx/dx_dm.htm.
But the work here was inspired by even earlier work that considered the
generalization of coordinates via fiber bundles by utilizing the
mathematics of differentiable manifolds (e.g., that of Dave Butler -- see
Butler, D. M. and M. H. Pendley. "A Visualization Model Based on the
Mathematics of Fiber Bundles". Computers in Physics, 3, n.5,
September/October 1989) as well as an API for access to self-describing
multi-dimensional arrays (my own work on CDF in the mid-80s, which led to
the separate development of netCDF). These developments have certainly
been noted by others as well. Most notable has been the effort to build a
more general data model by the ASCI program in the Department of Energy.
Bill Hibbard's work on Vis-AD is also worth noting as he has integrated a
data model with rich metadata as well as display and analysis tools.
For your reference, I have some old papers that talk about these ideas at:
http://www.research.ibm.com/people/l/lloydt/dm/DM.htm
http://www.research.ibm.com/people/l/lloydt/dm/function/dm_fn.htm
In more practical terms for netCDF user, yes you can consider variables as
carriers of the underlying information to represent a generalized field
(i.e., mapping of data to coordinates) via conventions. So, one
multi-dimensional array would have data, another could have sample points
in some space, another would have connectivity information that describes
the topological relationship between the sample points, etc. This could be
extended for other sorts of information, including graphics (e.g., colors,
normals). In fact, we developed some conventions for netCDF for just that,
which I posted about 10 years ago to the netCDF mailgroup. (See
http://www.research.ibm.com/dx/docs/legacyhtml/pages/usrgu067.htm and
http://www.research.ibm.com/dx/docs/legacyhtml/pages/usrgu068.htm for
details). Over the years I have occasionally participated in discussions
related to these ideas. Although the conventions were fully implemented in
DX long ago, they could have been more flexible or efficient. It was not
done due to lack of interest by potential users. However, some netCDF
users in the OpenDX community have recently looked at resurrecting these
ideas and enhancing the implementation.
--------------------------
Lloyd A. Treinish
Deep Computing Institute
IBM Thomas J. Watson Research Center
P. O. Box 218
Yorktown Heights, NY 10598
914-945-2770 (voice)
914-945-3434 (facsimile)
lloydt@xxxxxxxxxx
http://www.research.ibm.com/people/l/lloydt/
http://www.research.ibm.com/weather
"John Caron" <caron@xxxxxxxxxxxxxxxx>@unidata.ucar.edu on 02/05/2002
01:08:01 PM
Please respond to "John Caron" <caron@xxxxxxxxxxxxxxxx>
Sent by: owner-netcdfgroup@xxxxxxxxxxxxxxxx
cc: <jimc@xxxxxxxxxxxxxxxx>, "DODS Technical Discussions"
<dods-tech@xxxxxxxxxxxxxxxx>, "netcdfgroup"
<netcdfgroup@xxxxxxxxxxxxxxxx>
Hi Penny:
(I am taking the liberty of posting to some mail groups that might be
interested in this topic).
First I need to make a few claims that may not be that useful to you.
A coordinate system can be thought of as an invertible mapping from
indicial
space (I^n) onto a manifold embedded in R^n (see
http://www.unidata.ucar.edu/staff/caron/papers/CoordMath.htm for some
details). Netcdf 1-D coordinate variables are the simplest case of
coordinate systems, and the standard netcdf conventions have really only
dealt with coordinate variables and their use in mapping I^n -> R^n, in
fact
limited to the case of products of 1D functions, eg (i, j, k ) -> (f(i),
g(j) ,h(k)). In this case the requirement that the mapping be invertible is
equivalent to the requirement of monotonicity on each coordinate variable.
Whew, thanks for letting me get that off my chest ;^]
More general coordinate systems are possible, and I hope we are ready to
start defining some general conventions on how to express these, of which
COARDS and CF are special versions.
In particular, I think what you actually have is a sequence of
trajectories,
where a trajectory is a 1D sampling. (This should probably not be
considered
a grid, which are usually thought of as tesselating, since the sequences of
trajectories may be too far apart to make interpolation useful.)
If you dont think of it as a grid, then you have a 1D trajectory embedded
into 3D space, so the coordinate system for one trajectory looks like
CS:(i) -> (x(i), y(i), z(i)).
If you do want to think of it as a grid, then you have a 2D manifold
embedded into 3D space, and the coordinate system looks like CS:(i, j) ->
(x(i,j), y(i,j), z(i,j)).
In both these cases, all you need to be invertible is that the (x(i), y(i),
z(i)) points (or the (x(i,j), y(i,j), z(i,j) points) are unique. So the
sequence of lons in x(i) itself doesnt need to be monotonic. The underlying
reason is simple: the 1D or 2D sample is embedded in a higher (3D) space,
and you have a lot more freedom. The requirement of monotonicty only
applies
to the case of mapping equal dimensioned spaces.
Ok, thats an argument for when monotonicity is needed and when its not, in
a
theoretical sense.
Now what about in a practical sense where you actually want to use the
data?
The problem is that a number of viewers and analysis programs only know
about / can process the simple case of netcdf coordinate variables. Its
possible you might be able to get more functionality out of these packages
by shoehorning your data into these older conventions that werent designed
for your data. If thats the case, you need to identify what applications
you
need to use, and find out if its possible.
A better idea, if you can afford it, is to create a new, better convention,
and then help develop or extend applications to make use of those. I am
posting this message to the DODS tech group, because we have been
discussing
some of these issues, and the netcdf group, who might be game for yet
another try, or might know of some existing conventions that are more
appropriate. In fact, it looks like you might be able to use CF "auxiliary
coordinate variables", but i'll have to look at those closer.
I will have a look at your files and post some specific ideas.
----- Original Message -----
Cc: <jimc@xxxxxxxxxxxxxxxx>
Sent: Monday, February 04, 2002 12:39 PM
> Dear John,
> Steve Hankin has suggested that I let you know the issues that we are
> dealing with for the final issue of the WOCE data set in netCDF, and
> I write with the hope that you might be able to advise us.
>
> You may already know that all the WOCE data are being distributed as
> netCDF files, and currently we are grappling with the issues of
> making the files as consistent as possible across all the diverse
> data streams, and at the same time following existing conventions
> where possible.
>
> We have been trying to follow the conventions set by COARDS and CF,
> but have a problem with longitude in particular. We have files that
> may have a single value of latitude and longitude (eg a single CTD
> profile or a sea-level station) but also files that have a range of
> lat/lon (eg a float trajectory, or ADCP data along a ship track). At
> present our "WOCE convention" is for lat and lon to be variables (not
> global attributes) and to specify longitude as -180 to 180, degrees_E
> positive. However one of our members is arguing that this is not
> COARDS compliant and that we should make it so. However I'm not sure
> that COARDS provides a convention that suits all our files and so I
> am left wondering how best to treat our dataset as a whole over this
> thorny issue.
>
> Can you suggest a longitude convention that might suit the whole of
> our diverse dataset and that is an existing convention.... or should
> we accept that we may have to treat different data streams in the
> WOCE data resource slightly differently?
>
> Below is the recent discussion I have had with Steve Hankin on this
subject.
>
> Best regards,
> Penny
>
>
> >Date: Mon, 4 Feb 2002 19:25:38 +0000
> >To: Steve Hankin <hankin@xxxxxxxxxxxxx>
> >From: Penny Holliday <nph@xxxxxxxxxxxxxxxxxxxx>
> >Subject: Re: longitude again.
> >Cc: jimc@xxxxxxxxxxxxxxxx
> >Bcc:
> >X-Attachments:
> >
> >Hi Steve,
> >
> >>Hi Penny,
> >>
> >>COARDS was a standard designed to deal with grids. For a globe
encircling
> >>grid, COARDS specifies that the coordinates of a grid as encoded must
be
> >>monotonic -- i.e. that the branch point, at which the coordinates have
a
> >>360 degree discontinuity, is always at the edge of the grid.
> >>
> >>Am I correct in assuming that the WOCE data in question is a single
> >>profile per file?
> >
> >not always... we have shipboard ADCP and Met data which are underway
> >surface data that may span the dateline in one file (perhaps one
> >days-worth, or a single file per cruise). Also drifter and float
> >tracks that may wander across the dateline. It sounds to me as
> >though they would not be compliant.... but then if we chose 0 to 360
> >then we would have the same problem in the Atlantic.
> >
> >>The single-point longitude axis contained in each file
> >>is then by definition COARDS-like (monotonic in a degenerate sense).
But
> >>the data collection taken as a whole has a discontinuity somewhere ...
> >>unavoidably.
> >>
> >>So if I have interpreted your question correctly, the answer is
> >>
> >> 1. COARDS supplies an answer on a per-file basis, only, and your
> >> encoding sounds fine
> >> 2. for the collection as a whole, the important thing is that all
> >> individual files share the same convention about where the branch
> >> point is located.
> >>
> >>There is no existing COARDS-like standard that deals with a collection
of
> >>netCDF files viewed as a whole. It might be worth contacting John
Caron
> >>of Unidata/netCDF (caron@xxxxxxxx) to make him aware of the question
that
> >>WOCE faces. John is involved in the "THREDDS" project, which is
defining
> >>methodologies for handling large file collections through XML catalogs.
> >
> >I think I will do that since from your answer the COARDS convention
> >doesnt quite provide a solution for us.
> >
> >> Has this helped at all? - steve
> >
> >Yes I think it has - we may not have solved our dilemma quite yet,
> >but I do understand a bit more about the thinking behind the COARDS
> >convention.
> >
> >Thank you again for replying so quickly,
> >Penny
> >
> >>
> >>==========================================================
> >>
> >>Penny Holliday wrote:
> >>
> >> > Hi Steve, another question about longitude. I'd like to check with
> >> > you that our "WOCE convention" on longitude is CF/COARDS compliant.
> >> > We have specified that it should be written as -180 to 180 with
> >> > positive values east. However Bernie Kilonsky says this is not
> >> > compliant because the COARDS requires the sequence of values in a
> >> > single file to be "monotonic in a non-modulo sense". I confess to
> >> > not knowing what the phrase "non-modulo sense" means, but does this
> >> > mean that a file with longitudes in the Pacific that might range
from
> >> > -160 to 160 would not be compliant since it goes from -179 to 179
at
> >> > one point?
> >> > Thanks,
> >> > Penny
>
> ---------------------------------------------------------------
> Dr. N. Penny Holliday
> Deacon Hydrobiology Team and WOCE International Project Office
>
> Southampton Oceanography Centre
> Waterfront Campus, European Way
> SOUTHAMPTON, SO14 3ZH, UK
> Tel:+44-(0)23-80596206 Fax:+44-(0)23-80596204
>
> email:nph@xxxxxxxxxxxxxxx
> http://www.soc.soton.ac.uk/GDD/hydro/
> http://www.woce.org/ipo.html
> ---------------------------------------------------------------
>
