<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Thanks Jennifer <br>
for this history and related information. It provides me with some
energy to go back to our original gds/bufr attempts and focus on what
is likely the problem rather and reinvention. The data we're having
issues with is the official NWS U/A collectives <u>sent</u> from the
gateway! So our assumptions (no comment) was that the data feed was
correct. I'll bet you decoded upper air..... <br>
Regarding grib2, there is apparently some issues of release in that
NCEP has one while HQ (Glahn and NDFD) another? The NDFD (Nat'l
Digital Forecast Database) is the only "operational" product that I
have to build to for archive and access. Maybe a look at a url
relating to NDFD and docs might lead to what I see as the stable
templates for grib2. Not sure tho', but they seem to think it's been
blessed by WMO. With that, I have a call into Brian and hopefully we
can work something out. <br>
<br>
Again, thanks so much for all the help from COLA. Glenn. <br>
<br>
Jennifer Adams wrote:
<blockquote cite="midfc50d04de4cd8d8970e0c5f2bc3e54a1@xxxxxxxxxxxxx"
type="cite">Dear All Who Care --
<br>
Out of all the acronyms being tossed around in this thread, I can only
comment on two: BUFR and GRIB2.
<br>
<br>
1. BUFR:
<br>
BUFR is a WMO standard for storing station data (aka sequence data).
It's not the friendliest format in the world, but it can store a lot of
information in a small space with liberal use of look-up tables and
bit-by-bit packing. Some time ago, Joe Wielgosz, the guy who wrote the
GDS, also wrote a BUFR decoder -- it reads BUFR and spews out ascii.
After Joe wrote his decoder, I wrote an interface for GrADS, a client
that already has the ability to handle station data. Starting with
version 1.9, GrADS could read and display BUFR data from NCEP -- the
only BUFR data source we attempted to handle. BUFR is so complicated
and flexible -- it's a bit like netcdf in that you can probably stuff
any kind of crapola data into that format if you try hard enough --
there's just no way that we could write code to handle every possible
BUFR file out there. Oops, I digress... Anyway, Joe then wrote some
code for the GDS to serve station data, and for GrADS to be a client
for station data served via GDS. At that point, Joe left the
programming business, and I moved on to other development projects.
<br>
<br>
Since then, I have heard from more than a few BUFR users who needed
help in using the GrADS interface -- including Tennessee and the crew
at NCDC. I have explained how the BUFR decoder works, how to write
GrADS descriptor files, how to test with their own BUFR files, and (in
NCDC's case) how to patch the GrADS code so it would work with their
files. Working with BUFR is not that much fun, but much of the hard
work has already been done -- I'm a little dismayed to hear that both
Tennessee and Glenn's crew have given up on the GrADS interface and are
reinventing the wheel with perl and conversion to netcdf.
<br>
<br>
2. GRIB2:
<br>
Brian Doty and I have looked at one GRIB2 data sample: the GFS output
on a 0.5 degree grid, which uses data representation template number
5.4, and we haven't been able to find any documentation for this
template as yet. A simpler short-term solution is to uncompress and
read the GRIB2 data outside of GrADS (there are some libraries and code
samples available from NCEP to do this) and rewrite the data into a
format GrADS can read. Our impression is that GRIB2 is designed more
for storage and archival needs rather than access. We have plans to
support simple packing (template number 5.0) to take advantage of the
expanded header field options available with GRIB2. This will only
require changes to gribmap, an external GrADS utility, and not the
GrADS code itself.
<br>
<br>
-- Jennifer
<br>
<br>
Jennifer Miletta Adams
<br>
IGES/COLA
<br>
4041 Powder Mill Rd., Suite 302
<br>
Calverton, MD 20705
<br>
<br>
<br>
On Apr 15, 2005, at 12:31 PM, Glenn Rutledge wrote:
<br>
<br>
<blockquote type="cite">From a users perspective (and not speaking
for COLA), yes- they did BUFR w/ the latest 1.9, but we have some
issues with a couple of required datasets and trying to figure a
workaround (thus the perl bufr decoder). But indeed, GDS handles
plotting/decoding and gds serving of (most?) BUFR. My pblm might be
the data feed, the tables, or the supplied metadata from eh
originator. Regards, Glenn
<br>
<br>
James Gallagher wrote:
<br>
<br>
<blockquote type="cite"><br>
On Apr 15, 2005, at 6:52 AM, Glenn Rutledge wrote:
<br>
<br>
<blockquote type="cite">Wow,
<br>
<br>
This thread (pun intended) woke me up this Friday morning. WCS for
OPeDNAP will be a great leap forward for our shop and BUFR- the WMO
standard format for model input ("the BUFR tanks" at NCEP, Upper-Air
collectives, and other data near and dear to NWP, goes a long way
toward the implementation of a long awaited single (virtual) interface
to for anyone doing Model Intercomparison Projects, or
grib/NetCDF/BUFR... data analysis. We just wrote a BUFR decoder in
perl to overcome some of our issues with BUFR (as listed below), so
yes- NOMADS cares very much- more importantly users- last month we had
4M downloads, serving over 3.6Tb of data and several emails requesting
BUFR access. NOAA has a requirement for BUFR decode/access and display
as well.
<br>
So, the NCDC NOMADS shop will gladly test or otherwise try to help in
this endeavor. Granted, we don't have the expertise to develop this-
we do have some talent to install and flush out issues associated with
these formats and interfaces thru OpeNDAP. One final note:
Grib2 is
on the horizon....COLA is very much aware of this but not a top
priority. Anyone working this for GDS? Just wondering......
<br>
</blockquote>
<br>
<br>
On the GIS/WCS front: Dan Holloway is taking a short vacation, but when
he gets back he can provide first-hand information about the
MapServer-OPeNDAP gateway. Briefly, MapServer uses the GDAL library to
read from various sources. Frank Warmerdam and I have written a
GDAL-DAP driver, so GDAL can be used to read from DAP servers. There
are limits, but Frank took my driver code and spiffed it up quite a
bit. He also wrote an OGR driver; OGR is the point data equivalent to
GDAL (GDAL is designed to work with raster data). So OGR can be used to
read from 'DAP Sequences' like the EPIC-DAP server from PMEL. Caveat: I
haven't tested this myself.
<br>
<br>
On the BUFR front: There's been considerable interest in support for
BUFR for a long time. I though that COLA was working on this? Can some
one on DODS TEch comment?
<br>
<br>
James
<br>
<br>
<blockquote type="cite"><br>
Thanks to all for your leadership and vision in all this....Glenn
<br>
<br>
John Caron wrote:
<br>
<br>
<blockquote type="cite">Tennessee Leeuwenburg wrote:
<br>
<br>
<blockquote type="cite">Hi guys,
<br>
<br>
We're looking at phase two of the project I'm working on at the moment.
Phase one involved hooking up our weirdo database to OpenDAP, and it
looks that that will be a success. There's one thing that's not as
wonderful as we might hope, and a couple of cool extra things we'd like
to do.
<br>
<br>
In order to cope with observational data stored in BUFR format, we've
had to write scripts for conversion to NetCDF prior to serving it with
thredds. I don't really like messing about in thredds code if I can
help it, on account of it makes code maintainance that much harder. So
we convert to NetCDF on a product-by-product basis. But it would be way
cool if we could serve BUFR data more generally and without the
conversion step. So my question is : does anyone on this list care much
about BUFR data, and is anyone thinking much about how it might fit
into an OpenDAP context?
<br>
</blockquote>
<br>
<br>
<br>
What kind of data is stored in BUFR?
<br>
<br>
We have an "I/O service provider" framework in netcdf-java version 2.2,
where you can read non-native files as if they are netcdf files. We
have done GRIB files in this way, among others. It would probably be
very cool to be able to do that for BUFR files. If you're interested i
can tell you more.
<br>
<br>
<blockquote type="cite"><br>
The other angle is from the GIS world rather than the data sharing
world. We'd love to publish some of our data sets in such a way that
GIS packages can read them. To the best of my knowledge, no GIS package
reads OpenDAP or even NetCDF. This means we've either got to do a
conversion step from NetCDF into something that the GIS packages can
read, and then serve it out using a different server - like a shapefile
server for instance. So we would have to choose specific products to
make available, provide shapefile derivatives from our hi-res NetCDF
data, and tell people to go look there. It's not the worst plan I've
ever heard, but has anyone here heard about any efforts to integrate
OpenDAP more tightly with GIS software?
<br>
</blockquote>
<br>
<br>
<br>
The next version of the THREDDS server will have OpenDAP integrated
into it. It will also have an experimental WCS server for gridded
data. The idea is that anything that can be read in through
netcdf-java version 2.2 can be served through opendap. If theres enough
georeferencing, these can also be served through WCS. Again it would be
cool to extend this to a WFS server for features, although it would be
a lot of work.
<br>
<br>
We are also working with ESRI on their project to read in netcdf
gridded files, and another project with GDAL/Cadcorp experimenting with
netcdf over WCS. Still both in research stage.
<br>
<br>
</blockquote>
<br>
-- <br>
Glenn K. Rutledge
<br>
NOMADS Program Manager
<br>
NOAA Meteorologist / Physical Scientist
<br>
National Oceanic and Atmospheric Administration
<br>
National Climatic Data Center
<br>
151 Patton Ave
<br>
Asheville, North Carolina 28801
<br>
(828) 271-4097
<br>
<br>
The contents of this message are mine personally and do not necessarily
reflect any position of the Government or the National Oceanic and
Atmospheric Administration."
<br>
<br>
<br>
</blockquote>
-- <br>
James
Gallagher
jgallagher at opendap.org
<br>
OPeNDAP,
Inc
406.783.8663
<br>
<br>
</blockquote>
<br>
-- <br>
Glenn K. Rutledge
<br>
NOMADS Program Manager
<br>
NOAA Meteorologist / Physical Scientist
<br>
National Oceanic and Atmospheric Administration
<br>
National Climatic Data Center
<br>
151 Patton Ave
<br>
Asheville, North Carolina 28801
<br>
(828) 271-4097
<br>
<br>
The contents of this message are mine personally and do not necessarily
reflect any position of the Government or the National Oceanic and
Atmospheric Administration."
<br>
<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Glenn K. Rutledge
NOMADS Program Manager
NOAA Meteorologist / Physical Scientist
National Oceanic and Atmospheric Administration
National Climatic Data Center
151 Patton Ave
Asheville, North Carolina 28801
(828) 271-4097
The contents of this message are mine personally
and do not necessarily reflect any position of the
Government or the National Oceanic and Atmospheric Administration."</pre>
</body>
</html>
