« Black hole BUFR... | Main | Better Organization... »

06 September 2011

It was a dark and stormy night when we decided to publish the AMPS (Antarctic Mesoscale Prediction System) model output on Unidata's THREDDS server. We were getting that data from NCAR's MMM group already, so it was just a matter of adding it to our TDS configuration catalog. I thought I would just double check that the files, which are in GRIB-1, were being read correctly through the netCDF-Java (CDM) library. That's when lady Trouble walked into my office, and I never should'a let her sweet talk me into taking the job.

I notice that they were using center 60 (NCAR), subcenter 255 (Missing), and table version 2. The missing value is not great, but not uncommon. We assign GRIB tables in the CDM using those three values, and I see that we are using the file "ncar_0_200.tab". Unfortunately, we don't have any audit trail of where this file came from, and its been munged into our standard format so its impossible to tell anything from looking at it. About half the parameters in the AMPS file are local parameters with numbers > 128. So I send a note to the AMPS group to get their local tables. Luckily, they publish those along with the data. Oops, 7 of their local parameters, all used in the file, are different from what we have in "ncar_0_200.tab".

Ok, easy to fix in any given case, but can I assume that the table I just got from them can be used for all AMPS output, and how do I know when a file is AMPS output? A generic library like CDM only has the information contained in the file itself.

Corresponding with the very helpful Kevin Manning, I find that they 1) use NOAA's w3lib library for writing GRIB-1 files, 2) don't have a subcenter assigned to them so they used 255 (makes sense), and 3) set the version to 2, because they follow NCEP version 2 (which follows the standard WMO table version 2) and "128-254 are for local usage".

At this point I realize that I don't understand what the version number is supposed to be. The relevent sentence from the WMO manual on codes:

octet 4 contents: GRIB tables Version No. (currently 3 for international exchange) – Version numbers 128–254 are reserved for local use

After a sleepless night staring at my VCR blinking "12:00" over and over, I realize that the likely use of the table version is that when files are intended for "international exchange" one is supposed to use only the entries in the standard WMO table. So if you set the version octet to 0,1,2,or 3, this means "use the standard WMO table" for that version. If you are using entries other than from the standard table, you indicate this by setting the version value to 128-254, which names your local table version. If your local tables changes, you need to change your version number. (It seems to me that as long as your changes are backwards compatible, you can just update your table but not change the version number. If you choose to change something (a parameter meaning or its units), then you change the version number. But maybe I shouldn't have drunk the third martini.)

Ok, so the final piece to make this work is that you have to publish your local tables, and of course properly match their version number with the version octet in the GRIB message.

To summarize what probably was envisioned by the GRIB designers, or perhaps what actually would work from a generic reader's POV:

A version number of 0-3 indicates that you are using only entries from the standard WMO tables.
If you are using other entries, the version number must be 128-254.
You must "own" the center id, or a subcenter id within the center, so that you can version your tables.
You must correctly version your local table. If your local table changes in a way that is not backwards compatible, you must change the version number.
You must publish your local tables at some authoritative place. You must maintain these indefinitely.
You must put the correct center, subcenter and version number into each GRIB message.

I would also add:

You must publish your local tables in a machine readable format. Do not use HTML, PDF, MS Word etc, since these are not machine readable formats.
Add enough metadata so that a google search on "<your center name> GRIB tables" finds you within the first few pages.

So examining the AMPS files, two key pieces are missing:

They are not using a subcenter id.
They are not versioning their local tables.

(Kevin Manning and Bob Dattore (NCAR/DSS) and I are working on fixing these)

Having failed my spot check, I started wondering what else was wrong with our GRIB tables. Stop reading this aloud to your children now, the rest of this story is rated R for profanity and gratuitous violence.

First, I check our "standard WMO table" against what Bob Dattore has on the DSS web site. Five of our entries are incorrect compared with the WMO document. A random selection of 5 other tables each showed a number of problems, mostly with our tables, but also a few errors in the DSS tables (now fixed). Here's where the profanity starts: one likely source of these errors is that WMO doesn't publish these tables in a machine readable format, you have to cut and paste from PDF or MS-Word. I also guess that there have been typos in the WMO documents themselves over the years, or perhaps the units have actually changed, or perhaps its the OCR software that has trouble reading the printouts that the WMO makes when the changes come in by email. NCEP and ECMWF tables currently have to be screen scraped from HTML pages, and who knows what other error-prone processes were used to create the CDM GRIB-1 tables.

So now I'm experimenting with different tables, comparing them against each other, second guessing each one, and thinking about changing to a more satisfying career as a door-to-door encyclopedia salesman. Why is it that no two GRIB tables ever completely agree with each other? Is there a Pauli Exclusion Principle for GRIB? Or can intelligent life only arise in universes where you are never completely sure if you have the right GRIB table?

I download tables from the ECMWF GRIB-API package. These appear to include WMO standard tables and ECMWF local tables. Oops, no delimiter between the name and units in their ASCII tables, so these can't be parsed unsupervised (without a human). I'm wondering if getting canonical GRIB tables would violate causality? So I try the tables from the ECMWF GRIB-EX package. These at least are parseable and appear to include WMO standard tables and local tables for center 98 (ecmwf), 74 (ukmet RFSC), 80 (rome NMC), and 85 (toulouse RMSC). The WMO versions 1,2,3 all agree with each other (yay!), but there are 7 udunit differences with DSS wmo version 3, and comparison of a random table (98-0-140) with DSS: missing 5 params in DSS. Its clear that the Standard Model of particle physics will have to modified to account for this. Oh yeah, and that ECMWF routinely overrides entries in the "reserved for the WMO" section of GRIB1 parameter tables.

I am sent a sample file from center 9-157 (NWS - North Central River Forecasting Center), params 61 (version 2) and 237 (version 128). (The problem is that the reference dates change which we don't handle correctly, so the time coordinates are wrong (fixed now in TDS 4.3)). Hey, lets try to find out if our tables are correct for this one! Suspicious use of nceptab_3.tab for param 237 = "Total column ozone concentration". Why does a hydrology model output ozone? Enquiries follow. The issue is taken up by Important People. News at 11.

In one of those strange coincidences that happen only in reality and bad fiction, Bill Anderson at FNMOC calls me the next day. He and I had talked last October about some discrepencies in their GRIB-1 tables. He reports that he is retiring in 2 days (congratulations, Bill!), and wants to report that, nope, they still aren't versioning their GRIB tables for their operational model, that the tables occasionally change (even in the "reserved for WMO" section) and is not sure if they have an accurate record of when the changes happened. So CDM won't be able to read files from center 58 and accurately report whats in them. The only thing that's known for sure is what GRIB table is in current use, for their operational customers. I'm wondering what the CDM should do if it encounters such a file. Report an error? Refuse to read it? Erase the file? Hmmmm.

Stay tuned for more heart wrenching stories about table-driven file formats in the next exciting episode of "Life with GRIB and BUFR".

Posted by $entry.creator.screenName [ Comments [3] ]