Re: GRIB1 and BUFR in TDS

• To: Ethan Davis <edavis@xxxxxxxxxxxxxxxx>
• Subject: Re: GRIB1 and BUFR in TDS
• From: Robb Kambic <rkambic@xxxxxxxxxxxxxxxx>
• Date: Wed, 13 Dec 2006 16:35:11 -0700 (MST)

Hi,

There was some minor problems in decoding the files but they are now fixed and 
ci.  the grib file had a strange char in the header and the bufr decoder was 
trying to decode the qc section.  the test files are a bit strange, each 
parameter was broken into 2 or more files, PRO, 000001, 000002, 000003...  each 
file has a PRO as the header and data starts in the 000001.  this is not 
supported by either of Unidata's  grib or bufr libraries because it's not in 
either formal GRIB or BUFR specification. i only tested the 000001 files 
because the others would of required code changed to test.

so if the users want to use our libraries without modification then the files 
need to be combined into 1 file, that can contain a header and data without any 
auxiliary headers in the file. i assume that the files were broken into parts 
to make for easier data transfers. Also, i would suspect that there is a 
program to reassemble the files without the additional headers and no 
additional characters being inserted into the files.  i would like to test some 
of the combined files if they are available.


robb...

On Fri, 8 Dec 2006, Ethan Davis wrote:

Hi Robb,

I got this email last week but guess I never forwarded it on to you. Can  you 
take a look at the data files and see what it would take to read them? I've 
saved them in /upc/share/testdata/valentijnGRIB_BUFR. Sorry, I'm not sure which 
ones are GRIB and which are BUFR.

Thanks,

Ethan

Valentijn Venus wrote:

Hi Ethan,

I hope you are fine/

I just read the support message "TDS and BUFR" below (You, John, and Robb). Can 
you look into the attached example files as we receive them currently through EUmetcast 
from Eumetsat (http://www.eumetsat.int/Home/Main/Access_to_Data/Meteosat_Meteorologica 
l_Products/BUFR___GRIB2/DF_ACC2DATA_METP_BUFR?l=en). Can you see what is needed to 
support them within the TDS?

Cheers, Tyn

You may be helped with the following details:

I have attached one GRIB1 dataset of Total Ozone from Meteosat-8/SEVIRI:
<<L-000-MSG1__-MPEF________-TOZ______-PRO______-200511010845-__>>
<<L-000-MSG1__-MPEF________-TOZ______-000001___-200511010845-__>>

Also find attached two BUFR datasets (Cloud Layer Analysis):

*CLA* for which the corresponding BUFR tables can be found in here:
http://www.eumetsat.int/groups/ops/documents/document/pdf_mpef_local_buf
r_descript.pdf

The navigation is of the GRIB1  data is

still a bit foreign. Its the projection used by Eumetsat
(www.eumetsat.int) for its Meteosat-8 imagery. If it is the same as is
defined in the current HRIT MSG server code (msgtaget2.f) than it
wouldn't be to complicated. The 90 Space View Perspective or
Orthographic projection is defined in documents EUM/MSG/ICD/105 and
EUM/MSG/SPE/022 (available at their website under
Publications/Technical Documentation). Here is a section from it:

--------------------------------------------------------------------
                       Header : GRIB
                   Discipline : 3 Space products
                 GRIB Edition : 2
                  GRIB length : 3444913
           Originating Center : 254 EUMETSAT Operation Centre
       Originating Sub-Center : 0
Significance of Reference Time : 3 Observation time
               Reference Time : 2005:4:18T12:0:0
               Product Status : 1 Operational test products
                 Product Type : 6 Processed satellite observations
        Number of data points : 13778944
                    Grid Name : 90 Space View Perspective or

Orthographic

                    Grid Shape: 3 Earth oblate spheroid with axes
specified by produ cer
       Oblate earth major axis: 6378.14
       Oblate earth minor axis: 6356.7554 Number of points along
parallel: 3712 Number of points along meridian: 3712 Latitude of
sub-satellite point: 0.0
 Longitude of sub-satellite pt: 0.0
 Resolution & Component flags : 0
        i direction increment : 3566.0
        j direction increment : 3561.0  X-coordinate of sub-satellite
: 1856.0001  Y-coordinate of sub-satellite : 1856.0001
                Scanning mode : 192
                  Basic angle : 0
       Altitude of the camera : 7.4533146E8
       X-coordinate of origin : 0.0
       Y-coordinate of origin : 0.0
           Product Definition : 30 Satellite product
           Parameter Category : 0 Image_format_products
               Parameter Name : 7 Cloud_mask
              Parameter Units :
      Generating Process Type : 8
                 ForecastTime : 0
           First Surface Type : 0 Reserved

Re: TDS and BUFR

    * To: Ethan Davis <edavis@xxxxxxxxxxxxxxxx>
    * Subject: Re: TDS and BUFR
    * From: John Caron <caron@xxxxxxxxxxxxxxxx>
    * Date: Sat, 19 Aug 2006 12:47:11 -0600
    * Cc: James Gallagher <jhrg@xxxxxxx>, "'support-thredds'"

All correct.


We are just getting around to testing BUFR, I would consider it alpha software 
for now. The CDM data objects will likely evolve some more.

Ethan Davis wrote:


    Hi James,


    The short answer is yes but not all types of BUFR files are currently 
supported.

    Now for a slightly longer answer that John may want to correct as I'm not 
as up on our BUFR work as he is ...

    John and Robb (also here at Unidata) have developed a ucar.bufr library. The netCDF-Java library, which 
TDS uses for reading data files, uses ucar.bufr to read BUFR files. I think there is a lot of variation in 
the data structure represented in BUFR files. Each type of BUFR files require some mapping into the 
netCDF/CDM data model. This is the hard part and will require on going work as we find new types of BUFR 
files. However, I believe we already have a number of "standard" BUFR file types being read by 
netCDF-Java. Another complication is that, like GRIB, BUFR also external tables (for parameter info and such) 
like in GRIB. And, as with GRIB, the different "centers" can extend the "standard" 
tables. So that adds to the work.

    Ethan