[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20060626: trouble converting GRIB2 products with cnvgrib

Robb & John,

The message below indicated that the user had downloaded files from
motherlode. The pattern/action that I use to file the conduit
products on motherlode in pqact.conduit ensures that the inventory 
product that begins with .status is not processed by the FILE action.
These files are available for users to ftp from the unidata account
on motherlode- so it is not these files the user obtained.

However, I found that there is a separate pqact.threddsconduit file
where the products are being files separately for thredds. All of these
actions are not using the ^/afs beginning, so the actions
are filing the ascii text inventory that is .status./afs.etc...

The actual problem is that the user's program is too brittle to
properly look for a GRIB product in a file with other
stuff (such as would be the case with WMO headers etc in a file),
but it would prevent confusion if the thredds pattern/action
just put the grib products in the file and not the inventory.

Chiz


On Mon, 2006-06-26 at 14:36, Tom Yoksas wrote:
> >From:  Rorik Peterson <address@hidden>
> >Organization:  UAF
> >Keywords:  200606262015.k5QKFfSZ017377 LDM GRIB2 CONDUIT status message
> 
> Hi Rorik,
> 
> re:
> >GRIB2/cnvgrib gurus and other savy problem solvers,
> >
> >I'm trying to convert GRIB2 to GRIB1 files on a 64-bit Athlon linux 
> >machine using 'cnvgrib'.  I've tried 2 different types of files and 
> >succeeded with NAM Alaska grid 242, and failed with 0.5-degree GFS.  I 
> >get them both via the LDM/IDD.  With the GFS files, I actually succeed 
> >converting the first 270 or so messages; I've tried several different 
> >GFS files, and it usually stops around here.  There should be several 
> >thousand records according to 'degrib'.  I used a debugger to see if I 
> >could find out anything, although I don't know much about GRIB2.  This 
> >is what I see happening:
> >The program is generating an index of all records in the GRIB2 file.  It 
> >does this by looking for characters 'GRIB...[12]' at the beginning of 
> >each record, reads forward to where the next record should be, and looks 
> >again.  After about 270 or so records, the next record starts with 
> >'/afs/.nwstg.nws.noaa.gov/ftp/SL.us008001/ST.opnt/MT.gfs-CY' instead of 
> >'GRIB...[12]'.
> >This can't be a coincedence, being part of the product name I get from 
> >the IDD.
> 
> This seems to indicate that you are filing the ASCII status file
> sent in the CONDUIT stream along with the binary GRIB2 messages into
> a single output file.  You can learn more about the catalog file
> in the Unidata web page on CONDUIT data:
> 
> Unidata HomePage
> http://www.unidata.ucar.edu/
> 
>   Data
>   http://www.unidata.ucar.edu/data/
> 
>     CONDUIT
>     http://www.unidata.ucar.edu/data/
> 
>       What is CONDUIT?
>       http://www.unidata.ucar.edu/data/conduit/index.html
> 
>         CONDUIT
>         http://www.unidata.ucar.edu/data/conduit/ldm_idd/index.html
> 
> Here is the relevant comment in the last URL:
> 
>   Since the data is injected as individual products, a status file
>   consisting of the injection time, original file size and data file
>   name is inserted at the end of the product sequence, with the name:
>   .status.ncep_file_name end_sequence  The status file will contain a
>   catalog of each individual LDM product identifier within the original
>   data file, as well as the insertion status code returned
>   by the LDM.
> 
> Please check your ~ldm/etc/pqact.conf action(s) for CONDUIT data
> and make sure that you process the status file differently than
> the GRIB/GRIB2 messages.
> 
> >I don't know if this is suppose to be appearing within the 
> >GRIB2 file or not? (seems like it shouldn't, a waste of space, but I 
> >don't know the GRIB2 spec that well.)
> 
> It isn't.
> 
> >At this point, the program stops 
> >generating an index, and only processes these 270 or so records.
> 
> This is consistent with my theory that the status file is being written
> in the single output file.
> 
> >Being curious, I crudely scanned the entire 1.2GB file and matched for 
> >the 13 character '/afs/.nwstg.n' pattern, and it only begins to appear 
> >at the approximate point in the GRIB2 file that record 270 is, and 
> >occurs several thousand times after that.  Again, not a coincedence, I 
> >figure (although I haven't done the math for a 13 byte sequence in 1.2GB).
> >
> >At this point, I'm stuck.  Is the problem with 'cnvgrib', the GRIB2 
> >files, my LDM, or perhaps something else?
> 
> The problem is in the ~ldm/etc/pqact.conf action you are using to FILE
> the GRIB/GRIB2 products.
> 
> >I reproduced this problem with a GRIB2 file I downloaded straight from 
> >Unidata's motherlode http portal, suspecting problems with my LDM.
> 
> This must have been one of the status files.
> 
> Cheers,
> 
> Tom
> --
> +----------------------------------------------------------------------------+
> * Tom Yoksas                                            UCAR Unidata Program *
> * (303) 497-8642 (last resort)                                 P.O. Box 3000 *
> * address@hidden                                  Boulder, CO 80307 *
> * Unidata WWW Service                            http://www.unidata.ucar.edu/*
> +----------------------------------------------------------------------------+
> 
> ===============================================================================
> To unsubscribe ldm-users, visit:
> http://www.unidata.ucar.edu/mailing-list-delete-form.html
> ===============================================================================