[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

2000320: decoding NOAAPORT products

Alan,

The dcgrib decoder does not rely on the FOS headers, it will find the GRIB 
start of product in the input stream or a file no problem. It actually
keys on finding GRIB and then the 7777 to delineate the product.

The problem you have is that the text decoders are looking for the FOS 
style products in the data stream, and that is not what you have
from FSL. Our IDD stream creates the FOS style data wrappers on the 
products so that the data appears the same as how it was designed to
be sent over FOS. In this manner, our users were able to keep using the
same decoding software without having to make changes when we switched from
FOS to NOAAPORT.

The data format itself has nothing to do with the LDM product header that FSL 
uses, so don't get confused with that. Rather the decoders need a way to find
the start and end of a bulletin within the data stream.

In FOS, all products looked like

^A cr cr nl
seqno cr cr nl
TTAAII CCCC DDHHMM
...

data


...
cr cr nl ^C 



In NOAAPORT, the products do not have the start of header (ctrlA), carriage
return, carriage return newline, and sequence number, carriage return,
carriage return, newline before the actual start of the WMO header and data.
Moreover, they do not have the end of text (ctrl-C) at the end of the bulletin.
Our data ingest actually creates these before pqing inserts the product into
the IDD data stream.

The text decoders are keying on finding that sequence of ^A \r \r \n.

Your product from FSL that you sent does not have the FOS wrapper.
Even worse, it has the NOAAport CCB block before the WMO. You probably
have trouble looking at these text products in vi!

An "od -c" of the SPUS produfct you sent looks like:
0000000   @  \f  \0 001   R   U   K   W   B   C 002  \0  \0  \0  \0 003
0000020 024 024 020 001   K   D   E   N   S   P   U   S   8   0       K
0000040   W   B   C       2   0   2   0   1   3  \r  \r  \n   S   P   E
0000060   C   I  \r  \r  \n




The first 24 bytes of your product are the communications control block
from NOAAport. We get rid of that. Then you see that your SPUS80 KWBC
product begins. If you are going to try to decode these products with
FOS style decoders, you will need to create a filter on the PIPE to
the decoder which strips out that leading 24 byte CCB and replaces it
with an eleven byte: 
^A \r \r \n
####\r \r \n

and then tacks on the trailing \r \r \n ^C. (actually the 24 byte CCB length
is calculated from bits 2-16 of the CCB (the first 2 bits are a checksum 
identifier),

So, a quick filter would eat the first xx bytes of the ccb (generally 24 bytes),
then spit out an 11 byte FOS looking header (the #### sequence number doesn't 
matter,
it can just be 9999) followed by the resto of the data product from thw WMO 
TTAAii on,
then at the end, spit out a \r \r \n ETX sequence. The decoders will then be 
able
to find the start and end of all the products in the stream.

Steve Chiswell
Unidata User Support



>From: address@hidden
>Organization: .
>Keywords: 200003202023.NAA26844

>
>
>The -s and -p paths check out fine.  All of the 'dc' log files show the
>same 'processed: 0' except dcgrib.  The dcgrib.log is always 0 bytes and we
>do get all the files in the hds directory that the dcgribs produce.  I
>double checked and verified each and every file dcgrib is supposed to
>create and they are all there and garp will display them perfectly.
>
>In the surface directory, I do get a single daily file for the sao's
>(20000319_sao.gem size approx. 1.2Mb).  When I try to view it thru garp, I
>get a map of the US and nothing else.  I get no sb, syn, or upa files in
>the surface directory.  From the dc logfiles, it looks like something is
>getting piped to the decoder, I just don't get any output.
>
>Nothing in the mos or storm directories either.  I have double checked the
>paths and can verify that we are receiving the 'raw' files, they just don't
>get deocded.
>
>The only data getting deocded are the grids.
>
>One thing you did mention in your note:
>     Most of the text bulletin decoders are looking for the ^A for style
>start of the bulletin. Since you are getting the data from FSL,
>     I don't know what their feed looks like.
>
>This triggers a memory of working with Rob Kambic and pqsurf.  He and I
>wrote a quick script to modify the header (his script) and re-insert it
>into the ldm.pq (my script) so pqsurf would recognize them.  I assumed that
>since the dcgrib was working with what is piped to it, the other deocders
>would too.  Could it be that sending something like
>^NOAAPORT.NWSTG.TEXT.SAUS30.KDEN.000345659.20000611545002 be the problem?
>Will the decoders handle a header that big?
>
>I have a dog and pony show comming up next week for the Center, and I would
>like to be able to show these other decoders too.
>
>Thanks again...
>Alan.
>
>PS: we are very close to setting up a NOAAPort Archive.  These decoders are
>what we will be using to display and fill orders on archived data.  This is
>very integral to our archive solution.
>
>
>
>
>
>Unidata Support <address@hidden> on 03/20/2000 12:28:38 PM
>                                                              
>                                                              
>                                                              
> To:      Alan Hall/NCDC                                      
>                                                              
> cc:      address@hidden                     
>                                                              
>                                                              
>                                                              
> Subject: 20000316: nawips                                    
>                                                              
>
>
>
>
>
>
>
>
>>From: address@hidden
>>Organization: .
>>Keywords: 200003162054.NAA04914
>
>>
>>
>>I seem to be missing the profiler (bufr) decoder dcprof.  I found the .c
>>and .o files, but the executable is missing from /bin/sol.  Is there some
>>reason or I did I not get it installed properly?
>>
>>Also, I am getting alot of 'Number of bulletins read and processed: 0'.
>>This has been going on for days without a single bulletin decoded for ship
>>(dcsynop), any of the storm data (dcstorm), or mos (dcamos,dcnmos,dcmmos).
>>Is this normal or do I have something configured wrong?  I know these
>>programs are being executed because of the log, but nothing ever gets
>>decoded.
>>
>>Alan.
>>
>>
>>
>>        ****** Signature Tag ******
>>
>>        National Climatic Data Center
>>
>>
>>
>>
>>
>>
>
>Alan,
>
>I don't use the dcprof BUFR profiler decoder any longer since FSL provides
>us with the NetCDF hourly and 6 minute observations in the FSL2 IDD data
>stream.
>The BUFR files available on NOAAPORT only contain hourly summaries. There
>were also serious limitations to the NWS distribution of the profiler
>data under FOS. Since the data were not considered operational, they would
>delay their broadcaset until after models etc, so that the midday delivery
>was often 2-3 hours late. The dcncprof decoder will decode both the
>6 minute and hourly NetCDF files from FSL.
>
>If you are not seeing any decoder data files with the other decoders, you
>probably need to check your -s and -p station and packing table paths to
>the
>decoder entry in pqact.conf to make sure the decoder can find the necessary
>ancillary tables in the gempak tree. At the least, you should see the
>mos files every day (2x for the ngm andavn) and the dcsynop will run
>most of the day between the surface synoptic decoding and the ship/buoy
>decoding. Since the WWUS60 files ccome in hourly, you should see output
>there too.
>
>Are the dchrly and dcupair decoders working, and the ones you list not?
>Or are none of the decoders performing. Most of the text bulletin decoders
>are looking for the ^A for style start of the bulletin. Since you are
>getting the
>data from FSL, I don't know what their feed looks like.
>
>Steve Chiswell
>Unidata User Support
>***************************************************************************
>Unidata User Support                                    UCAR Unidata
>(303)497-8644                                                  P.O. Box
>address@hidden                                   Boulder, CO
>---------------------------------------------------------------------------
>Unidata WWW Service                        http://www.unidata.ucar.edu/
><
>***************************************************************************
>
>
>