[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Re[6]: NOAAPORT and LDM

On Tue, 2 Mar 1999, Gregory Grosshans wrote:

>      Robb,
>      
>      After some email with FSL I've been able to get LDM/pqact to work on 
>      specified products.  Here is some information that I've found out.  I 
>      do have some questions though.
>      
>      An entry like:
>         FSL5    ^NOAAPORT       FILE    data/np/npdata
>      
>      Does work.  In fact if it looks like this:
>         FSL5    (^NOAAPORT*)    FILE    data/np/npdata-fsl
>      
>      Both data files look exactly the same.  
>      
>      However, the following two entries don't produce the same results:
>      
>      FSL5       ^NOAAPORT.NWSTG.TEXT.SA FILE    data/np/sadata-normal
>      FSL5       (^NOAAPORT.NWSTG.TEXT.SA*)      FILE    
>      data/np/sadata-fslway
>      
>      The second entry/listing results in a data file that is much larger 
>      than the data file generated by the first entry.  I don't understand 
>      why this is happening.
>      
Greg,

The only difference I can see is the white space between the last of the
pattern and the action FILE needs to be a tab.  Spaces are legal parts to
product names, so tabs are used as deliminators.  This is the reason most
pqact entries are written on 2 lines making sure no spaces are trailing
the product pattern. 

The first line is <feedtype><tab><pattern>
The second line <tab><action><tab><params>

>      Here is some additional information and a question:
>      
>      The NOAAPORT header for each product is a lot different than the 
>      typical Family Of Services headers.  For example, on NOAAPORT the 
>      following metar ob was received:
>      
>      @^L^ARUKWBC^Bc^C^B^P^_^AKDENSPCN31 CWAO 021627^M^M
>      SPECI CYFC 021627Z 19014KT 6SM -SHRA BR OVC018 RMK SC8=^M^M ^M^M
>      ^C
>      
>      And on the FOS, Domestic Data Services feed the same product looks 
>      like:
>      
>      ^A^M^M
>      
>      570 ^M^M
>      SPCN31 CWAO 021627^M^M
>      SPECI CYFC 021627Z 19014KT 6SM -SHRA BR OVC018 RMK SC8=^M^M ^M^M
>      ^C
>      
>      What I don't understand is that to capture this product in pqact on 
>      the NOAAPORT feed the following must be used:
>      
>      FSL5    (^NOAAPORT.NWSTG.TEXT.SP*)      FILE    data/np/metar-data
>      
>      From the DDS feed the following can be used:
>      
>      DDS        ^S[AP].... .... ([0-3][0-9])    FILE    data/dds/metar-data
>      
This is the result of the software to inject the products into the queue
and the header created.  I suspect that FSL is using the pqsend program
with the Np header, FOS uses pqing using a WMO header. I would
contact FSL about what the full Np header looks like.  Then you can create
pqact entries to "pick" out certain products. The ldmadmin watch command
is probably truncating the header, so one doesn't see the unique part of
the header. This is my speculation.

>      
>      
>      Sometimes it seems that the location of the parenthesis for the 
>      NOAAPORT feed is important.  I've tried having them in different 
>      locations or not at all and then LDM doesn't act on the product.  
>      Perhaps it depends on if a '*' is being used or not.
>      
I don't envy you, you got to find out what the products complete patterns
look like. Also, product patterns are regular expressions.
On our LDM web page:
LDM5 Site Managers Guide in PDF, HTML, or Postscript 

look at section on LDM configurations where the detail information about
pqact and regular expressions.


>      Since the actual product doesn't even have 'NOAAPORT.NWSTG.TEXT' 
>      within it, it makes me suspect that the LDM queue has a product ID for 
>      each product within the queue.  Then if the product ID matches 
>      something in the pqact.conf file, then pqact acts on the data/product 
>      in the queue.  Is this correct??

Noop, the LDM is ignorant of product ids. No internal tables


>      
>      On the IDD feed containing the NOAAPORT data do the product headers 
>      look like they do on the FOS, or do they look similar to what I'm 
>      seeing on NOAAPORT?

IDD feed Np products headers look like FOS.

Robb...
>      
>      
>      Thanks,
>      Gregg
>      
>      
>      
>      ______________________________ Reply Separator 
>      _________________________________ Subject: Re: Re[6]: acqserver 
>      question
>      
>      ------------- Begin Forwarded Message -------------
>      
>      Gregg,
>      I suggest that you change the "pattern" part of your pqact.conf line 
>      for FSL5 feeds.
>      
>      I made a simple pqact.conf file, that copies metars (and more) to a 
>      file.
>      
>      #
>      FSL5    (^NOAAPORT.NWSTG.TEXT.SA*)        FILE    -strip  
>      /tmp/metar-data #
>      And this appears to be working.
>      
>      The key that we use in acqserver in the pq_insert, is probably 
>      different than that used in the other data feeds. Within our Nimbus 
>      system the keys we look for metars are:
>      
>      NOAAPORT.NWSTG.TEXT.SABA31*
>      NOAAPORT.NWSTG.TEXT.SACN31* 
>      NOAAPORT.NWSTG.TEXT.SACN4*  
>      NOAAPORT.NWSTG.TEXT.SACN51* 
>      NOAAPORT.NWSTG.TEXT.SACU31* 
>      NOAAPORT.NWSTG.TEXT.SAGC31* 
>      NOAAPORT.NWSTG.TEXT.SAHA31* 
>      NOAAPORT.NWSTG.TEXT.SAUC82*
>      NOAAPORT.NWSTG.TEXT.SAUE80*
>      NOAAPORT.NWSTG.TEXT.SAUS52*
>      NOAAPORT.NWSTG.TEXT.SAUS70*
>      NOAAPORT.NWSTG.TEXT.SAUS80* 
>      NOAAPORT.NWSTG.TEXT.SAUS90* 
>      NOAAPORT.NWSTG.TEXT.SAUW83* 
>      NOAAPORT.NWSTG.TEXT.SPUS70* 
>      NOAAPORT.NWSTG.TEXT.SPUS80* 
>      NOAAPORT.NWSTG.TEXT.SPUS90* 
>      
>      All the keys that I create in the acqserver source code, 
>      (storeProduct.c) are in the form:
>      
>      NOAAPORT.<type>.<cat>.<wmoHdr>.[BBB].<timestamp>
>      
>      Do your pattern matching based on this.
>      
>      The small file that I have put together that appears to work is: 
>      ######################################################################
>      ########## #
>      # I am writing metars to /tmp
>      FSL5    (^NOAAPORT.NWSTG.TEXT.SA*)      FILE    -strip  
>      /tmp/metar-data #
>      # mail for alert
>      FSL5    (^NOAAPORT.NWSTG.TEXT.ACUS01*)  PIPE    -close  -strip
>      mail    address@hidden
>      # mail for sigmets
>      FSL5    (^NOAAPORT.NWSTG.TEXT.WS*)      PIPE    -close  -strip
>      mail    address@hidden
>      #mail for a couple of metars
>      FSL5    (^NOAAPORT.NWSTG.TEXT.SPUS90*)  PIPE    -close  -strip
>      mail    address@hidden
>      FSL5    (^NOAAPORT.NWSTG.TEXT.SPUS80*)  PIPE    -close  -strip
>      mail    address@hidden
>      
>      Hope this helps.
>      
>      Leslie
> 
> 
> ______________________________ Reply Separator 
> _________________________________
> Subject: Re: Re[4]: NOAAPORT and LDM 
> Author:  Robb Kambic <address@hidden> at 
> NSSLGATE
> Date:    2/26/1999 1:35 PM
> 
> 
> On Thu, 25 Feb 1999, Gregory Grosshans wrote:
> 
> >      Yes, the data is getting in the queue.  I see it 
> with ldmadmin watch >      and its in the ldmd.log 
> file. Outside of spacing here is what one
> >      entry looks like from ldmadmin watch:
> 
> Greg,
> 
> Good, Sometime questions may trivial to you, but I need 
> to have a base to compose a solution.  If I can solve 
> the problem.
> 
> >      
> > Feb 25 23:25:23 pqutil:   906773 19990225232522.849   
> FSL5 000  > 
> NOAAPORT.GOES_EAST.IMAGE.TIGQ01.KNES.252315.19990562324
> 14504.* > 
> So an entry like;
> 
> FSL5    ^NOAAPORT
> FILE    data/noaaport/test.np
> 
> 
> Doesn't produce any output?  I'm trying to create a 
> generic entry to see if any data can be captured by 
> pqact. An observation, the date looks future wrong, ie 
> 1999056232  If this is an date?
> 
> > The above two lines are actually on one line.  > 
> > In the pqact file I've tried using the WMO header 
> 'TIGQ01' and also the > entire string from 
> NOAAPORT...TIGQ01.  
> > 
> > Is the way the data being injected into the queue 
> incorrect for naming > purposes?
> 
> That's a good question.  Does FLS have any sample pqact 
> entries that they could share with you. It might help 
> our guessing here. 
> > 
> > Should I have just the WMO header?  So the SSEC 
> software does place the control 
> > characters in the product prior to ingesting the 
> product into the queue, > correct?
> > 
> Yep, They create a complete WMO type of product from 
> the NOAAport stream to be handed off to the LDM.
> 
> Robb...
> > Thanks,
> > Gregg
> > 
> > 
> > ______________________________ Reply Separator 
> _________________________________
> > Subject: Re: Re[2]: NOAAPORT and LDM 
> > Author:  Robb Kambic <address@hidden> at 
> NSSLGATE > Date:    2/25/1999 4:09 PM
> > 
> > 
> > On Thu, 25 Feb 1999, Gregory Grosshans wrote: >      
> > >      Robb,
> > >      
> > >      I've recently obtained code, after much paper 
> work, from FSL that 
> > >      injects the NOAAPORT data feed into the LDM 
> queue as a specified feed > >      type (e.g. FSL5).  
> However, pqact never sees or acts on any items that > > 
>      i've configured in the pqact.conf file to work on 
> the FSL5 data 
> > >      stream.  
> >      
> > Greg,
> >      
> > Are you sure the data is getting into the LDM queue?  
> What's the output > of:
> >      
> > % ldmadmin watch
> >      
> > If it isn't showing FSL5 product headers then no data 
> is entering the 
> > queue.The product headers will show the feed type and 
> pattern.  Are there any > error log messages?  If no 
> product headers then the process of entering
> > the data to the queue is the culprit, at that point I 
> would contact FSL. > I don't know anything about the 
> FSL code. 
> >      
> >      
> >      
> > >      
> > >      From what I've been told LDM waits until it 
> sees a control-A and then > >      goes into a special 
> state and processes data until it receives a 
> > >      control-C.  From what I can tell the data 
> coming off of NOAAPORT > >      doesn't contain 
> control-As or Cs.  
> >      
> > Again this is FSL's implementation of reading 
> NOAAport and making > products.
> >      
> > >      
> > >      If NOAAPORT is missing these control 
> characters, is that why pqact 
> > >      isn't doing anything with the data in the LDM 
> queue?  Did UNIDATA or > >      SSEC have to modify the 
> SSEC/UNITDATA hardware/software to place 
> > >      control characters at the beginning and end of 
> each file for LDM and > >      the IDD to work 
> correctly?
> >      
> > The SSEC processes the NOAAport products by eating 
> the Np header and then > making a WMO header for the 
> product with the control chars.
> > >      
> > >      As an example, I modified pqact.conf to send 
> email of the ACUS1, > >      ACUS2, and ACUS3 bulletins 
> when received from the FOS and 
> > >      NOAAPORT/FSL5.  I only receive the bulletins 
> from the FOS feed.  
> > >      Similarly, I configured pqact.conf to decode 
> metar obs from the FSL5 > >      feed but it doesn't 
> work, yet I already decode metar obs from the FOS > >   
>    and AFOS feeds.
> > >      
> > It appears that no NOAAport data is entering the LDM. 
> >      
> >      
> > Robb...
> >      
> > >      Thanks,
> > >      Gregg
> > >      
> > > ---------------------------------------------------------------------- 
> > >      Gregory Grosshans
> > >      NWS/NCEP/Storm Prediction Center 
> > >      Norman, OK        
> > >      405-579-0720     fax : 405-579-0700 
> > >      address@hidden
> > >      address@hidden         
> > >      /* standard disclaimer */
> > >      
> > > ---------------------------------------------------------------------- 
> > >       
> > >      
> > > 
> >      
> > ==============================================================================
> = 
> > Robb Kambic                                Unidata Program Center
> > Software Engineer III                      Univ. Corp for Atmospheric 
> > Research
>      
> > address@hidden                   WWW: http://www.unidata.ucar.edu/ 
> > ==============================================================================
> =
> >      
> >      
> > 
>      
> ===============================================================================
>  
> Robb Kambic                                Unidata Program Center
> Software Engineer III                      Univ. Corp for Atmospheric 
> Research 
> address@hidden                   WWW: http://www.unidata.ucar.edu/ 
> ===============================================================================
>      
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================