[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
Re: pqsurf.conf
- Subject: Re: pqsurf.conf
- Date: Fri, 3 Dec 1999 11:18:10 -0700 (MST)
Alan,
Good news, just what I expected. Your headers are a little different from
the standard NOAAport headers. I wrote a perl script to modify the
headers to the correct format, that can be used in pqact.conf. The script
is an attachment, called modHdrs. This is an example pqact entry, the -d
flag is for a directory
IDS|DDPLUS ^(......) (....) (......)
PIPE modHdrs -d data/sao \1\2\3
modHdrs writes out a file with the correct header format.
Robb...
On Thu, 2 Dec 1999 address@hidden wrote:
>
>
> This is an excerpt from the ldmd.log:
>
> Dec 02 15:25:18 npfos pqact[10850]: Successfully read pattern file
> "/usr/local/ldm/etc/pqsurf.conf"
> Dec 02 15:59:43 npfos pqsurf[10849]: 2325 19991202155928.888 IDS
> 000 SAUS70 KWBC 021600 1999336155812572
> Dec 02 15:59:43 npfos pqsurf[10849]: surface_split: Can't handle
> MESSAGE_TYPE_UNKNOWN
>
> >From your previous message, this looks like what pqsurf is expecting. I'm
> still not finding why pqsurf chokes with
> MESSAGE_TYPE_UNKNOWN.
>
> I know I'm your worst nightmare...Alan.
>
>
>
>
>
> Robb Kambic <address@hidden> on 11/29/99 04:01:49 PM
>
>
>
> To: Alan Hall/NCDC
>
> cc: support-ldm <address@hidden>
>
>
>
> Subject: Re: pqsurf.conf
>
>
>
>
>
>
>
>
>
> On Mon, 29 Nov 1999 address@hidden wrote:
>
> >
> >
> > I have spent a frustrating two days trying to come up with a work around.
> > I have two possibilities:
> >
> > 1) file the products as they come in a file more like what pqsurf would
> be
> > expecting and then re-queue those files into the ldm.pq. I can store the
> > files with a filename of: NWSTG TEXT SAUS70 KDEN 281536 1999333154000765
> > Getting that inserted into the queue is the next trick and I have
> succeeded
> > at that by using pqinsert as follows:
> > pqinsert -f WMO 'NWSTG TEXT SAUS70 KDEN 281536 1999333154000765'
> > pqsurf still balks with Can't handle MESSAGE_TYPE_UNKNOWN. I have also
>
> Alan,
>
> That's a good idea to re-queue the products but you got the header wrong,
> it should start with the "SAUS70 KDEN 281536 1999333154000765". Here's
> the output from ldmadmin watch on the products:
>
> Nov 29 20:48:02 pqutil: 1271 19991129204801.533 IDS|DDPLUS 420 SRUS58
> KWBC 292044 /pRRSMT
> Nov 29 20:48:02 pqutil: 147 19991129204801.542 IDS|DDPLUS 421 SRUS51
> KWBC 292044 /pRRSHOU
> Nov 29 20:48:02 pqutil: 826 19991129204801.553 IDS|DDPLUS 422 SRUS57
> KWBC 292044 /pRRSMS
> Nov 29 20:48:02 pqutil: 197 19991129204801.558 IDS|DDPLUS 423 SRUS60
> KWBC 292044 /pRRSMHX
>
> I would also check surf_split.c to see what the header it's expecting.
> I think the routine to look at is: pqing/wmo_header.c:get_wmo_header
>
> I think this will work with a little more investigation.
>
>
> Robb...
>
>
>
>
> > attempted to use the filename with the last 16 character date/time stamp
> > omitted at the end still get the same results.
> > I can get this work if I type in the pqinsert command manually, but if I
> > put into pqact.conf I get 'child exited with status 127' and I can't
> figure
> > out what that means and the product never gets inserted.
> >
> > 2) modify the ingest code to insert it the right way from the begining.
> I
> > am still investigating this, but with the tests from above I can't get
> > pqsurf to work even if I do have it inserted into the queue as it is
> > expecting it.
> >
> > If I could just get pqsurf to work so I can see what it's going to do, I
> > could make all the code changes needed. I am completely blind about
> what
> > pqsurf is expecting, what it will do, what the pqsurf.conf file should
> look
> > like, what the 'standard WMO type of header' is supposed to look like?
> >
> > As you can see I'm getting very frustrated.
> >
> > Alan.
> >
> >
> >
> >
> >
> >
> > Robb Kambic <address@hidden> on 11/23/99 03:36:30 PM
> >
> >
> >
> > To: Alan Hall/NCDC
> >
> > cc: support-ldm <address@hidden>
> >
> >
> >
> > Subject: Re: pqsurf.conf
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Alan,
> >
> > I looked into the pqsurf code, here's the scoop. The pqsurf code is
> > looking for the standard WMO type of header before it can split the
> > bulletin into individual reports. There are 2 solutions:
> >
> > - modify your ingest code to produce standard WMO type headers, ie
> > eliminate the NOAAPORT prefix and change the "." to spaces.
> >
> > - modify the surf_split.c code in the LDM pqsurf directory to accept your
> > current type headers. At this time, I have many other top priority
> > projects so I could not get to this for months. Maybe you have the
> > resources to do the modifications.
> >
> > I know this is not what you wanted to hear but that's about it in a
> > nutshell. Maybe Leslie could help you out.
> >
> > Robb...
> >
> > On Mon, 22 Nov 1999 address@hidden wrote:
> >
> > >
> > >
> > > I think we are making progress. I'm getting some entries in the
> ldmd.log
> > > that tell me we are getting things to pqsurf. Here is a sample of the
> > log:
> > >
> > > Nov 22 13:30:15 npfos pqsurf[9283]: Starting Up (9279)
> > > Nov 22 13:30:15 npfos pqsurf[9283]: lastmatch:
> > > 6453cb112c44cf491c21769dee1f4138 987 19991122133008.749 WMO
> 000
> > > NOAAPORT.NW
> > > Nov 22 13:30:15 npfos pqsurf[9283]: 19991122123015.215 TS_ENDT {{WMO,
> > > "^NOAAPORT\.NWSTG\.TEXT\.S[AIMNP]"}}
> > > Nov 22 13:30:15 npfos pqact[9284]: Starting Up
> > > Nov 22 13:30:15 npfos pqact[9284]: Successfully read pattern file
> > > "/usr/local/ldm/etc/pqsurf.conf"
> > > Nov 22 13:30:15 npfos pqact[9284]: TS_ZERO TS_ENDT {{WMO, ".*"}}
> > > Nov 22 13:30:15 npfos pqact[9284]: 0 19991122133015.238 ANY
> > 000
> > > _BEGIN_
> > > Nov 22 13:30:17 npfos localhost[9291]: Connection from localhost
> > > Nov 22 13:30:17 npfos localhost[9291]: Connection reset by peer
> > > Nov 22 13:30:17 npfos localhost[9291]: Exiting
> > > Nov 22 13:30:30 npfos pqsurf[9283]: 103 19991122133016.828 WMO
> > 000
> > > NOAAPORT.NWSTG.TEXT.SPCN51.CWAO.221327.1999326133016827
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]: 121 19991122133016.830 WMO
> > 000
> > > NOAAPORT.NWSTG.TEXT.SPCN31.CWAO.221326.1999326133016830
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]: 139 19991122133016.832 WMO
> > 000
> > > NOAAPORT.NWSTG.TEXT.SPCN41.CWAO.221327.1999326133016832
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]: 111 19991122133016.841 WMO
> > 000
> > > NOAAPORT.NWSTG.TEXT.SPUS43.KWBC.221328.1999326133016840
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]: 586 19991122133016.843 WMO
> > 000
> > > NOAAPORT.NWSTG.TEXT.SPUS70.KWBC.221328.1999326133016843
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]: 144 19991122133016.845 WMO
> > 000
> > > NOAAPORT.NWSTG.TEXT.SPXX60.KWBC.221328.1999326133016844
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]: 115 19991122133019.927 WMO
> > 000
> > > NOAAPORT.NWSTG.TEXT.SAPL61.KRAK.221330.1999326133019926
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]: 545 19991122133024.005 WMO
> > 000
> > > NOAAPORT.NWSTG.TEXT.SNVB20.RJTD.221300.1999326133024005
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]: 107 19991122133026.080 WMO
> > 000
> > > NOAAPORT.NWSTG.TEXT.SPCN48.CWAO.221326.1999326133026080
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]: 133 19991122133026.083 WMO
> > 000
> > > NOAAPORT.NWSTG.TEXT.SACN87.CWAO.221325.1999326133026083
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > >
> > > What is MESSAGE_TYPE_UNKNOWN and how can I correct it?
> > >
> > >
> > >
> > >
> > >
> > > Robb Kambic <address@hidden> on 11/19/99 04:11:06 PM
> > >
> > >
> > >
> > > To: Alan Hall/NCDC
> > >
> > > cc: support-ldm <address@hidden>
> > >
> > >
> > >
> > > Subject: Re: pqsurf.conf
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, 19 Nov 1999 address@hidden wrote:
> > >
> > > >
> > > >
> > > > I have the following entry in my pqsurf.conf file (this very line
> works
> > > > perfectly in pqact.conf):
> > > >
> > > > WMO
> > > >
> > >
> >
> ^NOAAPORT\.NWSTG\.(TEXT)\.([A-Z]...).[0-9]\.[A-Z]..[A-Z]|[0-9]\.([0-3][0-9]
> > > > )....\.
> > > > FILE -strip
> > > > /npraid/noaaport/data/NWSTG/\1/SURF/\2(\3:yyyy)(\3:mmm)(\3:dd)
> > >
> > > Alan,
> > >
> > > I reread the man page, it seems there are flags to set the feedtype and
> > > pattern. use the following flags on the pqsuf line in ldmd.conf.
> > >
> > > exec pqsurf -f WMO -p "^NOAAPORT\.NWSTG\.(TEXT)\.S[AIMNP]"
> > >
> > > Robb...
> > >
> > >
> > >
> > > >
> > > > tabs separate all the fields. I can't get anything from pqsurf. I
> am
> > > > completely lost on what this is supposed to do...
> > > > What should an entry in pqsurf look like?
> > > >
> > > > Alan.
> > > >
> > > >
> > >
> > >
> >
> ===========================================================================
> > > ====
> > > Robb Kambic Unidata Program Center
> > > Software Engineer III Univ. Corp for Atmospheric
> Research
> > > address@hidden WWW: http://www.unidata.ucar.edu/
> > >
> >
> ===========================================================================
> > > ====
> > >
> > >
> > >
> > >
> >
> >
> ===========================================================================
> > ====
> > Robb Kambic Unidata Program Center
> > Software Engineer III Univ. Corp for Atmospheric Research
> > address@hidden WWW: http://www.unidata.ucar.edu/
> >
> ===========================================================================
> > ====
> >
> >
> >
> >
>
> ===========================================================================
> ====
> Robb Kambic Unidata Program Center
> Software Engineer III Univ. Corp for Atmospheric Research
> address@hidden WWW: http://www.unidata.ucar.edu/
> ===========================================================================
> ====
>
>
>
>
>
===============================================================================
Robb Kambic Unidata Program Center
Software Engineer III Univ. Corp for Atmospheric Research
address@hidden WWW: http://www.unidata.ucar.edu/
===============================================================================
#!/usr/local/bin/perl
#
# Program to modify headers and trailers of feedtype to look like the
# old FOS format.
#
# usage:
#
# IDS|DDPLUS
# written by Robb Kambic 12/3/99
#
# process command line switches, no valid switches at this time
while ($_ = $ARGV[0], /^-/) {
shift;
last if /^--$/;
/^(-v)/ && $verbose++;
/^(-d)/ && ($dir = shift) ;
}
$dir = "." if( ! $dir ) ;
# process input parameters
if( $#ARGV == 0 ) {
open( STDOUT, ">$dir/$ARGV[ 0 ]" ) || die "could not open $ARGV[ 0 ]
$!\n" ;
}
select( STDOUT ) ; $| = 1 ;
# set interrupt handler
$SIG{ 'INT' } = 'atexit' ;
$SIG{ 'KILL' } = 'atexit' ;
$SIG{ 'TERM' } = 'atexit' ;
$SIG{ 'QUIT' } = 'atexit' ;
# Now begin parsing stream and breaking on \cC
$/ = "\cC" ;
# main loop set select processing here from STDIN
START:
while( 1 ) {
open( STDIN, '-' ) ;
vec($rin,fileno(STDIN),1) = 1;
$timeout = 1800 ; # 30 minutes
$nfound = select( $rout = $rin, undef, undef, $timeout );
# timed out
if( ! $nfound ) {
print STDERR "Shut down, time out 30 minutes\n" ;
atexit() ;
}
atexit( "eof" ) if( eof( STDIN ) ) ;
$_ = <STDIN> ; # get product
#
# Check and modify NOAAport headers
#
s#^.*\cA.*\cA....#\cA\cM\r\n999 \cM\r\n# ;
#
print $_ ; # write product
#
atexit( "eof" ) if( eof( STDIN ) ) ;
} # end while( 1 )
atexit( "eof" );
exit( 0 ) ; #should never get here
# execute at exit
sub atexit
{
local( $sig ) = @_ ;
if( $sig eq "eof" ) {
print STDERR "eof on STDIN -- shutting down\n" ;
} elsif( defined( $sig )) {
print STDERR "Caught SIG$sig -- shutting down\n" ;
}
close( STDOUT ) ;
close( STDERR ) ;
exit( 0 ) ;
}
