[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
Re: Re[2]: LDM write errors
- Subject: Re: Re[2]: LDM write errors
- Date: Wed, 1 Mar 2000 16:00:29 -0700 (MST)
Ken,
I did the prelimary mod to your process script, called it process1.
Here's how I would run it:
AFOS ^(...)(...)(...|..)
PIPE /home/ldm/process1
Notice no parameters so only one script should run at a time. I put the
keyword Ken in the script to show the major changes in the code. Here's
what I would do, get a raw data file, execute process1 in debug mod, ie
% perl -d process1 < <rawAFOSfile>
You should be able to modify the lines to extract the headers once you see
them in the debugger. Make the changes to get the header info and change
the ARGV[0] and ARGV[1] to the proper variable names. You will also have
the whole product available to write it out in a file so you don't need
to make any tmp files. The script will wait 20 minutes before exiting so
there will be no conflicts on STDIN.
I attached the file process1.
This explaination is what I meant by the comment below: Another solution.
Robb...
On 1 Mar 2000, Ken Waters wrote:
>
> Robb,
>
> Thanks for the replies. My comments are below.
>
> Ken
>
>
> ______________________________ Reply Separator
> _________________________________
> Subject: Re: LDM write errors
> Author: address@hidden at EXTERNAL
> Date: 3/1/2000 12:52 PM
>
>
>
> >Their are a couple of things you might want to change to help. First the
> >above errors are caused by the PIPE action trying to write to the process
> >decoder. Inspecting the process decoder, I didn't find any <STDIN>
> >filehandle reads, therefore the LDM is writing to the decoder but the
> >process decoder is not reading anything. The PIPE action should be changed
> >to an EXEC action or make the process decoder read the <STDIN> input.
>
> I tried changing the action to an EXEC and the result was a lot of
> errors like this:
>
> pqact[1063]: child 1310 exited with status 127
>
> and no action taken. I don't know what was going wrong here, but I
> agree
> it makes more sense for it to be an 'EXEC' versus a 'PIPE', since I'm
> not
> using the <STDIN>.
>
>
> I also had been using <STDIN> input, but you'll recall that that
> failed miserably for me because the various instances of the script
> were having trouble "competing" for the <STDIN>. By changing to a
> file write in LDM and then a file read in the script this problem went
> away.
>
>
> >From your pqact.conf file:
>
> ># Run the process script to do the proper actions for this new message
>
> >AFOS ^(...)(...)(...|..)
> PIPE -strip /home/ldm/process \1 \2 \3
>
>
> >With this type of pqact entry, a new process decoder is spawned for
> >every product because the parameters "\1 \2 \3" change for every
> >product. The LDM currently has a limit on the number of processes of 32:
>
> >/*
> >* Tuning parameter: MAXENTRIES is the number of descriptors
> >* you wish to allocate to this list. You need to leave enough
> >around * for the error output and for rpc.
> >*/
> >#define MAXENTRIES 32
>
> >If estimate 2-3 products/second then there would be over 150 decoders
> >spawned / minute. You could up the limit to 200 if your computer
> >platform/os can handle that many processes. The downside is that your ldm
> >configuration would be no-standard. The file to change is: pqact/filel.c
> >This is the reason that the process decoder is being spawned twice or more
> >for every product, the LDM was reaping the process decoder before it was
> >done.
>
> This may be the solution unless I can figure out another way. I
> notice that the problem seems to only occur with the larger products,
> usually > 6000 bytes.
>
>
> >Another solution:
>
> >You could read the product in your decoder, extract the header then do the
> >rest of the decoder processing. That;s how the current UPC perl decoders
> >work. The major pro side is that there is only one decoder running.
>
>
> I'm sorry, I don't follow you here. Do you mean to jump to another
> script from the first one?
>
>
> >Observation:
>
> >In the decoder process there are many "system("mv ... " type calls. For
> >every system call a fork is done, this is time consuming and resourse
> >consuming to the LDM and OS. It would be better to extract all the system
> >calls to another external script that could be run out of cron. If you
> >named your file something.new, then the external script could do the moves
> >external to the LDM. In my opinion this would be a much cleaner solution.
>
> >Robb...
>
> Not sure how those calls could be run from cron, since their receipt
> is event-driven, rather than schedule-driven. Perhaps the Perl "copy"
> command would do better than using "system". I think when I tried to
> make this change I had errors; I could revisit the situation. Right
> now, system performance, based on "top" analysis and script execution
> times, does not seem to be hindered.
>
===============================================================================
Robb Kambic Unidata Program Center
Software Engineer III Univ. Corp for Atmospheric Research
address@hidden WWW: http://www.unidata.ucar.edu/
===============================================================================
#!/usr/bin/perl -w
#
# /home/ldm/process - this script processes ALL incoming products that come
from the LDAD/AWIPS
# Written by: Ken Waters, SRH/SSD
#
# Called with three parameters, each a part of the "AFOS PIL", for example,
# 'process FTW ZFP FTW'
# The product file must already exist in the $temp directory, for instance,
# /home/ldm/data/current/FTWZFPFTW.tmp
#
# Updates:
#
# Feb 2, 2000: Fixed METAR and TAF append to properly append at the top
# Added branches for tropical products
# Feb 8, 2000: Fixed the problem with hung "cat" calls
# Feb 9, 2000: Added the experimental "recent" versioning of ZFPs,
# AFDs, and NOWs
# Feb 23, 2000: Changed versioning to 25 for TORs and SVRs
#
### Ken Next 28 lines need to be moved/changed after extract header info here
line.
# For all tropical products, change the "AFOS Node" from FTW to TRO
if (($ARGV[1] eq 'TWD') or ($ARGV[1] eq 'TWO') or ($ARGV[1] eq 'DSA') or
($ARGV[1] eq 'TCD') or ($ARGV[1] eq 'TCE') or ($ARGV[1] eq 'TCM') or
($ARGV[1] eq 'TCP') or ($ARGV[1] eq 'TCU') or ($ARGV[1] eq 'TWS') or
($ARGV[1] eq 'SPF'))
{
$ARGV[0]='TRO';
system("mv /home/ldm/data/current/FTW$ARGV[1]$ARGV[2].tmp
/home/ldm/data/current/TRO$ARGV[1]$ARGV[2].tmp");
}
# Set the variables we will need
$filenm=$ARGV[0].$ARGV[1].$ARGV[2];
$work='/home/ldm/data';
$temp='/home/ldm/data/current';
$path=$ARGV[0].'/'.$ARGV[1];
# Write out log entry
load_time();
if ($ARGV[0] ne 'HNL')
{
system("echo '$filenm at $currtime' >> ~/data/logs/processlog.txt");
}
sub load_time
{
($month,$day,$hour,$min,$sec)=(localtime)[4,3,2,1,0];
$month=$month+1;
if (length($day)==1) {$day="0".$day; }
if (length($hour)==1) {$hour="0".$hour; }
if (length($min)==1) { $min="0".$min; }
if (length($sec)==1) { $sec="0".$sec; }
$currtime=$month."/".$day."/".$hour.":".$min.":".$sec;
$output=$ARGV[1]."-".$ARGV[2]." at ".$currtime."\n";
}
sub rotate
{
# if ($work/$path) { print "okay"; } else {print "not okay";}
if (($ARGV[1] eq 'TOR') or ($ARGV[1] eq 'SVR'))
{
system("mv $work/$path/$filenm.24.txt $work/$path/$filenm.25.txt");
system("mv $work/$path/$filenm.23.txt $work/$path/$filenm.24.txt");
system("mv $work/$path/$filenm.22.txt $work/$path/$filenm.23.txt");
system("mv $work/$path/$filenm.21.txt $work/$path/$filenm.22.txt");
system("mv $work/$path/$filenm.20.txt $work/$path/$filenm.21.txt");
system("mv $work/$path/$filenm.19.txt $work/$path/$filenm.20.txt");
system("mv $work/$path/$filenm.18.txt $work/$path/$filenm.19.txt");
system("mv $work/$path/$filenm.17.txt $work/$path/$filenm.18.txt");
system("mv $work/$path/$filenm.16.txt $work/$path/$filenm.17.txt");
system("mv $work/$path/$filenm.15.txt $work/$path/$filenm.16.txt");
system("mv $work/$path/$filenm.14.txt $work/$path/$filenm.15.txt");
system("mv $work/$path/$filenm.13.txt $work/$path/$filenm.14.txt");
system("mv $work/$path/$filenm.12.txt $work/$path/$filenm.13.txt");
system("mv $work/$path/$filenm.11.txt $work/$path/$filenm.12.txt");
system("mv $work/$path/$filenm.10.txt $work/$path/$filenm.11.txt");
system("mv $work/$path/$filenm.9.txt $work/$path/$filenm.10.txt");
}
system("mv $work/$path/$filenm.8.txt $work/$path/$filenm.9.txt");
system("mv $work/$path/$filenm.7.txt $work/$path/$filenm.8.txt");
system("mv $work/$path/$filenm.6.txt $work/$path/$filenm.7.txt");
system("mv $work/$path/$filenm.5.txt $work/$path/$filenm.6.txt");
system("mv $work/$path/$filenm.4.txt $work/$path/$filenm.5.txt");
system("mv $work/$path/$filenm.3.txt $work/$path/$filenm.4.txt");
system("mv $work/$path/$filenm.2.txt $work/$path/$filenm.3.txt");
system("mv $work/$path/$filenm.1.txt $work/$path/$filenm.2.txt");
system("cp $temp/$filenm.tmp $work/$path/$filenm.1.txt");
system("cat $temp/$filenm.tmp $work/$path/$filenm.txt >
$work/$path/$filenm.tmp");
system("cp $work/$path/$filenm.tmp $work/$path/$filenm.txt");
system("cp $temp/$filenm.tmp $work/$path/$filenm.1.txt");
system("mv $work/$path/$filenm.tmp $work/$path/$filenm.txt");
# last two commands have been duplicated since sometimes for unknown reasons
# these commands don't always happen for some products
}
sub assemble_html
{
system("~/bin/make_total");
system("~/bin/make_html");
}
############################### Start of main segment
###########################
# Ken Now begin parsing file and decoding observations breaking on cntrl C
$/ = "\cC" ;
# set select processing here from STDIN
START:
while( 1 ) {
open( STDIN, '-' ) ;
vec($rin,fileno(STDIN),1) = 1;
$timeout = 1200 ; # 20 minutes
$nfound = select( $rout = $rin, undef, undef, $timeout );
# timed out
if( ! $nfound ) {
print "Shut down, time out 20 minutes\n" ;
&atexit() ;
}
&atexit( "eof" ) if( eof( STDIN ) ) ;
# Process each line of AFOS bulletins, header first
$_ = <STDIN> ;
s#\cC## ;
s#\cM##g ;
s#\cA\n## ;
s#\c^##g ;
s#\d\d\d \n## ;
#
# extract header info here
#
s#\w{4}\d{1,2} \w{4} (\d{2})(\d{2})(\d{2})?.*\n## ;
$tday = $1 ;
$thour = $2 ;
$thour = "23" if( $thour eq "24" ) ;
$tmin = $3 ;
$tmin = "00" unless( $tmin ) ;
# First, screen off the "exception" products like METARs, TAFs, and Guidance
if ($ARGV[1] eq 'MTR')
{
if ($ARGV[0] eq 'HNL')
{
system("cat $temp/$filenm.tmp $work/PAC/$ARGV[2].txt >
$work/PAC/$ARGV[2].tmp");
system("mv $work/PAC/$ARGV[2].tmp $work/PAC/$ARGV[2].txt");
system("mv $temp/$filenm.tmp $work/PAC/$ARGV[2].1.txt");
}
else
{
system("cat $temp/$filenm.tmp $work/METAR/$ARGV[2].txt >
$work/METAR/$ARGV[2].tmp");
system("mv $work/METAR/$ARGV[2].tmp $work/METAR/$ARGV[2].txt");
system("mv $temp/$filenm.tmp $work/METAR/$ARGV[2].1.txt");
}
}
elsif ($ARGV[1] eq 'TAF')
{
system("cat $temp/$filenm.tmp $work/TAF/TAF$ARGV[2].txt >
$work/TAF/TAF$ARGV[2].tmp");
system("mv $work/TAF/TAF$ARGV[2].tmp $work/TAF/TAF$ARGV[2].txt");
system("mv $temp/$filenm.tmp $work/TAF/TAF$ARGV[2].1.txt");
}
elsif (($ARGV[1] eq 'FWC') or ($ARGV[1] eq 'FAN') or ($ARGV[1] eq 'FMR') or
($ARGV[1] eq 'FRH'))
{
system("mv $temp/$filenm.tmp $work/Guidance/$ARGV[1]$ARGV[2].txt");
}
elsif (($ARGV[1] eq 'RRS') or ($ARGV[1] eq 'RRA'))
{
system("mv $temp/$filenm.tmp $work/$ARGV[0]/$ARGV[1]/$filenm.1.txt");
}
# Now, deal with all the other products
else
{
rotate();
}
################### End of Main Segment ####################
################### Special Segment for additional actions ################
# Special actions for all warnings...
## First, screen for only SR products!
if (($ARGV[0] eq 'ABQ') or ($ARGV[0] eq 'LBB') or ($ARGV[0] eq 'SAT') or
($ARGV[0] eq 'FTW') or ($ARGV[0] eq 'OKC') or ($ARGV[0] eq 'LIT') or
($ARGV[0] eq 'NEW') or ($ARGV[0] eq 'MEM') or ($ARGV[0] eq 'JAN') or
($ARGV[0] eq 'MIA') or ($ARGV[0] eq 'ATL') or ($ARGV[0] eq 'BHM') or
($ARGV[0] eq 'SJU') or ($ARGV[0] eq 'SRH'))
{
if (($ARGV[1] eq 'SVR') or ($ARGV[1] eq 'TOR') or ($ARGV[1] eq 'SVS') or
($ARGV[1] eq 'FLW') or ($ARGV[1] eq 'FLS') or ($ARGV[1] eq 'NOW'))
{
load_time();
open(OFILE,">>/home/ldm/data/warnstat.txt");
print OFILE $output;
close(OFILE);
if ($ARGV[2] eq 'FTW')
{
system("mail -s '$output' kwaters\@imagin.net
<$work/$path/$filenm.1.txt");
}
if (($ARGV[1] eq 'TOR') or ($ARGV[1] eq 'SVR'))
{
system("mail -s '$output' richard.smith\@noaa.gov </home/ldm/blank");
}
if ($ARGV[1] eq 'TOR')
{
system("mail -s '$output' 5268073\@skytel.com </home/ldm/blank");
system("mail -s '$output' kwaters\@imagin.net </home/ldm/blank");
system("echo '<TR><TD>Last Tornado Warning in Southern Region issued by
<A href=\"$ARGV[0]/$ARGV[1]/$ARGV[0]$ARGV[1]$ARGV[2].1.txt\">$ARGV[2]</A>
at:<TD><H3>$month/$day/$hour:$min:$sec<BR>' > ~/data/last_tor.txt");
assemble_html();
system("echo 'ARGV[2]' >> ~/data/logs/tor.txt");
}
if ($ARGV[1] eq 'SVR')
{
system("echo '<TR><TD>Last Severe Thunderstorm Warning in Southern
Region issued by <A
href=\"$ARGV[0]/$ARGV[1]/$ARGV[0]$ARGV[1]$ARGV[2].1.txt\">$ARGV[2]</A>
at:<TD><H3>$month/$day/$hour:$min:$sec<BR>' > ~/data/last_svr.txt");
assemble_html();
system("echo '$ARGV[2]' >> ~/data/logs/svr.txt");
}
if (($ARGV[1] eq 'FLW') or ($ARGV[1] eq 'FLS'))
{
system("mail -s '$output' ben.weiger\@noaa.gov </home/ldm/blank");
system("echo '<TR><TD>Last Flood Warning in Southern Region issued by
<A href=\"$ARGV[0]/$ARGV[1]/$ARGV[0]$ARGV[1]$ARGV[2].1.txt\">$ARGV[2]</A>
at:<TD><H3>$month/$day/$hour:$min:$sec<BR>' > ~/data/last_flw.txt");
assemble_html();
system("echo '$ARGV[2]' >> ~/data/logs/flw.txt");
}
if ($ARGV[1] eq 'NOW')
{
load_time();
system("echo '<TR><TD>Last NOW in Southern Region issued by <A
href=\"/data/$ARGV[0]/$ARGV[1]/$ARGV[0]$ARGV[1]$ARGV[2].1.txt\">$ARGV[2]</A>
received at:<TD><H3>$month/$day/$hour:$min:$sec<BR>' > ~/data/last_now.txt");
assemble_html();
system("echo '$ARGV[2]' >> ~/data/logs/now.txt");
}
}
if ($filenm eq 'SRHADMTST')
{
load_time();
system("echo '<TR><TD>Last Keep-alive message received
at:<TD><H3>$month/$day/$hour:$min:$sec<BR>' > ~/data/last_alive.txt");
assemble_html();
}
if ($ARGV[1] eq 'OAV')
{
system("mail -s 'OAV Issued' judson.ladd\@noaa.gov <$temp/$filenm.tmp");
system("mv $temp/$filenm.tmp $work/OAV/$ARGV[1]$ARGV[2].1.txt");
}
} # end for Southern Region products
# Create the "last 3 versions" copy of ZFPs, AFDs, and NOWs
if (($ARGV[1] eq 'ZFP') or ($ARGV[1] eq 'AFD') or ($ARGV[1] eq 'NOW') or
($ARGV[1] eq 'CLI') or ($ARGV[1] eq 'STP') or ($ARGV[1] eq 'SWR') or
($ARGV[1] eq 'FWF'))
{
system("cat $work/$path/$filenm.1.txt $work/$path/$filenm.2.txt
$work/$path/$filenm.3.txt > $work/$path/$filenm.recent.txt");
}
# If the temporary file's still there then go ahead and delete it
system("rm $temp/$filenm.tmp");
}
#################### End of /home/ldm/process
########################################
