[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #YCI-227804]: Multiple Matches?

Hi Ben,

re:
> We're trying to do two things with the AWOS surface data that comes in
> off of the LDM.  First, we want to send the data to a processing script
> which will store the data in an SQL database, and second we want to also
> store the information in its raw form in separate files labeled by
> station id.  Below are an excerpt from our pqact.conf file in which we
> attempt this.  I thought this was working (its been this way since
> July), but we recently discovered that there are several files missing
> that we expected to be saving as part of the second action we're taking
> on the data (saving in the raw form).  For example, we aren't seeing
> KLGU.metars, and the data for KLGU isn't in any of the 116
> files/stations that are there.

It sounds like you did not receive data for KLGU.

> Can you offer any help or advice?

I see nothing wrong with your second pqact.conf actions, so I don't have
much, if anything, to offer there.

> I was thinking perhaps the ldm
> didn't like having two exact patterns to match (ie, the first pqact
> entry was "stealing" stuff from the second).

Your pqact.conf can have as many actions that match a product as you like.
The processing of the patterns proceeds from top to bottom through the actions
in the pqact.  The only way that a second (third, fourth, etc.) action would
not be processed is if the product got deleted from the queue before the
product was processed or if the second (or third, fourth, etc) action were
incorrectly formated.  It is not very likely that the product in question
would get deleted between the end of one processing action and another that
immediately follows it in pqact.conf (but it is possible, I guess).

If you are convinced that your first pqact.conf action is processing a product
that is not being processed by a second action, you might review how large
your LDM product queue is to make sure that it is large enough to allow time
for processing products before they get removed by the arrival of new products.
One good way to determine if your product queue is too small is to use the
LDM 'pqmon' utility.  The last piece of information that 'pqmon' prints
is the age of the oldest product in the LDM queue.  If this age is _very_ small
(like a few seconds), then it is likely that products are being scoured out
of your queue before they are processed by all the pqact.conf actions you
have.

Also, if you have LOTS of actions in your pqact.conf file (hundreds or 
thousands)
you may want to run more than one invocation of 'pqact' out of your 
~ldm/etc/ldmd.conf
file specifying different pqact.conf pattern/action files (GEMPAK users do this
routinely).  The key to using more than one pqact.conf file (will need to be
named differently) is to remove processing actions out of one and put it/them
in the other so they don't get executed more than once.

Questions:

- how large is your LDM product queue
- how much data are you ingesting
- how many 'pqact' processes do you have running trying to process your
  data
- how many actions do you have in your pqact.conf file(s)
- what is the age of the oldest product in your LDM queue

> Thanks!

No worries.

> <portion of the pqact.conf:>
> 
> # Metars
> 
> # Append all US metars.
> 
> # This action will slowly consume disk space.
> 
> IDS|DDPLUS    ^SAUS.. .... (..)(..)
> 
> PIPE    -close -strip
> /usr/local/ldm/bin/processMETARnow.pl
> 
> 
> 
> # Archive all records to a single file per station for backup purposes
> 
> IDS|DDPLUS    ^SAUS.. (....)
> 
> FILE     -close data/awos/\1.metar

Assuming that your pqact.conf actions are properly formatted (e.g., use of
tabs instead of spaces where needed), I see nothing wrong/alarming.  If your
processing problem is related to your processMETARnow.pl Perl script, there
is not much we can comment on since we don't know what your script does.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: YCI-227804
Department: Support IDD
Priority: Normal
Status: Closed