[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #ZOC-298338]: Missing data problems



Hi Mark,

re:
> Tom, I'd like to get more information on this multiple pqact instances.

OK.  Please see below.

> I'm thinking it might be a good idea for me to set up multiple
> instances, one for each regions I"m pulling (i.e. 2 per server) to see
> if this has any affect on our downstreams.

I agree.

Here goes:

'pqact' processes actions out of the pattern/action file it uses serially.
Because of this, single instances of 'pqact' running all actions in the
default pattern/action file '~ldm/etc/pqact.conf' can result in excessively
long times for execution of all actions which, in turn, can result in
products not being processed out of one's LDM queue before they get overwritten
by newly received products.  This happens routinely when:

- one's LDM product queue is "small" (small is a relative term)

- one has LOTS of actions in the default pattern/action file

- the actions in the pattern action file do a LOT of FILEing of products

- file system writes on one's machine are "slow" (slow is a relative term)


We first ran into this problem when NEXRAD Level III products from all 150+
Nexrads were added to the IDD.

The solution we (Steve Chiswell) came up with is to divide the actions from
the default pattern/action file into several files and then run a separate
instance of 'pqact' for each of the segmented files.  (Aside: this procedure
is akin to splitting feed requests.)

Here is an example:

1) remove all NEXRAD Level II actions from the default pqact.conf file
   and put them into a new file called ~ldm/etc/pqact_nexrad2.conf (the name
   of the new pattern/action file is unimportant; name it as you like)

2) run multiple instances of 'pqact' for NEXRAD Level II processing.  This
   is done by creating multiple 'exec' lines in ~ldm/etc/ldmd.conf:

exec    "pqact -f ANY-NEXRAD2
exec    "pqact -f NEXRAD2 -p BZIP2/K[A-D] /usr/local/ldm/etc/pqact_nexrad2.conf"
exec    "pqact -f NEXRAD2 -p BZIP2/K[E-K] /usr/local/ldm/etc/pqact_nexrad2.conf"
exec    "pqact -f NEXRAD2 -p BZIP2/K[L-R] /usr/local/ldm/etc/pqact_nexrad2.conf"
exec    "pqact -f NEXRAD2 -p BZIP2/K[S-Z] /usr/local/ldm/etc/pqact_nexrad2.conf"

   Things to note:

   - the first 'exec' line starts an instance of 'pqact' that processes any/all
     feeds EXCEPT NEXRAD2

   - the next 4 'exec' lines each start a 'pqact' that uses actions from a
     single pattern/action file.  One might choose to make different 
pattern/action
     files, but in this case it is likely that you will want to do the exact
     same thing for each Nexrad station

   - the 4 'exec' lines operate on mutually-exclusive subsets of the products
     available in the NEXRAD2 datastream.  This is IMPORTANT since the objective
     is to minimize the amount of work that each 'pqact' has to perform.

After performing the modifications to 'ldmd.conf' and 'pqact.conf' and moving
the NEXRAD2 processing to 'pqact_nexrad2.conf', you will need to:

- check to make sure that you did not make any typos.  You do this using 
'ldmadmin':

  <as 'ldm'>
  cd ~ldm
  ldmadmin pqactcheck                             <- checks ~ldm/etc/pqact.conf 
for syntax errors
  ldmadmin pqactcheck -p etc/pqact_nexrad2.conf   <- checks 
~ldm/etc/pqact_nexrad2.conf

- if you have no typos, you will need to restart your LDM:

  ldmadmin restart

- finally, always check to make sure that there were no typos in 
~ldm/etc/ldmd.conf:

  ldmadmin tail

That's about it.  You will need to fine tune your processing setup based on
the amount of data you are receiving and the speed of the machine doing your
processing.

Please let me know if you have questions on the procedure outlined above.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: ZOC-298338
Department: Support LDM
Priority: Normal
Status: Closed