[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #GMU-310745]: Error/unexpected behavior with script called from pqact.c onf



Hi Matthew,

re:
> I'm starting to get a bit more into LDM, and I've run into a snag, and
> wanted to run past you what I'm trying to do with hopes that you can
> perhaps help:

Very good.

> I've got some data being injected into the Ant-IDD (aka LDM) on the
> machine aws.ssec.wisc.edu.  They are simply jpegs of Antarctic
> composite satellite imagery.  I have a machine I'm testing receiving
> them on, penquin.ssec.wisc.edu (yes, it is spelled incorrectly - a
> typo I'm sure...its going to get a new name in a new domain someday
> anyways).

:-)

> When the data arrives on penquin, it it not only filed away on disk to
> a static file, but I'm testing out piping the same product to a script
> to try to maintain a queue of the last 16 jpegs.

OK, you want to maintain a set of 16 images named composite1.jpg, 
composite2.jpg,
..., composite16.jpg

> So my pqact.conf has entries like this:
> 
> EXP
> ANT\.AMRC\.Composite\.infrared\.(.*)\.jpg
> FILE  -overwrite      -close
> data/images/composite.jpg
> EXP
> ANT\.AMRC\.Composite\.infrared\.(........)\.(..)\.jpg
> PIPE  -close
> util/ldmcomploop.sh composite\1.\2.jpg
> 
> The script ldmcomploop.sh is attached to this e-mail.

Thanks this will help.

> and I'm seeing errors in my ldm log like this:
> 
> Nov 14 03:53:57 penquin pqact[22972] ERROR: pbuf_flush (6) write:
> Broken pipe
> Nov 14 03:53:57 penquin pqact[22972] ERROR: pipe_put: -closeutil/
> ldmcomploop.shcomposite20081114.01.jpg write error
> Nov 14 03:53:57 penquin pqact[22972] ERROR: pipe_prodput: trying
> again:   882138 20081114035328.697     EXP 000
> ANT.AMRC.Composite.infrared.20081114.01.jpg
> Nov 14 03:53:57 penquin pqact[22972] ERROR: pbuf_flush (6) write:
> Broken pipe
> Nov 14 03:53:57 penquin pqact[22972] ERROR: pipe_put: -closeutil/
> ldmcomploop.shcomposite20081114.01.jpg write error
> 
> and I'm seeing my script ldmcomploop.sh run twice, which gives me some
> bad results in trying to keep a queue of the last 16 jpegs - with the
> script run twice at the same time, I end up with a mess: I'm expecting
> to see, over time, a nice collection of the last 15 composites --
> composite0,jpg, composite1.jpg, composite2.jpg, etc.  But instead, I'm
> getting composite0.jpg, composite2.jpg, composite4.jpg, etc. due to
> the script being called twice (I've got log file entries that prove
> that).

Here is what is happening:

- your script, ldmcomploop.sh, is not doing anything with the product that
  has been piped to it via pqact's STDOUT.  There is code to do something,
  but it is commented out:

# Write stdin to the designated file and log its filing
#echo cat > /home/ldm/data/images/$1 >>$logfile 2>&1
#cat > /home/ldm/data/images/$1 >>$logfile 2>&1

- 'pqact' checks to see if the product sent to a "decoder" (script in
  your case) has been read by the decoder (i.e., the pipe has been emptied).
  If it has not, then it tries to execute the action one more time.

  This is why you are seeing the script get executed twice.

A quick glance at your script tells me that all you are using it for
is to rename a set of images.

You have a couple/three of options at this point:

- consolidate the two pqact.conf actions into a single one that first
  writes the file and then does the renaming you want

- modify your ldmcomploop.sh script to do something with the product
  so that 'pqact' will see that it has been entirely read:

echo cat > /dev/null

- change the PIPE action to an EXEC action.  EXEC actions run a "decoder"
  but do not provide the product in a pipe

I would suggest that the cleanest approach would be the first one in this
list.  The "worst" action is the second (performs unneeded work however
insignificant).  The third is the easiest to implement:

EXP    ANT\.AMRC\.Composite\.infrared\.(.*)\.jpg
      FILE -overwrite -close data/images/composite.jpg
EXP    ANT\.AMRC\.Composite\.infrared\.(........)\.(..)\.jpg
      EXEC util/ldmcomploop.sh composite.jpg

I recommend the first option which would require the single pattern-action:

EXP    ANT\.AMRC\.Composite\.infrared\.(.*)\.jpg
      PIPE -close util/ldmcomploop.sh data/images/composite.jpg

The operative change to ldmcomploop.sh is:

# Write stdin to the designated file and log its filing
cat > $1

> Do I have something badly setup here that can be fixed?

Not badly setup; just overly complicated.

> I'm not sure
> why I'm getting "broken pipe" and write errors. Have I setup my
> pqact.conf poorly or perhaps the script is poorly setup?

Please see the explanation above.

> What I have done is based off of a conversation/e-mail I had with Tom
> Yoksas a few years back that I never got to act on, and now I am, with
> the script that has a different goal.  I've attached that e-mail
> exchange below.  I've also attached my pqact.conf, my script
> ldmcomploop.sh.  Let me know if you need anything else.

Thanks for attaching the script.  It made seeing what was wrong very easy!

> Thanks for any help!  I appreciate it - being a still "wet behind the
> ears" LDM user.

No worries.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: GMU-310745
Department: Support IDD
Priority: Normal
Status: Closed