[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20030522: LDM upgrade on Plymouth State machines (cont.)

>From: Jim Koermer <address@hidden>
>Organization: Plymouth State
>Keywords: 200305212159.h4LLxclL042738 LDM IDD pqing Unisys NOAAPORT

Hi Jim,

>The NNEXRAD data (all level III) products for all 154 sites comes in via
>the pqing process from 158.136.73.144. It usually comes in a a big
>spurt, once every minute. Since I wasn't getting anything, I set things
>up to get a subset of the products from the nsf.gov computer.

I took a look at the products coming into your system to see what was
going on.  The NNEXRAD products from the IDD are typically between 4000
and 40000 bytes in length.  What I saw on your system was products that
look like this coming form atm, but looking quite a bit different
coming from nooaport4.  Here is one example taken from a notifyme that
was looking for patterns that match /pN0R:

May 22 17:10:29 notifyme[81496]: 9d224e3cb5d6c4161c8d0c08bcd5cbee     6240 
20030522165231.429 NNEXRAD 14525614  SDUS51 KCLE 221652 /pN0RCLE
May 22 17:10:29 notifyme[81496]: 64a05b0e555b155a5d654737b09e4289  1995613 
20030522165231.748 NNEXRAD 745  SDUS54 KBMX 221650 /pN0RBMX
May 22 17:10:29 notifyme[81496]: fe8b5b899c072bc6baf794e5cfb951f8    14605 
20030522165237.649 NNEXRAD 14525656  SDUS53 KDTX 221651 /pN0RDTX

The product from your NOAAPORT ingest is the one with the header:

NNEXRAD 745  SDUS54 KBMX 221650 /pN0RBMX

Its size is much much larger than that for any Level III product from
any NEXRAD, 1995613.  The sizes for N0R produts coming from atm are in
the expected size range: 6240, and 14605.

There was a bug in pqing that got fixed some time ago, but never got
generally released (it was in LDM-5.3.0) except to SSEC NOAAPORT ingest
systems.  It looks like the ingest of NNEXRAD data from your Unisys
NOAAPORT was working using the old pqing program, and upgrading pqing
to the latest release (which contained the fix for the bug I referred
to) made it stop working.  The question is why.

To test the hypothesis about pqing versions, I decided to do the following:

- stop the LDM

- cd ~ldm/bin (which is ldm-6.0.11/bin) and rename pqing to pqing.6.0.11

- copy the 5.1.4 version of pqing to ~ldm/bin

- comment out the ingest of NNEXRAD data from atm

- restart the LDM

Then, I saw a lot of products being read from your NOAAPORT machine and
put into the LDM queue.  Whether or not these products are totally
correct (meaning that they are complete but do not contain more than
one product's worth of data) was unknown to me until we took a harder
look.

What we were seeing is the NNEXRAD products coming from the pqing
ingest from noaaport4 do not look like what the current implementation
of pqing is looking for,  pqing is looking for the end of product
sequence, CR CR NL ETX, followed immediately by the beginning of the
next product, SOH CR CR NL.  The products from your NOAAPORT system are
very close to this, but not exact.  They are the end of product
sequence, CR CR NL ETX followed most of the time by a NL, and then
followed by the start of the new product, SOH CR CR NL.  We think that
this is some sort of a bug in the Unisys code that is putting the FOS
wrappers on the product.

>After lunch, I'll turn the NNEXRAD feed from nsf.gov off again to see if
>anything is coming in from 158.136.73.144.

Data was coming in from 158.136.73.144 (aka noaaport4.plymouth.edu),
but the size of the products indicates that several products were being
combined into single, large ones.  Using the 5.1.4 version of pqing,
and turning off NNEXRAD ingest from atm, we once again saw products
coming in from noaaport4, and they come in spurts as you noted above.

We then made a one line change to the 6.0.11 version of pqing that
would accept one NL between the end of product and beginning of product
sequences.  That new version is running on pscwx right now and is
working correctly.

We feel that Unisys should be contacted to see if their system's
product sequencing can be changed to remove the extraneous NL
character.  Again, since the NL is not between every product, it is
possible that the inconsistency is an indication of some other bug in
their code.  Also, we trapped an example where the Unisys system
had a garbled end of product, and that resulted in two products
being combined and put into the LDM queue.

I left your system setup to just ingest NNEXRAD data using pqing.
I setup a single line request to atm for NNEXRAD using the ALTERNATE
tag I referred to in a previous email.  I suggest using this line
if/when you turn on IDD NNEXRAD ingest to atm, and not the three
line split request that is also in ldmd.conf but commented out.

Tom