[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #SJZ-870882]: Problem ingesting archive files

Hi Jim,

re:
> As Murphy's Law has it, Brendon went on vacation yesterday... I'm down in
> Florida again this summer, working at CCAFS/KSC.

Wouldn't you know!

re:
> Here is what I get from running "pqing -l - -q /usr/local/ldm/data/ldm.pq -f 
> DDPLUS SURFACE_DDPLUS_20120720_0000.txt"
> 
> Jul 20 18:18:41 pqing[26176] NOTE: Starting Up
> Jul 20 18:18:41 pqing[26176] NOTE: FILE "SURFACE_DDPLUS_20120720_0000.txt"
> Jul 20 18:18:41 pqing[26176] ERROR: Lone SOH error 140 374 SRCA20 KWAL 200000
> Jul 20 18:18:41 pqing[26176] ERROR: Not a WMO format message. 88 -01 nb"a00 
> oA4F 201818
> Jul 20 18:18:41 pqing[26176] ERROR: Not a WMO format message. 101 -01 a8v05 . 
> 201818
> Jul 20 18:18:41 pqing[26176] ERROR: Lone SOH error 89 126 SRCO20 KWAL 200002
> Jul 20 18:18:41 pqing[26176] ERROR: Not a WMO format message. 52 -01 QK A00 
> BIP 201818
> Jul 20 18:18:41 pqing[26176] ERROR: Lone ETX error 25 -01 ): K00 ( 201818
> Jul 20 18:18:41 pqing[26176] ERROR: Lone SOH error 84 493 SRID20 KWAL 200003
 ...
> Jul 20 18:18:49 pqing[26176] ERROR: Lone ETX error 44 -01 mN9609 j>7 201818
> Jul 20 18:18:49 pqing[26176] ERROR: Lone ETX error 17 -01 +0NP16 4W 201818
> Jul 20 18:18:49 pqing[26176] NOTE: Exiting
> Jul 20 18:18:49 pqing[26176] NOTE: Queue usage (bytes):1000001536
> Jul 20 18:18:49 pqing[26176] NOTE: (nregions): 244139
> Jul 20 18:18:49 pqing[26176] NOTE: Duplicates rejected: 0
> Jul 20 18:18:49 pqing[26176] NOTE: WMO Messages seen: 39658
> Jul 20 18:18:49 pqing[26176] NOTE: SOH/ETX missing : 53
> Jul 20 18:18:49 pqing[26176] NOTE: parity/chksum err: 0
> Jul 20 18:18:49 pqing[26176] NOTE: WMO format errors: 31
> Jul 20 18:18:49 pqing[26176] NOTE: FILE Bytes read: 43630282
> 
> I'm not sure if all the WMO errors are for the surface data, but the input 
> text file looks fine to me.

The number in square brackets is the process ID for the process doing the 
logging.
It sure looks to me like there are missing SOH (start of header) and ETX (end 
of text)
characters missing in the file pqing is trying to insert.  I would say that is 
the
cause of the problems.

re:
> Here are the results if I change the feed to ANY:
> 
> Jul 20 18:27:52 pqing[2691] NOTE: Starting Up
> Jul 20 18:27:52 pqing[2691] NOTE: FILE "SURFACE_DDPLUS_20120720_0000.txt"
> Jul 20 18:27:52 pqing[2691] NOTE: Exiting
> Jul 20 18:27:52 pqing[2691] NOTE: Queue usage (bytes):1000001536
> Jul 20 18:27:52 pqing[2691] NOTE: (nregions): 244139
> Jul 20 18:27:52 pqing[2691] NOTE: Duplicates rejected: 39627
> Jul 20 18:27:52 pqing[2691] NOTE: WMO Messages seen: 39683
> Jul 20 18:27:52 pqing[2691] NOTE: SOH/ETX missing : 0
> Jul 20 18:27:52 pqing[2691] NOTE: parity/chksum err: 0
> Jul 20 18:27:52 pqing[2691] NOTE: WMO format errors: 0
> Jul 20 18:27:52 pqing[2691] NOTE: FILE Bytes read: 43630282
> 
> Do I need to rebuild the queue?

It doesn't look like there is anything wrong with the LDM queue, so there
is no pressing need to rebuild it.  That being said, deleting and remaking
the queue is such a simple process that the "it can't hurt" approach
should not be ignored:

<as 'ldm'>

ldmadmin stop && ldmadmin delqueue && ldmadmin mkqueue && ldmadmin start

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: SJZ-870882
Department: Support LDM
Priority: Normal
Status: Closed