[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

19990610: ldm crashes



Mike,

The "Deleting Oldest Product" messages show that your data queue is
too small. When we changed to NOAAport in December, the amount of
data on the IDD increased. You might try defining your product
queue size ~200MB in ~ldmadmin, deleting the old queue and
then remaking the queue. Running with a queue too small causes the
LDM to be very inefficient since it has to scour for every product
instead of once every 5 minutes.

The message "Assertion failed" shows that your product queue is corrupt.
This could be a problem with pqexpire trying to scour out old data at the
same time the rpc.ldmd is trying to "Delete oldest product".
Making a bigger queue should releive this conflict.

A little peculiarity is your pqact.conf entry for:
Realtime/nwx/pubprod/city/(US:yy)(US:mm)US08.city
It looks like your pqact action for this product is munged.
You should be expecting something like:
WMO     ^FP(US|AK|GM)4[0-9] .... ([0-3][0-9])([0-2][0-9])
        FILE    data/gempak/nwx/pubprod/city/(\2:yy)(\2:mm)\2\3.city

It looks like you have \1and \2 instead of \2 and \3, so you
are getting the US match instead of the DD and HH fields.

Steve Chiswell
Unidata User SUpport




>From: Mike Trexler <address@hidden>
>Organization: .
>Keywords: 199906101809.MAA06904

>This is a multi-part message in MIME format.
>--------------4A578704579B8BAEBDA36513
>Content-Type: text/plain; charset=us-ascii
>Content-Transfer-Encoding: 7bit
>
>Hi,
>
>A couple months ago I had a problem with the ldm mysteriously
>crashing and killing itself for no apparent reason. Additionally,
>when you would start it back up it gave a funny message, but still
>seemed to start up fine.  You guys gave my a few things to check 
>out, but those things really showed no clues as to what was going
>on.  Well, then the ldm began humming along fine on it's own and
>has been doing well for the last couple of months....up until
>about a week or so ago when it began crashing, again for no apparent
>reason.  I tried restarting (including deleting and making a new 
>queue) and then it would run for about a day and a half and then
>crash again.  I have included two bits of info for you that might
>help you in making a diagnosis.  The first the output from:
>ldmadmin check
>here it is....
>
>--------------
>LDM status report from the logs for the last 28 hours.
>
>Currently meaculpa.nrrc.ncsu.edu is running 100 percent idle
>load average: 0.13, 0.37, 0.59
>Running version number Unknown.
>LDM was restarted 1 time(s)
>        Last LDM restart at Jun 07 15:08:08
>
>
>Critical LDM problems that need immediate attention:
>assertion failure message occurred 1 time(s).
>        Last one at:  Jun 08 19:42:46
>        For pqexpire[20721] it happened 1 time(s).
>
>Potential LDM Problems:
>Non-zero Status message occurred 90 time(s).
>        Last one at:  Jun 08 18:31:23
>'Breaking connection' message occurred 74 time(s).
>        Last one at:  Jun 08 19:40:22
>        For [20724] it happened 27 time(s).
>        For er[20725] it happened 8 time(s).
>        For ng[20726] it happened 39 time(s).
>'RPC: Program not registered' message occurred 3 time(s).
>        Last one at:  Jun 08 19:15:42
>        For RPC it happened 3 time(s).
>'NULLPROC error' message occurred 8 time(s).
>        Last one at:  Jun 07 21:38:56
>        For er[20725] it happened 2 time(s).
>        For ng[20726] it happened 6 time(s).
>
>Decoder LDM Problems:
>--------------
>
>
>I have also attached a text file which is the end of the ldm
>log file.  If you need any more info, just let me know and I'll
>get it for you.  Thanks for your help!
>
>Mike Trexler
>
>
>-- 
>__________________________________________________
>**************************************************
>******          C. Michael Trexler          ******
>******      N C State Univ. - Box 8208      ****** 
>******  Dept. of Mar/Earth/Atmos. Sciences  ******
>******        Research III   Rm. 100        ******
>******        Raleigh, NC 27695-8208        ****** 
>******        (919) 515-1447  Phone         ******   
>******         (919) 515-1683  Fax          ******
>**************************************************
>--------------------------------------------------
>******        address@hidden       ******
>__________________________________________________
>--------------4A578704579B8BAEBDA36513
>Content-Type: text/plain; charset=us-ascii; name="log"
>Content-Transfer-Encoding: 7bit
>Content-Disposition: inline; filename="log"
>
>Jun 08 19:05:57 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 1040 bytes
>Jun 08 19:05:57 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 2080 bytes
>Jun 08 19:05:58 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 2848 bytes
>Jun 08 19:05:58 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 6864 bytes
>Jun 08 19:05:58 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 8160 bytes
>Jun 08 19:05:59 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 1800 bytes
>Jun 08 19:05:59 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 2928 bytes
>Jun 08 19:06:00 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 1728 bytes
>Jun 08 19:06:01 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 24968 bytes
>Jun 08 19:06:01 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 18192 bytes
>Jun 08 19:06:03 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 8488 bytes
>Jun 08 19:06:04 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 20336 bytes
>Jun 08 19:06:06 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 19272 bytes
>Jun 08 19:06:08 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 1080 bytes
>Jun 08 19:06:09 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 19832 bytes
>Jun 08 19:06:10 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 216 bytes
>Jun 08 19:06:12 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 1664 bytes
>Jun 08 19:06:12 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 336 bytes
>Jun 08 19:06:12 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 5392 bytes
>Jun 08 19:06:14 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 2008 bytes
>Jun 08 19:06:14 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 2224 bytes
>Jun 08 19:06:15 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 664 bytes
>Jun 08 19:06:17 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 1384 bytes
>Jun 08 19:06:17 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 632 bytes
>Jun 08 19:06:18 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 1928 bytes
>Jun 08 19:06:19 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 1720 bytes
>Jun 08 19:06:20 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 3040 bytes
>Jun 08 19:06:20 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 1720 bytes
>Jun 08 19:06:21 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 20512 bytes
>Jun 08 19:06:22 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 3768 bytes
>Jun 08 19:06:22 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 1888 bytes
>Jun 08 19:06:25 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 1856 bytes
>Jun 08 19:06:25 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 2936 bytes
>Jun 08 19:06:26 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 3752 bytes
>Jun 08 19:06:28 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 1168 bytes
>Jun 08 19:06:28 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 48072 bytes
>Jun 08 19:06:29 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 16104 bytes
>Jun 08 19:06:29 meaculpa.nrrc.ncsu.edu lwtoa3[10448]: PRODUCT CODE=U3         
>  99159       190621
>Jun 08 19:06:30 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 3592 bytes
>Jun 08 19:06:30 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 2488 bytes
>Jun 08 19:06:31 meaculpa.nrrc.ncsu.edu lwtoa3[10448]:  Done -- AREA= 209
>Jun 08 19:06:31 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 808 bytes
>Jun 08 19:06:33 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 248 bytes
>Jun 08 19:06:34 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 7072 bytes
>Jun 08 19:06:35 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 584 bytes
>Jun 08 19:06:37 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 7360 bytes
>Jun 08 19:06:46 meaculpa.nrrc.ncsu.edu striker[20725]: Deleting oldest to make
>  space 43288 bytes
>Jun 08 19:06:49 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 20064 bytes
>Jun 08 19:06:51 meaculpa.nrrc.ncsu.edu pqact[20723]: pbuf_flush 4: time elapse
> d   2.802573
>Jun 08 19:07:19 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 28496 bytes
>Jun 08 19:07:50 meaculpa.nrrc.ncsu.edu sysu1[18271]: Deleting oldest to make s
> pace 159320 bytes
>Jun 08 19:09:17 meaculpa.nrrc.ncsu.edu pluto[20724]: Connection reset by peer
>Jun 08 19:09:17 meaculpa.nrrc.ncsu.edu pluto[20724]: Disconnect
>Jun 08 19:09:33 meaculpa.nrrc.ncsu.edu pqexpire[20721]: > Recycled  20243.107 
> kb/hr (  5435.050 prods per hour)
>Jun 08 19:09:47 meaculpa.nrrc.ncsu.edu pluto[20724]: run_requester: 1999060818
> 1812.357 TS_ENDT {{UNIDATA,  ".*"}}
>Jun 08 19:09:48 meaculpa.nrrc.ncsu.edu pluto[20724]: FEEDME(pluto.met.fsu.edu)
> : RPC: Program not registered
>Jun 08 19:09:59 meaculpa.nrrc.ncsu.edu pqact[20723]: bad day of month in ident
> : /newdata/gempak/Realtime/nwx/pubprod/city/(US:yy)(US:mm)US08.city
>Jun 08 19:09:59 meaculpa.nrrc.ncsu.edu pqact[20723]: bad day of month in ident
> : /newdata/gempak/Realtime/nwx/pubprod/city/(US:yy)(US:mm)US08.city
>Jun 08 19:10:18 meaculpa.nrrc.ncsu.edu pluto[20724]: FEEDME(pluto.met.fsu.edu)
> : OK
>Jun 08 19:10:42 meaculpa.nrrc.ncsu.edu pluto[20724]: Connection reset by peer
>Jun 08 19:10:42 meaculpa.nrrc.ncsu.edu pluto[20724]: Disconnect
>Jun 08 19:11:12 meaculpa.nrrc.ncsu.edu pluto[20724]: run_requester: 1999060818
> 1731.213 TS_ENDT {{UNIDATA,  ".*"}}
>Jun 08 19:11:12 meaculpa.nrrc.ncsu.edu pluto[20724]: FEEDME(pluto.met.fsu.edu)
> : RPC: Program not registered
>Jun 08 19:11:42 meaculpa.nrrc.ncsu.edu pluto[20724]: FEEDME(pluto.met.fsu.edu)
> : OK
>Jun 08 19:11:56 meaculpa.nrrc.ncsu.edu sysu1[18271]: Deleting oldest to make s
> pace 158312 bytes
>Jun 08 19:15:12 meaculpa.nrrc.ncsu.edu pluto[20724]: Connection reset by peer
>Jun 08 19:15:12 meaculpa.nrrc.ncsu.edu pluto[20724]: Disconnect
>Jun 08 19:15:42 meaculpa.nrrc.ncsu.edu pluto[20724]: run_requester: 1999060818
> 1938.011 TS_ENDT {{UNIDATA,  ".*"}}
>Jun 08 19:15:42 meaculpa.nrrc.ncsu.edu pluto[20724]: FEEDME(pluto.met.fsu.edu)
> : RPC: Program not registered
>Jun 08 19:17:05 meaculpa.nrrc.ncsu.edu lwtoa3[19269]: PRODUCT CODE=UB         
>  99159       191641
>Jun 08 19:17:10 meaculpa.nrrc.ncsu.edu lwtoa3[19269]:  Done -- AREA= 170
>Jun 08 19:17:12 meaculpa.nrrc.ncsu.edu pluto[20724]: FEEDME(pluto.met.fsu.edu)
> : OK
>Jun 08 19:17:55 meaculpa.nrrc.ncsu.edu lwtoa3[20057]: PRODUCT CODE=U5         
>  99159       191642
>Jun 08 19:18:18 meaculpa.nrrc.ncsu.edu lwtoa3[20057]:  Done -- AREA= 131
>Jun 08 19:18:33 meaculpa.nrrc.ncsu.edu lwtoa3[20580]: PRODUCT CODE=U9         
>  99159       191641
>Jun 08 19:19:17 meaculpa.nrrc.ncsu.edu lwtoa3[20580]:  Done -- AREA= 121
>Jun 08 19:19:24 meaculpa.nrrc.ncsu.edu pqact[20723]: pbuf_flush 35: time elaps
> ed   3.819870
>Jun 08 19:19:40 meaculpa.nrrc.ncsu.edu pqact[20723]: bad day of month in ident
> : /newdata/gempak/Realtime/nwx/pubprod/city/(US:yy)(US:mm)US08.city
>Jun 08 19:19:40 meaculpa.nrrc.ncsu.edu pqact[20723]: bad day of month in ident
> : /newdata/gempak/Realtime/nwx/pubprod/city/(US:yy)(US:mm)US08.city
>Jun 08 19:20:01 meaculpa.nrrc.ncsu.edu pqexpire[20721]: > Recycled  20806.617 
> kb/hr (  5645.174 prods per hour)
>Jun 08 19:23:56 meaculpa.nrrc.ncsu.edu ihlseng[20726]: RECLASS: 19990608182356
> .423 TS_ENDT {{MCIDAS|IDS|HDS|DDPLUS,  ".*"}}
>Jun 08 19:23:56 meaculpa.nrrc.ncsu.edu ihlseng[20726]: skipped: 19990608182256
> .205 (60.218 seconds)
>Jun 08 19:25:26 meaculpa.nrrc.ncsu.edu ihlseng[20726]: RECLASS: 19990608182526
> .617 TS_ENDT {{MCIDAS|IDS|HDS|DDPLUS,  ".*"}}
>Jun 08 19:25:26 meaculpa.nrrc.ncsu.edu ihlseng[20726]: skipped: 19990608182426
> .266 (60.351 seconds)
>Jun 08 19:26:57 meaculpa.nrrc.ncsu.edu pqact[20723]: bad day of month in ident
> : /newdata/gempak/Realtime/nwx/pubprod/city/(US:yy)(US:mm)US08.city
>Jun 08 19:26:57 meaculpa.nrrc.ncsu.edu pqact[20723]: bad day of month in ident
> : /newdata/gempak/Realtime/nwx/pubprod/city/(US:yy)(US:mm)US08.city
>Jun 08 19:27:14 meaculpa.nrrc.ncsu.edu ihlseng[20726]: RECLASS: 19990608182714
> .356 TS_ENDT {{MCIDAS|IDS|HDS|DDPLUS,  ".*"}}
>Jun 08 19:27:14 meaculpa.nrrc.ncsu.edu ihlseng[20726]: skipped: 19990608182612
> .401 (61.955 seconds)
>Jun 08 19:28:40 meaculpa.nrrc.ncsu.edu pqact[20723]: pbuf_flush 10: time elaps
> ed   2.168467
>Jun 08 19:28:50 meaculpa.nrrc.ncsu.edu ihlseng[20726]: RECLASS: 19990608182850
> .806 TS_ENDT {{MCIDAS|IDS|HDS|DDPLUS,  ".*"}}
>Jun 08 19:28:50 meaculpa.nrrc.ncsu.edu ihlseng[20726]: skipped: 19990608182750
> .795 (60.011 seconds)
>Jun 08 19:31:30 meaculpa.nrrc.ncsu.edu lwtoa3[921]: PRODUCT CODE=UX          9
> 9159       193108
>Jun 08 19:31:32 meaculpa.nrrc.ncsu.edu lwtoa3[921]:  Done -- AREA= 108
>Jun 08 19:31:38 meaculpa.nrrc.ncsu.edu ihlseng[20726]: RECLASS: 19990608183138
> .780 TS_ENDT {{MCIDAS|IDS|HDS|DDPLUS,  ".*"}}
>Jun 08 19:31:38 meaculpa.nrrc.ncsu.edu ihlseng[20726]: skipped: 19990608183038
> .692 (60.088 seconds)
>Jun 08 19:32:34 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 497632 bytes
>Jun 08 19:32:41 meaculpa.nrrc.ncsu.edu lwtoa3[1871]: PRODUCT CODE=UA          
> 99159       193216
>Jun 08 19:32:49 meaculpa.nrrc.ncsu.edu lwtoa3[1871]:  Done -- AREA= 164
>Jun 08 19:33:04 meaculpa.nrrc.ncsu.edu pqact[20723]: pbuf_flush 10: time elaps
> ed   2.149141
>Jun 08 19:34:36 meaculpa.nrrc.ncsu.edu lwtoa3[4409]:  Done -- AREA= 63
>Jun 08 19:36:29 meaculpa.nrrc.ncsu.edu lwtoa3[7187]:  Done -- AREA= 110
>Jun 08 19:36:29 meaculpa.nrrc.ncsu.edu pqact[20723]: pbuf_flush (4) write: Bro
> ken pipe
>Jun 08 19:36:29 meaculpa.nrrc.ncsu.edu pqact[20723]: pipe_dbufput: bin/lwtoa3-
> d/newdata/mcidas write error
>Jun 08 19:36:29 meaculpa.nrrc.ncsu.edu pqact[20723]: pipe_prodput: trying agai
> n
>Jun 08 19:36:31 meaculpa.nrrc.ncsu.edu lwtoa3[7219]:  Done -- AREA= 110
>Jun 08 19:36:31 meaculpa.nrrc.ncsu.edu pqact[20723]: pbuf_flush (4) write: Bro
> ken pipe
>Jun 08 19:36:31 meaculpa.nrrc.ncsu.edu pqact[20723]: pipe_dbufput: bin/lwtoa3-
> d/newdata/mcidas write error
>Jun 08 19:37:00 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 1413496 bytes
>Jun 08 19:37:18 meaculpa.nrrc.ncsu.edu lwtoa3[8675]: PRODUCT CODE=UI          
> 99159       193546
>Jun 08 19:37:37 meaculpa.nrrc.ncsu.edu lwtoa3[8995]: PRODUCT CODE=UW          
> 99159       193654
>Jun 08 19:37:38 meaculpa.nrrc.ncsu.edu lwtoa3[8675]:  Done -- AREA= 152
>Jun 08 19:37:41 meaculpa.nrrc.ncsu.edu lwtoa3[8995]:  Done -- AREA= 216
>Jun 08 19:38:17 meaculpa.nrrc.ncsu.edu pluto[20724]: Connection reset by peer
>Jun 08 19:38:17 meaculpa.nrrc.ncsu.edu pluto[20724]: Disconnect
>Jun 08 19:38:19 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Deleting oldest to make
>  space 1964896 bytes
>Jun 08 19:40:22 meaculpa.nrrc.ncsu.edu pluto[20724]: run_requester: 1999060819
> 3708.342 TS_ENDT {{UNIDATA,  ".*"}}
>Jun 08 19:40:22 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Connection reset by pee
> r
>Jun 08 19:40:22 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Disconnect
>Jun 08 19:40:22 meaculpa.nrrc.ncsu.edu pluto[20724]: FEEDME(pluto.met.fsu.edu)
> : OK
>Jun 08 19:40:52 meaculpa.nrrc.ncsu.edu ihlseng[20726]: run_requester: 19990608
> 193708.342 TS_ENDT {{UNIDATA,  ".*"}}
>Jun 08 19:40:52 meaculpa.nrrc.ncsu.edu ihlseng[20726]: FEEDME(ihlseng.ugems.ps
> u.edu): reclass: 19990608193708.342 TS_ENDT {{MCIDAS|IDS|HDS|DDPLUS,  "
>Jun 08 19:40:52 meaculpa.nrrc.ncsu.edu ihlseng[20726]: FEEDME(ihlseng.ugems.ps
> u.edu): OK
>Jun 08 19:40:53 meaculpa.nrrc.ncsu.edu ihlseng[20726]: 19114e7b5bbed2947edec53
> e3367f17b: never completed
>Jun 08 19:41:07 meaculpa.nrrc.ncsu.edu lwtoa3[9805]: PRODUCT CODE=UV          
> 99159       193656
>Jun 08 19:41:32 meaculpa.nrrc.ncsu.edu lwtoa3[9805]:  Done -- AREA= 140
>Jun 08 19:42:46 meaculpa.nrrc.ncsu.edu pqexpire[20721]: assertion "info->sz <=
>  xdrs->x_handy" failed: file "pq.c", line 4021
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu rpc.ldmd[20720]: child 20721 terminated
>  by signal 6
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu rpc.ldmd[20720]: Killing (SIGINT) proce
> ss group
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu pqact[20723]: Interrupt
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu pluto[20724]: Interrupt
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu pluto[20724]: Exiting
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu pqact[20723]: Exiting
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu rpc.ldmd[20720]: Interrupt
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu rpc.ldmd[20720]: Exiting
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Interrupt
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu ihlseng[20726]: Exiting
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu pqbinstats[20722]: Interrupt
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu sysu1[18271]: Interrupt
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu sysu1[18271]: Exiting
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu DCNLDN[25839]: Interrupt Signal
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu striker[20725]: Interrupt
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu DCSYNOP[6415]: Interrupt Signal
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu striker[20725]: Exiting
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu DCUAIR[21402]: Interrupt Signal
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu udp.ldmd[20727]: Interrupt
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu DCSYNOP[27362]: Interrupt Signal
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu pqbinstats[20722]: Exiting
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu rpc.ldmd[20720]: Terminating process gr
> oup
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu udp.ldmd[20727]: Exiting
>Jun 08 19:42:53 meaculpa.nrrc.ncsu.edu DCHRLY[19741]: Terminate Signal
>
>--------------4A578704579B8BAEBDA36513--
>