[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 19991229: iita LDM failure

Celia,

There are 2 things to do, first when the LDM is stopped,  make sure
that all the ldm processes are gone before restarting the LDM.  Also
check that none of the pqact entries are creating a bunch of extra
processes.  You could comment out the exec pqact line in the ldmd.conf
as a test. If you still have the same problem, have your system admin up
the number of processes that the ldm user can obtain.

Robb...




On Wed, 29 Dec 1999, Unidata Support wrote:

> >To: address@hidden
> >cc: address@hidden (Celia Chen)
> >From: address@hidden (Celia Chen)
> >Subject: iita LDM failure
> >Organization: .
> >Keywords: 199912291753.KAA12810
> 
> We are very puzzled by the failure of LDM on iita last night
> and would like to get some advice from you folks.  
> 
> It looks like there were too many connections to all the 
> downstream servers before LDM died. There were "no more
> processes" left on iita this morning.  I am including part of
> the log file for your reference.  The file name is ldmd.log.1
> if you need to see the entire log file.  
> 
> Thanks in advance.
> 
> Celia
> 
> 
> Dec 29 05:29:08 iita rcsu1(feed)[10486]: topo:  rcsu1.ust.hk WMO
> Dec 29 05:29:09 iita sable[10487]: Connection from sable
> Dec 29 05:29:09 iita sable(feed)[10487]: Starting Up: 19991229043244.417 
> TS_ENDT {{EXP|SEISMIC|GPS|FAA604|DIFAX|NLDN|NMC|FSL|UNIDATA|0xbff00000,  
> ".*"}}
> Dec 29 05:29:09 iita sable(feed)[10487]: topo:  sable 
> EXP|SEISMIC|GPS|FAA604|DIFAX|NLDN|NMC|FSL|UNIDATA|0xbff00000
> Dec 29 05:35:42 iita iita[10532]: Connection from iita.rap.ucar.edu
> Dec 29 05:35:42 iita iita[10532]: Connection reset by peer
> Dec 29 05:35:42 iita iita[10532]: Exiting
> Dec 29 05:36:50 iita iita2[10539]: Connection from iita2.rap.ucar.edu
> Dec 29 05:36:50 iita iita2[10539]: Connection reset by peer
> Dec 29 05:36:50 iita iita2[10539]: Exiting
> Dec 29 05:37:59 iita mm5-dev[10544]: Connection from mm5-dev.rap.ucar.edu
> Dec 29 05:37:59 iita mm5-dev(feed)[10544]: Starting Up: 19991229043842.081 
> TS_ENDT {{ANY,  ".*"}}
> Dec 29 05:37:59 iita mm5-dev(feed)[10544]: topo:  mm5-dev.rap.ucar.edu ANY
> Dec 29 05:38:00 iita anakin[10545]: Connection from anakin.witicorp.com
> Dec 29 05:38:00 iita anakin(feed)[10545]: Starting Up: 19991229041959.273 
> TS_ENDT {{WMO,  ".*"}}
> Dec 29 05:38:00 iita anakin(feed)[10545]: topo:  anakin.witicorp.com WMO
> Dec 29 05:40:41 iita cessna[10561]: Connection from cessna.rap.ucar.edu
> Dec 29 05:40:41 iita cessna(feed)[10561]: Starting Up: 19991229044030.578 
> TS_ENDT {{WMO,  ".*"}}
> Dec 29 05:40:41 iita cessna(feed)[10561]: topo:  cessna.rap.ucar.edu WMO
> 
> Dec 29 05:40:43 iita ldm[10562]: Connection from ldm.comet.ucar.edu
> Dec 29 05:40:43 iita ldm(feed)[10562]: Starting Up: 19991229054028.689 
> TS_ENDT {{MCIDAS,  ".*"},{WMO,  ".*"}}
> Dec 29 05:40:43 iita ldm(feed)[10562]: topo:  ldm.comet.ucar.edu UNIDATA
> Dec 29 05:41:07 iita wetterhorn[10565]: Connection from wetterhorn.ucar.edu
> Dec 29 05:41:07 iita wetterhorn(feed)[10565]: Starting Up: 19991229042249.590 
> TS_ENDT {{FSL2|IDS|HDS|DDS,  "^[A-Z]"}}
> Dec 29 05:41:07 iita wetterhorn(feed)[10565]: topo:  wetterhorn.ucar.edu 
> FSL2|IDS|HDS|DDS
> Dec 29 05:41:07 iita ypg[10568]: Connection from ypg.rap.ucar.edu
> Dec 29 05:41:07 iita ypg(feed)[10568]: Starting Up: 19991229043843.807 
> TS_ENDT {{EXP|SEISMIC|GPS|FAA604|DIFAX|NLDN|NMC|FSL|UNIDATA|0xbff00000,  
> ".*"}}
> Dec 29 05:41:07 iita ypg(feed)[10568]: topo:  ypg.rap.ucar.edu 
> EXP|SEISMIC|GPS|FAA604|DIFAX|NLDN|NMC|FSL|UNIDATA|0xbff00000
> Dec 29 05:41:08 iita rain[10569]: Connection from rain.cosmic.ucar.edu
> Dec 29 05:41:08 iita rain(feed)[10569]: Starting Up: 19991229044054.854 
> TS_ENDT {{WMO,
> ".*"}}
> Dec 29 05:41:08 iita rain(feed)[10569]: topo:  rain.cosmic.ucar.edu WMO
> Dec 29 05:41:08 iita torrent-fddi[10570]: Connection from 
> torrent-fddi.joss.ucar.edu
> Dec 29 05:41:08 iita torrent-fddi(feed)[10570]: Starting Up: 
> 19991229054053.656 TS_ENDT
> {{FSL2|WMO,  ".*"}}
> Dec 29 05:41:08 iita torrent-fddi(feed)[10570]: topo:  
> torrent-fddi.joss.ucar.edu FSL2|W
> Dec 29 05:41:31 iita mop1[10574]: Connection from mop1.eos.ucar.edu
> Dec 29 05:41:31 iita mop1(feed)[10574]: Starting Up: 19991229044114.801 
> TS_ENDT {{WMO,
> ".*"},{MCIDAS,  ".*"}}
> Dec 29 05:41:31 iita mop1(feed)[10574]: topo:  mop1.eos.ucar.edu UNIDATA
> Dec 29 05:41:31 iita front[10576]: Connection from front.rap.ucar.edu
> Dec 29 05:41:31 iita front(feed)[10576]: Starting Up: 19991229054158.608 
> TS_ENDT {{IDS|DDPLUS,  ".*"}}
> Dec 29 05:41:31 iita front(feed)[10576]: topo:  front.rap.ucar.edu IDS|DDPLUS
> Dec 29 05:41:32 iita ofour[10578]: Connection from ofour.rap.ucar.edu
> Dec 29 05:41:32 iita ofour(feed)[10578]: Starting Up: 19991229054158.608 
> TS_ENDT {{WMO,
>  ".*"},{WSI,  "^NEX/(...)/(BREF1|VEL1|VIL|VAD).*"},{WSI,  
> "^NEX/(FTG|CYS|PUX).*"},{WSI,
>  "^NOW.*"},{FSL2,  ".*"}}
> Dec 29 05:41:32 iita ofour(feed)[10578]: topo:  ofour.rap.ucar.edu 
> WSI|FSL2|WMO
> Dec 29 05:41:32 iita box[10579]: Connection from box.mmm.ucar.edu
> Dec 29 05:41:32 iita box(feed)[10579]: Starting Up: 19991229054052.614 
> TS_ENDT {{EXP|SEISMIC|GPS|FAA604|DIFAX|NLDN|NMC|FSL|UNIDATA|0xbff00000,  
> ".*"}}
> Dec 29 05:41:32 iita box(feed)[10579]: topo:  box.mmm.ucar.edu 
> EXP|SEISMIC|GPS|FAA604|DIFAX|NLDN|NMC|FSL|UNIDATA|0xbff00000
> Dec 29 05:41:55 iita aldehyde[10583]: Connection from aldehyde.sprl.umich.edu
> Dec 29 05:41:55 iita aldehyde(feed)[10583]: Starting Up: 19991229054137.212 
> TS_ENDT {{WMO,  ".*"},{FSL2,  ".*"}}
> 
> Dec 29 05:41:55 iita aldehyde(feed)[10583]: topo:  aldehyde.sprl.umich.edu 
> FSL2|WMO
> Dec 29 05:41:56 iita rcsu1[10584]: Connection from rcsu1.ust.hk
> Dec 29 05:41:56 iita rcsu1(feed)[10584]: Starting Up: 19991229044138.709 
> TS_ENDT {{WMO,
>  ".*"}}
> Dec 29 05:41:56 iita rcsu1(feed)[10584]: topo:  rcsu1.ust.hk WMO
> Dec 29 05:41:56 iita blueskies[10585]: Connection from 
> blueskies.sprl.umich.edu
> Dec 29 05:41:56 iita blueskies(feed)[10585]: Starting Up: 19991229054158.615 
> TS_ENDT {{WMO,  ".*"},{FSL2,  ".*"}}
> Dec 29 05:41:56 iita blueskies(feed)[10585]: topo:  blueskies.sprl.umich.edu 
> FSL2|WMO
> Dec 29 05:41:56 iita sable[10586]: Connection from sable
> Dec 29 05:41:56 iita sable(feed)[10586]: Starting Up: 19991229044518.574 
> TS_ENDT {{EXP|SEISMIC|GPS|FAA604|DIFAX|NLDN|NMC|FSL|UNIDATA|0xbff00000,  
> ".*"}}
> Dec 29 05:41:56 iita sable(feed)[10586]: topo:  sable 
> EXP|SEISMIC|GPS|FAA604|DIFAX|NLDN|NMC|FSL|UNIDATA|0xbff00000
> Dec 29 05:50:35 iita mm5-dev[10615]: Connection from mm5-dev.rap.ucar.edu
> Dec 29 05:50:35 iita mm5-dev(feed)[10615]: Starting Up: 19991229045112.361 
> TS_ENDT {{ANY,  ".*"}}
> Dec 29 05:50:35 iita mm5-dev(feed)[10615]: topo:  mm5-dev.rap.ucar.edu ANY
> Dec 29 05:50:36 iita anakin[10616]: Connection from anakin.witicorp.com
> Dec 29 05:50:36 iita anakin(feed)[10616]: Starting Up: 19991229043229.411 
> TS_ENDT {{WMO,  ".*"}}
> Dec 29 05:50:36 iita anakin(feed)[10616]: topo:  anakin.witicorp.com WMO
> 
> Dec 29 05:50:37 iita iita[10620]: Connection from iita.rap.ucar.edu
> Dec 29 05:50:37 iita iita[10620]: Connection reset by peer
> Dec 29 05:50:37 iita iita[10620]: Exiting
> Dec 29 05:51:50 iita iita2[10621]: Connection from iita2.rap.ucar.edu
> Dec 29 05:51:50 iita iita2[10621]: Connection reset by peer
> Dec 29 05:51:50 iita iita2[10621]: Exiting
> Dec 29 05:53:30 iita cessna[10622]: Connection from cessna.rap.ucar.edu
> Dec 29 05:53:30 iita cessna(feed)[10622]: Starting Up: 19991229045311.701 
> TS_ENDT {{WMO,  ".*"}}
> Dec 29 05:53:30 iita cessna(feed)[10622]: topo:  cessna.rap.ucar.edu WMO
> Dec 29 05:53:31 iita ldm[10623]: Connection from ldm.comet.ucar.edu
> Dec 29 05:53:31 iita ldm(feed)[10623]: Starting Up: 19991229055258.532 
> TS_ENDT {{MCIDAS,  ".*"},{WMO,  ".*"}}
> Dec 29 05:53:31 iita ldm(feed)[10623]: topo:  ldm.comet.ucar.edu UNIDATA
> Dec 29 05:53:56 iita wetterhorn[10624]: Connection from wetterhorn.ucar.edu
> Dec 29 05:53:56 iita wetterhorn(feed)[10624]: Starting Up: 19991229043533.574 
> TS_ENDT {{FSL2|IDS|HDS|DDS,  "^[A-Z]"}}
> Dec 29 05:53:56 iita wetterhorn(feed)[10624]: topo:  wetterhorn.ucar.edu 
> FSL2|IDS|HDS|DDS
> Dec 29 05:53:56 iita ypg[10625]: Connection from ypg.rap.ucar.edu
> Dec 29 05:53:56 iita ypg(feed)[10625]: Starting Up: 19991229045127.488 
> TS_ENDT {{EXP|SEISMIC|GPS|FAA604|DIFAX|NLDN|NMC|FSL|UNIDATA|0xbff00000,  
> ".*"}}
> 
> Dec 29 05:53:56 iita ypg(feed)[10625]: topo:  ypg.rap.ucar.edu 
> EXP|SEISMIC|GPS|FAA604|DIFAX|NLDN|NMC|FSL|UNIDATA|0xbff00000
> Dec 29 05:53:57 iita rain[10626]: Connection from rain.cosmic.ucar.edu
> Dec 29 05:53:57 iita rain(feed)[10626]: Starting Up: 19991229045338.756 
> TS_ENDT {{WMO,
> ".*"}}
> Dec 29 05:53:57 iita rain(feed)[10626]: topo:  rain.cosmic.ucar.edu WMO
> Dec 29 05:53:57 iita torrent-fddi[10627]: Connection from 
> torrent-fddi.joss.ucar.edu
> Dec 29 05:53:57 iita torrent-fddi(feed)[10627]: Starting Up: 
> 19991229055332.564 TS_ENDT
> {{FSL2|WMO,  ".*"}}
> Dec 29 05:53:57 iita torrent-fddi(feed)[10627]: topo:  
> torrent-fddi.joss.ucar.edu FSL2|WMO
> Dec 29 05:54:22 iita mop1[10628]: Connection from mop1.eos.ucar.edu
> Dec 29 05:54:22 iita mop1(feed)[10628]: Starting Up: 19991229045401.486 
> TS_ENDT {{WMO,
> ".*"},{MCIDAS,  ".*"}}
> Dec 29 05:54:22 iita mop1(feed)[10628]: topo:  mop1.eos.ucar.edu UNIDATA
> Dec 29 05:54:22 iita front[10629]: Connection from front.rap.ucar.edu
> Dec 29 05:54:22 iita front(feed)[10629]: Starting Up: 19991229055358.278 
> TS_ENDT {{IDS|DDPLUS,  ".*"}}
> Dec 29 05:54:22 iita front(feed)[10629]: topo:  front.rap.ucar.edu IDS|DDPLUS
> Dec 29 05:54:22 iita rpc.ldmd[380]: fork: Resource temporarily unavailable
> Dec 29 05:54:22 iita ofour[10630]: Connection from ofour.rap.ucar.edu
> Dec 29 05:54:22 iita ofour(feed)[10630]: Starting Up: 19991229055358.278 
> TS_ENDT {{WMO,
> 
>  ".*"},{WSI,  "^NEX/(...)/(BREF1|VEL1|VIL|VAD).*"},{WSI,  
> "^NEX/(FTG|CYS|PUX).*"},{WSI,
>  "^NOW.*"},{FSL2,  ".*"}}
> Dec 29 05:54:22 iita ofour(feed)[10630]: topo:  ofour.rap.ucar.edu 
> WSI|FSL2|WMO
> Dec 29 05:54:26 iita rpc.ldmd[380]: fork: Resource temporarily unavailable
> Dec 29 05:54:27 iita last message repeated 3 times
> Dec 29 05:57:02 iita rpc.ldmd[380]: fork: Resource temporarily unavailable
> Dec 29 05:58:30 iita rpc.ldmd[380]: fork: Resource temporarily unavailable
> Dec 29 05:58:36 iita last message repeated 4 times
> Dec 29 06:00:09 iita last message repeated 2 times
> Dec 29 06:01:32 iita rpc.ldmd[380]: fork: Resource temporarily unavailable
> Dec 29 06:02:40 iita last message repeated 4 times
> Dec 29 06:03:06 iita last message repeated 3 times
> Dec 29 06:04:36 iita last message repeated 3 times
> Dec 29 06:06:00 iita rpc.ldmd[380]: fork: Resource temporarily unavailable
> Dec 29 06:06:52 iita last message repeated 16 times
> Dec 29 06:07:57 iita last message repeated 9 times
> Dec 29 06:09:06 iita last message repeated 10 times
> Dec 29 06:09:57 iita last message repeated 7 times
> Dec 29 06:11:11 iita last message repeated 13 times
> Dec 29 06:12:06 iita last message repeated 7 times
> Dec 29 06:12:52 iita last message repeated 7 times
> Dec 29 06:13:57 iita last message repeated 9 times
> 
> .
> .
> .
> Dec 29 07:57:57 iita last message repeated 8 times
> Dec 29 07:58:58 iita last message repeated 6 times
> Dec 29 08:00:10 iita last message repeated 10 times
> Dec 29 08:00:26 iita rpc.ldmd[380]: gethostbyaddr: failed for 128.117.64.23
> Dec 29 08:00:26 iita rpc.ldmd[380]: Denying connection from 128.117.64.23
> Dec 29 08:00:27 iita rpc.ldmd[380]: gethostbyaddr: failed for 128.117.200.89
> Dec 29 08:00:27 iita rpc.ldmd[380]: fork: Resource temporarily unavailable
> Dec 29 08:00:28 iita rpc.ldmd[380]: accept: Too many open files
> Dec 29 08:01:01 iita last message repeated 9 times
> Dec 29 08:02:02 iita last message repeated 25 times
> 
> .
> .
> .
> 
> Dec 29 14:04:20 iita last message repeated 32 times
> Dec 29 14:05:20 iita last message repeated 32 times
> Dec 29 14:05:42 iita last message repeated 13 times
> Dec 29 14:05:48 iita rpc.ldmd[380]: accept: Too many open files
> Dec 29 14:06:19 iita last message repeated 17 times
> .
> .
> .
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================