[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #MKY-400850]: Continuing issue with no TIRC headers getting into secondary LDM box



Gilbert,

I got on to Ldm01 and Bird01 and took a quick look. I'll investigate further 
tomorrow.

One thing I noticed immediately is that there's a delay of several minutes from 
when an NOTHER/TIRC data-product is ingested on Bird01 and when it's inserted 
on Ldm01 -- so there's some kind of bottleneck between the two actions. Both 
systems were lightly loaded.

What's the bandwidth between Bird01 and Ldm01?

> Doing a restart on ldm01, this is what I see on bird01:
> 
> 20170901T205231.132940Z ldm01.allisonhouse.com(feed)[11597] NOTE 
> uldb.c:1535:sm_vetUpstreamLdm() Terminated redundant upstream LDM 
> (addr=104.197.198.101, pid=14907, vers=6, type=feeder, mode=primary, 
> sub=(20170831224548.461820 TS_ENDT {{ANY, "IXTR99 KNES"}}))
> 20170901T205231.133042Z ldm01.allisonhouse.com(feed)[14907] NOTE 
> ldmd.c:306:signal_handler() SIGTERM received
> 20170901T205231.133133Z ldm01.allisonhouse.com(feed)[14907] NOTE 
> ldmd.c:187:cleanup() Exiting
> 20170901T205231.553312Z ldm01.allisonhouse.com(feed)[11588] NOTE 
> up6.c:445:up6_run() Starting Up(6.13.6/6): 20170901203818.479016 TS_ENDT 
> {{NIMAGE, ".*"}}, SIG=fa8f676da516c6938d2945fec8725e2a, Primary
> 20170901T205231.553367Z ldm01.allisonhouse.com(feed)[11588] NOTE 
> up6.c:448:up6_run() topo:  ldm01.allisonhouse.com {{NIMAGE, (.*)}}
> 20170901T205231.556623Z ldm01.allisonhouse.com(feed)[11589] NOTE 
> up6.c:445:up6_run() Starting Up(6.13.6/6): 20170901203818.479594 TS_ENDT 
> {{NOTHER, "TIRC[0-1][0-9] KNES"}}, SIG=7960bcb03812828252a5aae70d35f395, 
> Primary
> 20170901T205231.556668Z ldm01.allisonhouse.com(feed)[11589] NOTE 
> up6.c:448:up6_run() topo:  ldm01.allisonhouse.com {{NOTHER, (.*)}}
> 20170901T205231.558296Z ldm01.allisonhouse.com(feed)[11593] NOTE 
> error.c:236:err_log() Data-product with signature 
> 73a095251edf2db906515dcd7fcbd9be wasn't found in product-queue
> 20170901T205231.558621Z ldm01.allisonhouse.com(feed)[11587] NOTE 
> up6.c:445:up6_run() Starting Up(6.13.6/6): 20170901203818.478643 TS_ENDT 
> {{NEXRAD3, ".*"}}, SIG=2473dca5b630038a27f63c54fdfb63c2, Primary
> 20170901T205231.558633Z ldm01.allisonhouse.com(feed)[11593] NOTE 
> up6.c:445:up6_run() Starting Up(6.13.6/6): 20170901203818.480235 TS_ENDT 
> {{ANY, "TIP.15 KNES"}}, SIG=73a095251edf2db906515dcd7fcbd9be, Primary
> 20170901T205231.558675Z ldm01.allisonhouse.com(feed)[11593] NOTE 
> up6.c:448:up6_run() topo:  ldm01.allisonhouse.com {{ANY, (.*)}}
> 20170901T205231.558675Z ldm01.allisonhouse.com(feed)[11587] NOTE 
> up6.c:448:up6_run() topo:  ldm01.allisonhouse.com {{NEXRAD3, (.*)}}
> 20170901T205231.559450Z ldm01.allisonhouse.com(feed)[11591] NOTE 
> up6.c:445:up6_run() Starting Up(6.13.6/6): 20170901203818.480192 TS_ENDT 
> {{NOTHER, "TIRU[0-1][0-9] KNES"}}, SIG=4027f9cc7e565e573e641d0b095ba75d, 
> Primary
> 20170901T205231.559500Z ldm01.allisonhouse.com(feed)[11591] NOTE 
> up6.c:448:up6_run() topo:  ldm01.allisonhouse.com {{NOTHER, (.*)}}
> 20170901T205231.561795Z ldm01.allisonhouse.com(feed)[11590] NOTE 
> up6.c:445:up6_run() Starting Up(6.13.6/6): 20170901203818.479669 TS_ENDT 
> {{WMO, "IXT... KNES"}}, SIG=8606c3d06681bad846b27089645a2293, Primary
> 20170901T205231.561850Z ldm01.allisonhouse.com(feed)[11590] NOTE 
> up6.c:448:up6_run() topo:  ldm01.allisonhouse.com {{WMO, (.*)}}
> 20170901T205231.564276Z ldm01.allisonhouse.com(feed)[11594] NOTE 
> up6.c:445:up6_run() Starting Up(6.13.6/6): 20170901203818.482074 TS_ENDT 
> {{IDS|DDPLUS, ".*"}}, SIG=de9912d44528989508e27e50e648de22, Primary
> 20170901T205231.564315Z ldm01.allisonhouse.com(feed)[11594] NOTE 
> up6.c:448:up6_run() topo:  ldm01.allisonhouse.com {{IDS|DDPLUS, (.*)}}
> 20170901T205231.572902Z ldm01.allisonhouse.com(feed)[11595] NOTE 
> up6.c:445:up6_run() Starting Up(6.13.6/6): 20170901203818.498043 TS_ENDT 
> {{NGRID, "...... KWBY"}}, SIG=78693a9f0d6b3259c04a3662f0bee00b, Primary
> 20170901T205231.572938Z ldm01.allisonhouse.com(feed)[11595] NOTE 
> up6.c:448:up6_run() topo:  ldm01.allisonhouse.com {{NGRID, (.*)}}
> 20170901T205231.577045Z ldm01.allisonhouse.com(feed)[11592] NOTE 
> error.c:236:err_log() Data-product with signature 
> ac9b8e96bac6bd269c29edf62f343034 wasn't found in product-queue
> 20170901T205231.577500Z ldm01.allisonhouse.com(feed)[11592] NOTE 
> up6.c:445:up6_run() Starting Up(6.13.6/6): 20170901203818.480412 TS_ENDT 
> {{ANY, "TI..99 KNES"}}, SIG=ac9b8e96bac6bd269c29edf62f343034, Primary
> 20170901T205231.577541Z ldm01.allisonhouse.com(feed)[11592] NOTE 
> up6.c:448:up6_run() topo:  ldm01.allisonhouse.com {{ANY, (.*)}}
> 20170901T205232.123413Z ldm01.allisonhouse.com(feed)[11596] NOTE 
> up6.c:445:up6_run() Starting Up(6.13.6/6): 20170901203819.061085 TS_ENDT 
> {{ANY, "TIN.0[6789] KNES"}}, SIG=NONE, Primary
> 20170901T205232.123460Z ldm01.allisonhouse.com(feed)[11596] NOTE 
> up6.c:448:up6_run() topo:  ldm01.allisonhouse.com {{ANY, (.*)}}
> 20170901T205232.133408Z ldm01.allisonhouse.com(feed)[11597] NOTE 
> up6.c:445:up6_run() Starting Up(6.13.6/6): 20170901203819.062742 TS_ENDT 
> {{ANY, "IXTR99 KNES"}}, SIG=NONE, Primary
> 20170901T205232.133455Z ldm01.allisonhouse.com(feed)[11597] NOTE 
> up6.c:448:up6_run() topo:  ldm01.allisonhouse.com {{ANY, (.*)}}
> 
> 
> Doing a notifyme on LDM, I see on bird01:
> 
> 
> 20170901T204939.001920Z 104.197.198.101(noti)[11196] NOTE 
> forn5_svc.c:468:forn_5_svc() Starting Up(6.13.6/5): 20170901204938.949075 
> TS_ENDT {{NOTHER, "TIRC"}}
> 20170901T204939.001959Z 104.197.198.101(noti)[11196] NOTE 
> forn5_svc.c:471:forn_5_svc() topo:  104.197.198.101 NOTHER
> 20170901T205017.480248Z 104.197.198.101(noti)[11196] ERROR 
> forn5_svc.c:273:noti5_sqf() TIRC05 KNES 012047 PBA: RPC: Unable to receive
> 20170901T205017.480317Z 104.197.198.101(noti)[11196] ERROR 
> forn5_svc.c:554:forn_5_svc() pq_sequence failed: Input/output error (errno = 
> 5)
> 20170901T205017.480367Z 104.197.198.101(noti)[11196] NOTE 
> ldmd.c:187:cleanup() Exiting
> 20170901T205017.482858Z ldmd[2785] NOTE ldmd.c:170:reap() child 11196 exited 
> with status 1
> 20170901T205017.800643Z noaaportIngester[2790] ERROR gb22gem.c:76:gb2_2gem() 
> [GB 1] Couldn't get parameter values
> 20170901T205017.800686Z noaaportIngester[2790] ERROR 
> grib2name.c:780:grib2name() Couldn't decode GRIB2 message. WMO header="YRAC01 
> KWEB 012000"
> 20170901T205018.270324Z noaaportIngester[2790] ERROR gb22gem.c:76:gb2_2gem() 
> [GB 1] Couldn't get parameter values
> 20170901T205018.270366Z noaaportIngester[2790] ERROR 
> grib2name.c:780:grib2name() Couldn't decode GRIB2 message. WMO header="YRAC14 
> KWEB 012000"
> 20170901T205018.723706Z noaaportIngester[2790] ERROR gb22gem.c:76:gb2_2gem() 
> [GB 1] Couldn't get parameter values
> 20170901T205018.723752Z noaaportIngester[2790] ERROR 
> grib2name.c:780:grib2name() Couldn't decode GRIB2 message. WMO header="YRAC03 
> KWEB 012000"
> 20170901T205019.292089Z noaaportIngester[2790] ERROR gb22gem.c:76:gb2_2gem() 
> [GB 1] Couldn't get parameter values
> 20170901T205019.292137Z noaaportIngester[2790] ERROR 
> grib2name.c:780:grib2name() Couldn't decode GRIB2 message. WMO header="YSAB21 
> KWEB 012000"
> 20170901T205019.797265Z noaaportIngester[2790] ERROR gb22gem.c:76:gb2_2gem() 
> [GB 1] Couldn't get parameter values
> 20170901T205019.797313Z noaaportIngester[2790] ERROR 
> grib2name.c:780:grib2name() Couldn't decode GRIB2 message. WMO header="YSAB23 
> KWEB 012000"
> 20170901T205020.289710Z noaaportIngester[2790] ERROR gb22gem.c:76:gb2_2gem() 
> [GB 1] Couldn't get parameter values
> 20170901T205020.289758Z noaaportIngester[2790] ERROR 
> grib2name.c:780:grib2name() Couldn't decode GRIB2 message. WMO header="YSAB24 
> KWEB 012000"
> 20170901T205020.698128Z noaaportIngester[2790] ERROR gb22gem.c:76:gb2_2gem() 
> [GB 1] Couldn't get parameter values
> 20170901T205020.698172Z noaaportIngester[2790] ERROR 
> grib2name.c:780:grib2name() Couldn't decode GRIB2 message. WMO header="YWAB24 
> KWEB 012000"
> 20170901T205021.077742Z noaaportIngester[2790] ERROR gb22gem.c:76:gb2_2gem() 
> [GB 1] Couldn't get parameter values
> 20170901T205021.077785Z noaaportIngester[2790] ERROR 
> grib2name.c:780:grib2name() Couldn't decode GRIB2 message. WMO header="YWAB21 
> KWEB 012000"
> 20170901T205021.144325Z 104.197.198.101(noti)[11299] NOTE 
> forn5_svc.c:468:forn_5_svc() Starting Up(6.13.6/5): 20170901205021.091841 
> TS_ENDT {{NOTHER, "TIRC"}}
> 20170901T205021.144365Z 104.197.198.101(noti)[11299] NOTE 
> forn5_svc.c:471:forn_5_svc() topo:  104.197.198.101 NOTHER

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: MKY-400850
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.