[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
Re: 20020915 SJSU Latency
- Subject: Re: 20020915 SJSU Latency
- Date: Tue, 17 Sep 2002 16:41:51 -0600 (MDT)
HI Mike,
Barring aeolus's shut down today, it looks as if data is flowing better.
Is that your take on it as well?
We will need to migrate HDS off of motherlode eventually, it is only a
filing and acting machine, we relay off of thelma, but lets hope to get to
a stable state before we go mucking around...
Comments?
-Jeff
____________________________ _____________________
Jeff Weber address@hidden
Unidata Support PH:303-497-8676
COMET Case Study Library FX:303-497-8690
University Corp for Atmospheric Research 3300 Mitchell Ln
http://www.unidata.ucar.edu/staff/jweber Boulder,Co 80307-3000
________________________________________ ______________________
On Tue, 17 Sep 2002, Mike Voss wrote:
> Jeff,
>
> I think you caught Aeolus as it was coming back online or something, it's
> been mostly down since 05Z and down again it seems. I've been having trouble
> connecting:
> ----
> rossby:~>ldmping aeolus.ucsd.edu
> Sep 17 16:20:21 State Elapsed Port Remote_Host rpc_stat
> Sep 17 16:20:31 SVC_UNAVAIL 9.997790 0 aeolus.ucsd.edu
> h_clnt_create(aeolus.ucsd.edu): Timed out while creating connection
> ------
>
> but to motherlode and sunny89 are fine:
> -----
> rossby:~>ldmping motherlode.ucar.edu
> Sep 17 16:22:18 State Elapsed Port Remote_Host rpc_stat
> Sep 17 16:22:18 RESPONDING 0.073225 388 motherlode.ucar.edu
> -----
> and
> -------
> rossby:~>ldmping sunny89.atmos.washington.edu
> Sep 17 16:23:17 State Elapsed Port Remote_Host rpc_stat
> Sep 17 16:23:17 RESPONDING 0.085064 388 sunny89.atmos.washington.edu
> --------
>
> My failover to Sunny89 is problematic, The HDS from motherlode just flies in
> on time, but I'm not catching up on IDS...strange:
> --------
> rossby:~>ldmadmin watch -f IDS
> (Type ^D or ^C when finished)
> Sep 17 16:32:37 pqutil: 492 20020917152016.968 IDS|DDPLUS 046 NTUS99
> KNCF 171519 /pTSTNCF
> Sep 17 16:32:37 pqutil: 492 20020917152017.190 IDS|DDPLUS 051 NTUS96
> KNCF 171519 /pWTSNCF
> Sep 17 16:32:40 pqutil: 9678 20020917152024.316 IDS|DDPLUS 076 SRUS86
> KRSA 171519 /pRR8RSA
> Sep 17 16:32:40 pqutil: 1818 20020917152024.345 IDS|DDPLUS 081 FGUS56
> KPTR 171519 /pRVFCW
> Sep 17 16:32:41 pqutil: 120 20020917152024.456 IDS|DDPLUS 086 SRUS55
> KPIH 171519 /pRR3PIH
> Sep 17 16:32:41 pqutil: 113 20020917152023.843 IDS|DDPLUS 091 SAUS41
> KBTV 171519 /pMTRBTV
> Sep 17 16:32:41 pqutil: 272 20020917152028.844 IDS|DDPLUS 101 FTUS46
> KMTR 171100 AAA /pTAFSFO
> Sep 17 16:32:41 pqutil: 98 20020917152028.873 IDS|DDPLUS 106 NZUS87
> KMFL 171519 /pKPAMFL
> Sep 17 16:32:42 pqutil: 3003 20020917152029.043 IDS|DDPLUS 111 SXUS55
> KTFX 171519 /pOSOINF
> Sep 17 16:32:42 pqutil: 5873 20020917152029.098 IDS|DDPLUS 116 SRUS54
> KTUA 171519 /pRR4TUR
> Sep 17 16:32:42 pqutil: 4007 20020917152029.151 IDS|DDPLUS 121 ASUS45
> KGTF 171519 /pSWRMT
> Sep 17 16:32:43 pqutil: 2327 20020917152029.220 IDS|DDPLUS 126 SXUS08
> KAKQ 171518 /pOMRAKQ
> Sep 17 16:32:51 pqutil: 98 20020917152029.318 IDS|DDPLUS 131 NZUS87
> KALR 171519 /pKPAALR
> Sep 17 16:32:57 pqutil: 1248 20020917152037.515 IDS|DDPLUS 156 SXHW50
> PHFO 171519 /pOSOHI
> Sep 17 16:32:58 pqutil: 602 20020917152039.400 IDS|DDPLUS 166 SRUS56
> KOTX 171519 /pRR1GEG
> Sep 17 16:32:59 pqutil: 98 20020917152040.103 IDS|DDPLUS 176 NZUS87
> KSJT 171519 /pKPASJT
> Sep 17 16:33:00 pqutil: 406 20020917152040.328 IDS|DDPLUS 186 SRUS52
> KGSP 171519 /pRR3GSP
> Sep 17 16:33:05 pqutil: 287 20020917152048.194 IDS|DDPLUS 206 FTUS44
> KFWD 171519 AAA /pTAFACT
> Sep 17 16:33:25 pqutil: 492 20020917152101.844 IDS|DDPLUS 342 NTUS96
> KNCF 171520 /pWTSNCF
> Sep 17 16:33:25 pqutil: 492 20020917152101.913 IDS|DDPLUS 347 NTUS99
> KNCF 171520 /pTSTNCF
> Sep 17 16:33:26 pqutil: 82 20020917152101.934 IDS|DDPLUS 352 SRUS52
> KMLB 171520 /pRR2MLB
> Sep 17 16:33:30 pqutil: 842 20020917152106.506 IDS|DDPLUS 412 SXUS51
> KAKQ 171520 /pOSOAKQ
> Sep 17 16:33:33 pqutil: 90 20020917152108.375 IDS|DDPLUS 472 SRUS55
> KCYS 171520 /pRR3CYS
> Sep 17 16:33:33 pqutil: 198 20020917152108.521 IDS|DDPLUS 477 FTUS46
> KMTR 171100 AAA /pTAFOAK
> Sep 17 16:33:35 pqutil: 594 20020917152112.759 IDS|DDPLUS 522 ASUS51
> KALB 171520 /pSWRQE
> Sep 17 16:33:35 pqutil: 1028 20020917152112.822 IDS|DDPLUS 527 AWUS43
> KGID 171520 /pAWSGRI
> Sep 17 16:33:35 pqutil: 115 20020917152113.233 IDS|DDPLUS 547 SRUS52
> KCHS 171520 /pRR3CHS
> Sep 17 16:33:39 pqutil: 124 20020917152119.699 IDS|DDPLUS 612 SXUS43
> KGLD 171520 /pMTTSYF
> Sep 17 16:33:41 pqutil: 3862 20020917152119.850 IDS|DDPLUS 627 SXUS55
> KTFX 171520 /pOSOGTF
> Sep 17 16:33:41 pqutil: 463 20020917152119.895 IDS|DDPLUS 632 SRUS56
> KPDT 171520 /pRR3PDT
> Sep 17 16:33:43 pqutil: 3349 20020917152122.900 IDS|DDPLUS 682 NZUS01
> KBTV 171520 /pSWRNOW
> Sep 17 16:33:54 pqutil: 121 20020917152128.372 IDS|DDPLUS 762 SAUS46
> KLOX 171520 /pMTRCQT
> Sep 17 16:33:55 pqutil: 6792 20020917152128.556 IDS|DDPLUS 772 SXUS86
> KLOX 171520 /pOMRSGX
> Sep 17 16:34:04 pqutil: 144 20020917152134.808 IDS|DDPLUS 887 SRUS55
> KVEF 171520 /pRR3VEF
> Sep 17 16:34:05 pqutil: 970 20020917152139.970 IDS|DDPLUS 923 WWUS82
> KMLB 171521 /pSPSMLB
> Sep 17 16:34:05 pqutil: 492 20020917152140.520 IDS|DDPLUS 953 NTUS96
> KNCF 171521 /pWTSNCF
> Sep 17 16:34:06 pqutil: 492 20020917152140.623 IDS|DDPLUS 958 NTUS99
> KNCF 171521 /pTSTNCF
> Sep 17 16:34:06 pqutil: 4302 20020917152140.779 IDS|DDPLUS 968 NZUS11
> KBYZ 171521 /pWRKRSA
> Sep 17 16:34:08 pqutil: 1344 20020917152146.076 IDS|DDPLUS 018 FXUS63
> KGRR 171521 CCA /pAFDGRR
> Sep 17 16:34:10 pqutil: 23020 20020917152149.758 IDS|DDPLUS 043 FPUS55
> KBYZ 171521 AAA /pZFPBYZ
> Sep 17 16:34:11 pqutil: 1325 20020917152156.601 IDS|DDPLUS 104 FGUS52
> KCHS 171521 /pRVDCHS
> Sep 17 16:34:12 pqutil: 2445 20020917152156.027 IDS|DDPLUS 112 SRUS81
> KPHI 171521 /pRRAPHL
> Sep 17 16:34:12 pqutil: 108 20020917152157.714 IDS|DDPLUS 140 SRUS56
> KOTX 171521 /pRR3OTX
> Sep 17 16:34:13 pqutil: 85 20020917152202.447 IDS|DDPLUS 164 SRUS54
> KSHV 171521 /pRR3SHV
> Sep 17 16:34:20 pqutil: 605 20020917152203.243 IDS|DDPLUS 188 FNUS72
> KTAE 171521 /pSMFTLH
> Sep 17 16:34:21 pqutil: 109 20020917152203.567 IDS|DDPLUS 194 SRUS55
> KLKN 171521 /pRR3LKN
> Sep 17 16:34:33 pqutil: 2453 20020917152211.970 IDS|DDPLUS 287 FZUS63
> KLOT 171521 CCA /pGLFLM
> Sep 17 16:34:33 pqutil: 591 20020917152211.997 IDS|DDPLUS 292 NZUS50
> KBYZ 171521 /pWRKCCF
> Sep 17 16:34:34 pqutil: 449 20020917152213.421 IDS|DDPLUS 370 FPUS74
> KLZK 171521 /pNOWLIT
> Sep 17 16:34:34 pqutil: 2918 20020917152213.448 IDS|DDPLUS 372 FOUS61
> KTAR 171521 /pFFGME
> Sep 17 16:34:34 pqutil: 235 20020917152213.653 IDS|DDPLUS 388 FTUS44
> KMEG 171521 AAB /pTAFMEM
> Sep 17 16:34:41 pqutil: 3007 20020917152218.252 IDS|DDPLUS 509 FOUS61
> KTAR 171521 /pFFGMA
> Sep 17 16:34:41 pqutil: 1782 20020917152218.324 IDS|DDPLUS 524 FXUS64
> KBRO 171521 /pAFDBRO
> Sep 17 16:34:41 pqutil: 3695 20020917152223.213 IDS|DDPLUS 606 FOUS71
> KTAR 171521 /pFFHHFD
> Sep 17 16:34:41 pqutil: 5265 20020917152223.234 IDS|DDPLUS 610 FOUS61
> KTAR 171521 /pFFGNY
> Sep 17 16:34:41 pqutil: 125 20020917152224.379 IDS|DDPLUS 700 SRUS55
> KBOI 171521 /pRR3BOI
> Sep 17 16:34:42 pqutil: 3146 20020917152229.065 IDS|DDPLUS 769 FZPN03
> KNHC 171522 /pHSFEP2
> Sep 17 16:34:43 pqutil: 492 20020917152234.517 IDS|DDPLUS 917 NTUS96
> KNCF 171522 /pWTSNCF
> Sep 17 16:34:44 pqutil: 1297 20020917152235.979 IDS|DDPLUS 032 SRUS44
> KBRO 171522 /pRVABRO
> Sep 17 16:34:44 pqutil: 99 20020917152239.566 IDS|DDPLUS 067 SRUS52
> KFFC 171522 /pRR3ATL
> Sep 17 16:34:44 pqutil: 492 20020917152239.570 IDS|DDPLUS 068 NTUS99
> KNCF 171522 /pTSTNCF
> Sep 17 16:34:44 pqutil: 289 20020917152240.176 IDS|DDPLUS 116 SRUS54
> KTUA 171522 /pRR2OKC
> ------------
>
> No joy in mudville yet....later,
>
> -Mike
>
>
>
>
>
>
>
>
> At 09:55 AM 9/17/2002 -0600, Jeff Weber wrote:
> >Hi Mike,
> >
> >aeolus is up currently...
> >
> >aeolus.ucsd.edu> ldmadmin watch
> >(Type ^D or ^C when finished)
> >Sep 17 15:50:27 pqutil: 2706 20020917150418.944 IDS|DDPLUS 125 WTNT45
> >KNHC 171504 /pTCDAT5
> >Sep 17 15:50:27 pqutil: 1653 20020917150419.024 HDS 131 SDUS85
> >KABQ 171500 /pDPAFDX
> >Sep 17 15:50:27 pqutil: 149 20020917150419.062 HDS 134 NXUS64
> >KLCH 171503 /pGSMLCH
> >Sep 17 15:50:27 pqutil: 149 20020917150419.081 HDS 136 NXUS62
> >TJSJ 171502 /pGSMJUA
> >Sep 17 15:50:27 pqutil: 35928 20020917150419.178 HDS 140 ZHRK99
> >KWBE 171200 /mETA_84
> >Sep 17 15:50:27 pqutil: 35928 20020917150423.371 HDS 143 ZHRK93
> >KWBE 171200 /mETA_84
> >Sep 17 15:50:27 pqutil: 42 20020917150423.376 IDS|DDPLUS 144 SIVE25
> >RJTD 170900
> >Sep 17 15:50:27 pqutil: 42 20020917150423.378 IDS|DDPLUS 145 SIVX25
> >RJTD 170900
> >Sep 17 15:50:27 pqutil: 1013 20020917150423.380 IDS|DDPLUS 146 UMZZ42
> >KAWN 171200 RRX
> >Sep 17 15:50:27 pqutil: 287 20020917150423.383 IDS|DDPLUS 147 FTKO51
> >RJTD 171400 CCA
> >Sep 17 15:50:27 pqutil: 177 20020917150423.386 IDS|DDPLUS 149 UDEU02
> >EGRR 171450
> >Sep 17 15:50:28 pqutil: 35928 20020917150423.436 HDS 151 ZHRK95
> >KWBE 171200 /mETA_84
> >Sep 17 15:50:28 pqutil: 35928 20020917150423.516 HDS 154 ZHRK92
> >KWBE 171200 /mETA_84
> >Sep 17 15:50:28 pqutil: 35928 20020917150423.563 HDS 157 ZHRK91
> >KWBE 171200 /mETA_84
> >Sep 17 15:50:29 pqutil: 172188 20020917150423.023 FSL2 000
> >FSL.NetCDF.NOAAnet.windprofiler.06min.20022601454.*
> >Sep 17 15:50:29 pqutil: 35928 20020917150423.599 HDS 158 ZHRK90
> >KWBE 171200 /mETA_84
> >Sep 17 15:50:29 pqutil: 119 20020917150423.681 IDS|DDPLUS 165 SXXX40
> >KWAL 171449
> >Sep 17 15:50:29 pqutil: 132 20020917150423.692 IDS|DDPLUS 166 SXNE55
> >KWAL 171449
> >Sep 17 15:50:29 pqutil: 167 20020917150423.695 HDS 170 SACN94
> >CWAO 171400 RRD
> >Sep 17 15:50:29 pqutil: 113 20020917150423.696 IDS|DDPLUS 168 UDEU02
> >EGRR 171450
> >Sep 17 15:50:29 pqutil: 110 20020917150423.697 IDS|DDPLUS 169 SACN89
> >CWAO 171400 RRE
> >Sep 17 15:50:29 pqutil: 100 20020917150423.698 IDS|DDPLUS 171 SACN91
> >CWAO 171400 RRD
> >Sep 17 15:50:29 pqutil: 94 20020917150423.699 IDS|DDPLUS 172 SNTC01
> >LFPW 162300
> >Sep 17 15:50:29 pqutil: 94 20020917150423.699 IDS|DDPLUS 173 SNTC01
> >LFPW 162200
> >
> >
> >Tho they seem a bit behind ~46 minutes...
> >
> >Darn!
> >
> >Sunny89 is not working properly as a failover..?
> >
> >Keep me posted!
> >
> >
> >-Jeff
> >____________________________ _____________________
> >Jeff Weber address@hidden
> >Unidata Support PH:303-497-8676
> >COMET Case Study Library FX:303-497-8690
> >University Corp for Atmospheric Research 3300 Mitchell Ln
> >http://www.unidata.ucar.edu/staff/jweber Boulder,Co 80307-3000
> >________________________________________ ______________________
> >
> >On Tue, 17 Sep 2002, Mike Voss wrote:
> >
> >> Mornin' Jeff,
> >>
> >> No data since about 5Z from aeolus.ucsd.edu. Can you see from their end if
> >> they are down or not? I don't see anything with NOTIFYME. I guess I better
> >> get my ldmfail script working correctly. I've failed over to Sunny89, but
> >> I'm not getting anything....I imagine it has to figure out which products
> >> to send...hmm, I don't recall the LDM having this problem before I changed
> >> to 5.2.
> >>
> >> -Mike
> >>
> >> At 12:34 PM 9/16/2002 -0600, you wrote:
> >> >Hi Mike,
> >> >
> >> >Thanks for adding rtstats, was going to do it while I was on, but thought
> >> >I would let you designate when rossby went down...
> >> >
> >> >No, it is not normal for these "other" feeds to lag..It did this when I
> >> >added motherlode as the HDS feed, odd...
> >> >
> >> >I suspect the IDS and NNEXRAD will catch up, but it is curious why they
> >> >take so long to fire up, this may be part of the HDS problem from aeolus.
> >> >
> >> >Still investigating..
> >> >
> >> >Thanks for adding the rtstats, this will help all of us better diagnose
> >> >events at SJSU!
> >> >
> >> >More later,
> >> >
> >> >-Jeff
> >> >____________________________ _____________________
> >> >Jeff Weber address@hidden
> >> >Unidata Support PH:303-497-8676
> >> >COMET Case Study Library FX:303-497-8690
> >> >University Corp for Atmospheric Research 3300 Mitchell Ln
> >> >http://www.unidata.ucar.edu/staff/jweber Boulder,Co 80307-3000
> >> >________________________________________ ______________________
> >> >
> >> >On Mon, 16 Sep 2002, Mike Voss wrote:
> >> >
> >> >> Jeff,
> >> >>
> >> >> I added " rtstat" to ldmd.conf.
> >> >>
> >> >> When I restarted the LDM, HDS started flowing right away, but
> >> >> IDS|NEXRAD..etc is not flowing. It's like the program is having
> >> >> trouble syncing up product queue info to see what products should be
> >> >> sent. Is this normal? In my experience it is not normal.
> >> >>
> >> >> Data did flow well over the weekend even though our network still has
> >> >> a bottle neck with large packets at 130.65.80.1. This problem is still
> >> >> being looked at by network engineers...no answer yet on why it's
> >> >> happening or what the fix is. I think the data flow has been able to
> >> >> overcome this because the rest of the network between here and
> >> >> UCSD/UCAR has been fine.
> >> >>
> >> >> Gotta go hack away and get my data flowing. Oh, I see IDS|NEXRAD data
> >> >> is flowing again, but with a 30 minute latency. Should that really
> >> >> happen when I just stop and start the LDM? It was flowing with no
> >> >> latency previous to the restart.
> >> >>
> >> >> Thanks,
> >> >>
> >> >> -Mike
> >> >>
> >> >>
> >> >>
> >> >> At 10:12 AM 9/16/2002 -0600, Jeff Weber wrote:
> >> >> >HI Mike,
> >> >> >
> >> >> >Data seems to be more timely, but we are coming off the weekend..
> >> >> >
> >> >> >Will you please add:
> >> >> >
> >> >> >exec "rtstats -h rtstats.unidata.ucar.edu"
> >> >> >
> >> >> >in your ldmd.conf file.
> >> >> >
> >> >> >And then start and stop..Do this at your convenience.
> >> >> >
> >> >> >
> >> >> >You can place this directly under:
> >> >> >
> >> >> >exec "pqbinstats"
> >> >> >
> >> >> >This will allow us to more closely monitor traffic, and you can access
> >> >> >these stats as well at:
> >> >> >
> >> >> >http://www.unidata.ucar.edu/staff/chiz/latency/rtstats/index2.shtml
> >> >> >
> >> >> >Thanks,
> >> >> >
> >> >> >-Jeff
> >> >> >____________________________ _____________________
> >> >> >Jeff Weber address@hidden
> >> >> >Unidata Support PH:303-497-8676
> >> >> >COMET Case Study Library FX:303-497-8690
> >> >> >University Corp for Atmospheric Research 3300 Mitchell Ln
> >> >> >http://www.unidata.ucar.edu/staff/jweber Boulder,Co 80307-3000
> >> >> >________________________________________ ______________________
> >> >>
> >> >> --------------------------
> >> >> Mike Voss
> >> >> Department of Meteorology
> >> >> San Jose State University
> >> >> One Washington Square
> >> >> San Jose, CA 95192-0104
> >> >>
> >> >> 408.924.5204 voice
> >> >> 408.924.5191 fax
> >> >>
> >> >>
> >>
> >> --------------------------
> >> Mike Voss
> >> Department of Meteorology
> >> San Jose State University
> >> One Washington Square
> >> San Jose, CA 95192-0104
> >>
> >> 408.924.5204 voice
> >> 408.924.5191 fax
> >>
> >>
>
> --------------------------
> Mike Voss
> Department of Meteorology
> San Jose State University
> One Washington Square
> San Jose, CA 95192-0104
>
> 408.924.5204 voice
> 408.924.5191 fax
>
>
