[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
Re: 20011008: IDD latencies at PSU (cont.)
- Subject: Re: 20011008: IDD latencies at PSU (cont.)
- Date: Tue, 30 Oct 2001 09:22:58 -0700 (MST)
Art,
For your comparisons with NMC2 latencies, the NMC2 feed was substantially
improved last night:
http://www.unidata.ucar.edu/staff/chiz/latency/stats/NMC2_motherlode.ucar.edu.gif
The behavior returned to the pattern we saw before Wednesday of last week.
The slow down appeared to be with a Solaris system monitoring program "esd"
which was turneed off late yesterday afternoon, coincident with the
improvement in the data latencies.
FYI, Illinois is a top tier site along with motherlode and U. Wisconsin
for the NMC2 data feed. If you are able to get the CONDUIT data reliably from
motherlode, then we can keep it that way.
Steve Chiswell
Unidata User Support
On Tue, 30 Oct 2001, Arthur A. Person wrote:
> Anne,
>
> I'm ready to conclude this investigation into the latencies assuming no
> further problems arise. Here's what I'm concluding:
>
> 1) The original problem which prompted this latency investigation was
> caused by a hardware problem in a local PSU router which was slowing down
> data delivery. That problem was resolved, probably sometime last week.
>
> 2) Data latencies continued after the local router problem was fixed,
> but weren't as bad. Collecting DDPLUS via the temporary machine
> nutone.ems.psu.edu (which used a different, faster, path through PSU) made
> collection of DDPLUS|IDS|HDS arrive on time (NMC2 collection was still via
> motherlode). While trying to run DDPLUS|IDS|HDS *and* NMC2 from
> motherlode produced delays. However, nutone was not collecting NMC2 until
> yesterday and what I discovered was when nutone started collecting NMC2
> data along with DDPLUS|IDS|HDS, I began seeing delays on nutone's DDPLUS
> stuff. I determined this using notifyme's from nutone. So, I got the
> bright idea (duh) of breaking my ldm.meteo feed from motherlode into two
> sockets using DNS name and IP address for motherlode. I started
> collecting this way last evening and saw an immediate improvement. My
> conclusion is that NMC2 cannot be fed on the same socket as anything else
> or they will both suffer.
>
> I don't entirely understand why the above should be the case. The
> immediate conclusion might be that there are too many products/time for
> the network turnaround latencies on the socket. If this were the case I
> would expect to see delays in all the data arriving through that socket.
> Last evening, however, I seemed to notice that when both were on the same
> socket, NMC2 was only running a few minutes behind (comparing notifyme's
> from motherlode with ldmadmin watch's here) but the DDPLUS stuff fell to
> ~15 minutes behind. If things were sent serially in time from the product
> queue on the same socket, I guess I would have expected to see similar
> delays on both. Not knowing the internals of the LDM, this makes me think
> the LDM is somehow round-robining feed stream requests and somehow the
> DDPLUS|IDS|HDS stuff is not getting into the feed hopper as much as NMC2.
> Without further investigation, I don't know how to explain the
> discrepancy, unless my observation of delays in transmission was a fluke.
>
> Whatever the case, it would seem that a capability in the LDM to put
> requests from a single machine on different sockets would be useful,
> formallizing the backhanded approach of using name/IP address. Something
> like motherlode.ucar.edu:0, motherlode.ucar.edu:1, etc... like the X
> window system does for displays.
>
> I'll continue to watch the streams and see how things do. FYI, here are
> the stats for NMC2 data for the last several days on ldm.meteo showing the
> improvement last evening (assuming the NMC2 feed itself didn't improve
> coincidentally):
>
> 20011025113901 NMC2 tgsv32 3587 71479274
> 751.77 1027@3901 20011025115608
> 20011025123512 NMC2 tgsv32 898 17644380
> 91.66 169@3512 20011025123802
> 20011025135954 NMC2 tgsv32 14686 291948815
> 355.75 1026@1758 20011025140945
> 20011025145900 NMC2 tgsv32 23598 481060465
> 925.19 1894@4840 20011025152758
> 20011025155953 NMC2 tgsv32 15837 503270823
> 2396.47 3660@5458 20011025165958
> 20011025165936 NMC2 tgsv32 10256 445737565
> 3340.08 3638@3600 20011025175746
> 20011025175829 NMC2 tgsv32 3768 311421104
> 2720.05 3344@1016 20011025183030
> 20011025185547 NMC2 tgsv32 3140 259527130
> 942.21 1817@0300 20011025185831
> 20011025195322 NMC2 tgsv32 3901 95864247
> 95.49 202@5322 20011025195643
> 20011025205856 NMC2 tgsv32 3593 70379790
> 102.08 208@5146 20011025205914
> 20011025212257 NMC2 tgsv32 731 10854222
> 45.28 88@0510 20011025212407
> 20011025225913 NMC2 tgsv32 10921 422113711
> 322.13 691@4924 20011025230703
> 20011025235726 NMC2 tgsv32 12624 251112258
> 342.03 854@1954 20011025235734
> 20011026005916 NMC2 tgsv32 15559 321063706
> 215.52 524@3456 20011026010611
> 20011026015559 NMC2 tgsv32 12002 433257737
> 440.73 1083@5559 20011026021402
> 20011026024406 NMC2 tgsv32 12121 610452700
> 2247.62 3660@4406 20011026034506
> gap
> 20011026113726 NMC2 tgsv32 1564 33006142
> 3565.61 3660@3726 20011026123826
> 20011026123458 NMC2 tgsv32 898 18079486
> 860.03 949@3458 20011026125047
> 20011026135959 NMC2 tgsv32 15714 320132048
> 439.80 1085@1547 20011026141424
> 20011026145959 NMC2 tgsv32 22845 435065678
> 916.68 1511@5908 20011026152500
> 20011026155949 NMC2 tgsv32 15379 442493007
> 1563.95 1951@5536 20011026163123
> 20011026161945 NMC2 tgsv32 4713 226606661
> 2916.22 3660@1945 20011026172045
> 20011026182203 NMC2 tgsv32 268 4806152
> 3646.13 3660@2034 20011026192303
> 20011026233840 NMC2 tgsv32 3587 73669888
> 3028.07 3323@3840 20011027003403
> 20011027003442 NMC2 tgsv32 898 18218170
> 109.92 193@3442 20011027003755
> 20011027015959 NMC2 tgsv32 12786 261503524
> 293.60 863@1605 20011027020550
> 20011027025912 NMC2 tgsv32 26111 540802605
> 1039.52 1942@5912 20011027033134
> 20011027035920 NMC2 tgsv32 13587 525678769
> 1906.44 2316@5920 20011027043756
> 20011027045949 NMC2 tgsv32 15314 603505665
> 2960.37 3457@4709 20011027055709
> 20011027055405 NMC2 tgsv32 5704 292888911
> 3519.62 3660@2603 20011027065505
> 20011027065917 NMC2 tgsv32 7884 240337083
> 1823.52 2090@4751 20011027073348
> 20011027075529 NMC2 tgsv32 21942 572682384
> 2676.36 3661@5054 20011027085629
> 20011027095959 NMC2 tgsv32 6108 507120622
> 2484.78 3023@5959 20011027105022
> 20011027101406 NMC2 tgsv32 4103 200509688
> 3336.36 3660@1303 20011027111506
> 20011027113946 NMC2 tgsv32 3310 63992744
> 2980.96 3179@3946 20011027123245
> 20011027123420 NMC2 tgsv32 26 320315
> 27.57 29@3420 20011027123449
> 20011027135942 NMC2 tgsv32 14480 278511452
> 291.46 837@1605 20011027140632
> 20011027145959 NMC2 tgsv32 25294 516371110
> 787.15 1628@5959 20011027152707
> 20011027155957 NMC2 tgsv32 17213 552317195
> 1820.29 2559@5901 20011027164144
> 20011027165959 NMC2 tgsv32 10828 608893315
> 3023.72 3378@5225 20011027175326
> 20011027174004 NMC2 tgsv32 9203 340549178
> 3370.65 3660@2454 20011027184045
> 20011027184415 NMC2 tgsv32 14958 496865870
> 2467.71 3658@4415 20011027194513
> 20011027195951 NMC2 tgsv32 16838 344062234
> 2294.22 3264@4850 20011027204520
> 20011027203944 NMC2 tgsv32 13109 270728545
> 2811.48 3316@1903 20011027212713
> 20011027215955 NMC2 tgsv32 11606 678330442
> 1647.73 2855@5955 20011027224730
> 20011027224000 NMC2 tgsv32 7254 262585268
> 2763.58 3610@0914 20011027232238
> 20011027233926 NMC2 tgsv32 3587 71767406
> 238.45 505@3926 20011027234751
> 20011028003411 NMC2 tgsv32 898 17795880
> 93.88 172@3411 20011028003703
> 20011028015954 NMC2 tgsv32 12733 254906988
> 292.69 831@1428 20011028020758
> 20011028025958 NMC2 tgsv32 22125 412893974
> 497.05 985@4818 20011028031259
> 20011028035959 NMC2 tgsv32 12093 506966736
> 450.41 906@0518 20011028041332
> 20011028045747 NMC2 tgsv32 14883 624274810
> 1852.94 2877@3333 20011028053417
> 20011028053905 NMC2 tgsv32 8954 312357651
> 2296.27 2746@1937 20011028061709
> 20011028065904 NMC2 tgsv32 9059 264692217
> 491.99 905@4341 20011028071222
> 20011028075903 NMC2 tgsv32 28064 696635046
> 1718.35 3237@5903 20011028085300
> 20011028083711 NMC2 tgsv32 10442 209979131
> 3411.51 3660@1731 20011028093811
> 20011028095905 NMC2 tgsv32 8556 537700052
> 2048.60 2795@5905 20011028104539
> 20011028105937 NMC2 tgsv32 6729 310947298
> 3214.76 3660@2032 20011028115737
> 20011028114000 NMC2 tgsv32 5830 116642823
> 2799.08 3660@0346 20011028122235
> 20011028123455 NMC2 tgsv32 898 17542070
> 94.26 166@3455 20011028123741
> 20011028135907 NMC2 tgsv32 16452 335896998
> 405.36 931@1405 20011028141416
> 20011028145957 NMC2 tgsv32 24506 491850722
> 1079.96 1838@5809 20011028153030
> 20011028155959 NMC2 tgsv32 15865 603071754
> 2015.36 3001@5648 20011028164741
> 20011028165915 NMC2 tgsv32 9821 582541948
> 3168.00 3463@0631 20011028175353
> 20011028173905 NMC2 tgsv32 6252 121848498
> 2588.97 3655@0217 20011028181544
> 20011028185011 NMC2 tgsv32 13726 438614559
> 1582.78 2614@5011 20011028193345
> 20011028195945 NMC2 tgsv32 16927 343949301
> 1220.30 2179@4833 20011028202827
> 20011028203750 NMC2 tgsv32 12459 255794679
> 1716.91 2089@2033 20011028210802
> 20011028215959 NMC2 tgsv32 11679 659652471
> 1422.87 2513@5934 20011028224146
> 20011028223957 NMC2 tgsv32 7013 264606178
> 2347.94 3110@1147 20011028231642
> 20011028233943 NMC2 tgsv32 3587 70871512
> 240.71 480@3943 20011028234742
> 20011029003541 NMC2 tgsv32 898 17580432
> 93.95 167@3541 20011029003828
> 20011029015959 NMC2 tgsv32 13374 261164634
> 325.77 875@1452 20011029020732
> 20011029025940 NMC2 tgsv32 22635 457179418
> 707.39 1499@5056 20011029031925
> 20011029035906 NMC2 tgsv32 20101 731595730
> 1852.82 3307@5906 20011029045412
> 20011029045948 NMC2 tgsv32 9111 510570633
> 2962.72 3361@0310 20011029055320
> 20011029053810 NMC2 tgsv32 7720 253438369
> 3227.20 3660@1508 20011029063227
> 20011029065955 NMC2 tgsv32 9650 275810446
> 566.36 949@4804 20011029071447
> 20011029075653 NMC2 tgsv32 25479 632618917
> 1681.79 2791@5653 20011029084324
> 20011029083823 NMC2 tgsv32 15555 339253771
> 3022.74 3499@2120 20011029093547
> 20011029095943 NMC2 tgsv32 10664 599343953
> 1967.24 2870@5943 20011029104733
> 20011029105950 NMC2 tgsv32 6452 322146567
> 3277.17 3660@2133 20011029114936
> 20011029114113 NMC2 tgsv32 6252 124740496
> 2454.25 3377@0249 20011029121210
> 20011029125956 NMC2 tgsv32 1215 22249970
> 87.17 142@5706 20011029125959
> 20011029135953 NMC2 tgsv32 12488 255303234
> 304.43 844@1847 20011029140626
> 20011029145934 NMC2 tgsv32 21601 403208947
> 486.20 955@5028 20011029150906
> 20011029155926 NMC2 tgsv32 15933 550413449
> 728.77 1307@5618 20011029161931
> 20011029165955 NMC2 tgsv32 13488 614393360
> 1906.22 2823@5624 20011029174608
> 20011029175105 NMC2 tgsv32 10292 406366344
> 2884.64 3232@5105 20011029184457
> 20011029185956 NMC2 tgsv32 14536 485593177
> 2076.37 2905@5738 20011029194705
> 20011029195851 NMC2 tgsv32 14270 287765999
> 1738.11 2827@0006 20011029203852
> 20011029203637 NMC2 tgsv32 14126 309353543
> 2754.00 3361@3603 20011029213209
> 20011029213716 NMC2 tgsv32 6154 444125157
> 2121.66 3117@3716 20011029222913
> 20011029223752 NMC2 tgsv32 2718 54970495
> 208.25 420@3725 20011029224425
> 20011029233729 NMC2 tgsv32 2718 54605755
> 175.17 368@3647 20011029234255
> 20011030003542 NMC2 tgsv32 898 18015543
> 112.27 209@3542 20011030003911
> 20011030015959 NMC2 tgsv32 10143 207262144
> 459.50 1186@5959 20011030021945
> 20011030023820 NMC2 tgsv32 7876 167338005
> 812.61 1343@1439 20011030024914
> 20011030034344 NMC2 tgsv32 2594 146168550
> 235.00 613@2549 20011030034354
> 20011030043900 NMC2 tgsv32 2718 53618265
> 190.43 383@3815 20011030044443
> 20011030053827 NMC2 tgsv32 2718 53477487
> 190.73 384@3737 20011030054405
> 20011030065704 NMC2 tgsv32 3802 140366729
> 92.14 251@5030 20011030065840
> 20011030075957 NMC2 tgsv32 17116 328190560
> 735.16 1281@4556 20011030081917
> 20011030083726 NMC2 tgsv32 9693 212115708
> 935.11 1446@1718 20011030085147
> 20011030095710 NMC2 tgsv32 13147 670018502
> 1411.16 2393@5710 20011030103703
> 20011030103852 NMC2 tgsv32 2718 52954933
> 319.65 506@3823 20011030104654
> 20011030114036 NMC2 tgsv32 2718 53012503
> 153.75 332@3942 20011030114519
> 20011030123436 NMC2 tgsv32 898 17463101
> 95.88 178@3436 20011030123734
>
> At this point only ldm.meteo.psu.edu should be feeding from motherlode.
> Jeff has turned off nutone. Let me know if you still need me to switch
> ldm.meteo to flood for the NMC2 feed. Is flood a top level feed source?
> My only concern with a switch would be getting too far down the chain
> which would potentially introduce more points of congestion/delay for our
> feed.
>
> Thanks.
>
> Art.
>
> Arthur A. Person
> Research Assistant, System Administrator
> Penn State Department of Meteorology
> email: address@hidden, phone: 814-863-1563
>
>
