[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: LDM: rtstats(1) limitation on number of reporting "bins"

Virginia,

No. Every 60 seconds, rtstats will check for any bin that has received a
product since the last successful transmission, and report those
numbers. If an attempt to report statistics fails, the bin is not marked
as clean, and so every 60 seconds an attempt will be made until that
infrequently received bin is marked clean. Only if a bin is marked clean
will it be passed over when stats are reported.

The limitation there would be if you were running the older version
which only had 96 bins, and your success at reporting the
stats was very unreliable such that the bin was reclaimed before the
stats were sucessfully reported. That shouldn't be a problem
with the more recent versions of LDM with increased numbers of bins.

Steve Chiswell
Unidata User Support




On Wed, 2005-06-15 at 12:57, Virginia Galvin wrote:
> Steve,
> In any of the ldm/rtstats versions is  there  a  frequency  threshhold 
> for data statistics?
> Such that infrequently distributed data would not be counted in the 
> statistics?
> gini
> 
> Steve Emmerson wrote:
> 
> >Warren & Gini,
> >
> >The rtstats(1) utility in version 6.0 of the LDM does, indeed, have a
> >limitation on the number of "bins" on which it can report.  That limit
> >is 96.
> >
> >That limit was increased to 4000 in version 6.1.0 of the LDM (which
> >was released on 2004-08-25).  Subsequent versions also have the higher
> >limit.  (We bumped the limit up because encountered the same monitoring
> >problem here).
> >
> >What do you want to do?  Options include the following:
> >
> >    Replacing the LDM-s with a later version.
> >
> >    Replacing just the rtstats(1) utility with a later version.
> >
> >Either of the above would have to be done for every LDM system that
> >collected NEXRAD2 data from more than 96 upstream LDM-s.
> >
> >All versions of the LDM are "backward" compatible: any LDM will always
> >work with an older version.
> >
> >Regards,
> >Steve Emmerson
> >LDM Developer
> >
> >  
> >
> >
> > ------------------------------------------------------------------------
> >
> > Subject:
> > Re: rtstats
> > From:
> > Virginia Galvin <address@hidden>
> > Date:
> > Wed, 15 Jun 2005 13:03:28 -0500
> > To:
> > Steve Emmerson <address@hidden>, Virginia Galvin 
> > <address@hidden>
> >
> > To:
> > Steve Emmerson <address@hidden>, Virginia Galvin 
> > <address@hidden>
> > CC:
> > Warren Blanchard <address@hidden>, David S Alden 
> > <address@hidden>, Allan Darling <address@hidden>
> >
> >
> > Steve,
> > Our RadarII developers are running into limitation with rtstats
> > and  believe that rtstats can only keep track of 100 sources of data 
> > per feedtype (CRAFT)
> > and that it drops sources that have not reported over an extended 
> > period of time.
> >
> > We need to keep track of at least 300 sources and need to be able to 
> > configure tracking of sources that may have extended periods of outage.
> >
> > How can we do this?
> > Thanks
> > gini galvin
> > NWS/OCIO/TOC/Telecommunications Software Branch
> > Radar II  Administrator
> > 240 393 3348 c
> > 301 713 0882 x 176 work
> >
> > David S Alden wrote:
> >
> >
> >>> Hi Warren,
> >>>     Gini sent me a message Monday indicating she would check with her
> >>> supervisor about resolving the 'rtstats' source tracking problems but
> >>> I have not heard about any progress she might have made up to this
> >>> point.  I suspect she is still working the problem in the absence of
> >>> an update on her progress.
> >>>     As you may know, the only information I have on the functionality
> >>> of the LDM 'rtstats' utility is the 'man' page provided with the LDM
> >>> distribution.  This documentation does not go into detail related to
> >>> the inner workings and/or ideosyncracies it may have so we are learning
> >>> by experience how it works.  There appears to be at present two 
> >>> problems
> >>> that affect its use in our Level II Radar Monitor project.  The first
> >>> is an apparent limitation it has for keeping track of only about 100
> >>> sources of data ingesting into any particular feedtype (ours is CRAFT)
> >>> and the second is its apparent inability to report on sources that have
> >>> been out of service for extended periods of time.  The man page does
> >>> not indicate any limitations with regard to either of these
> >>> considerations or any others for that matter.  Upon reflection,
> >>> however, one can easily see how in the general case, 'rtstats' would
> >>> drop a source for which nothing had been received for some internally
> >>> predetermined period of time because there are many applications of
> >>> LDM for which the sources of data are not as stable as the AWIPS
> >>> radars (stable in terms of number not necessarily operational
> >>> consistency).  I would propose that 'rtstats' provide a configurable
> >>> source retention option that prevents it from dropping sources before
> >>> they are restored to service.  If, in fact, it has an internal
> >>> limitation of 100 sources it can keep track of, that this also be
> >>> configurable if it is not already.  Of course, it is possible to code
> >>> around such limitations but, in the absence of specific documentation,
> >>> we could not determine the need for this beforehand.  Any help you can
> >>> get from Steve Emerson would be appreciated.
> >>>
> >>  
> >>
> >>>> >Date: Wed, 15 Jun 2005 07:38:51 -0400
> >>>> >From: "Warren.Blanchard" <address@hidden>
> >>>> >Organization: DOC/NOAA/NWS - National Weather Service
> >>>> >To: David S Alden <address@hidden>
> >>>> >CC: Allan Darling <address@hidden>,
> >>>> >   Jami Casamento <address@hidden>,
> >>>> >   "address@hidden" <address@hidden>
> >>>> >Subject: Level II radar monitor
> >>>> >
> >>>> >Dave,
> >>>> >I was following your e-mails relative to the rtstats limitations. 
> >>>> If not
> >>>> >already resolved, we could consult with Steve Emerson (Unidata
> >>>> >LDM developer) on if and how to correct that problem.
> >>>> >I also noticed, when checking the tgweb Level II page that on Wed.
> >>>> >morning only 130 sites were being monitored. I believe the number
> >>>> >of sites should always be 133 unless more are added to the system.
> >>>> >It looks as though three sites (KHDX, KOKX and KVNX) were
> >>>> >not included. I guessed this was because each of these had been
> >>>> >down for an extended period of time, 11 to 18 hours. I would think
> >>>> >all 133 should be included even if some are down for long periods.
> >>>> >Attached is the page at that time, and the TOC e-mail notice of
> >>>> >outages/latencies.
> >>>> >Hope this helps...
> >>>> >Warren
> >>>> >
> >>>> >Subject: NWS Region WSR-88D Level II Data Outage/Data Latency Update
> >>>> >Date: Wed, 15 Jun 2005 10:18:05 GMT
> >>>> >From: address@hidden
> >>>> >To: address@hidden
> >>>> >CC: address@hidden
> >>>> >
> >>>> >NWS Region WSR-88D Level II Data Outage/Data Latency Update FOR 
> >>>> WEDNESDAY
> >>>> >15JUN05 AT 6AM EST LOCAL:
> >>>> >
> >>>> >KHDX = 11 HRS 53 MINS
> >>>> >KIND =  9 HRS 22 MINS
> >>>> >KOKX = 17 HRS 30 MINS
> >>>> >KMOB =  1 HRS 28 MINS
> >>>> >KVNX = 18 HRS 23 MINS
> >>>
> >>>   
> >>
> >
> >     
> > Warren B
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > Subject:
> > ldm rtstats usage
> > From:
> > "Warren.Blanchard" <address@hidden>
> > Date:
> > Wed, 15 Jun 2005 13:49:33 -0400
> > To:
> > Steve Emmerson <address@hidden>
> >
> > To:
> > Steve Emmerson <address@hidden>
> >
> >
> >Hi Steve,
> >In the concluding phases of our NEXRAD Level II project, the
> >NWS Telecommunications Operations Center is tasked with developing
> >a level II monitoring and statistics capability. It is expected to be similar
> >to that developed by Carl Sinclair at OU for IRADS which you are
> >probably familiar with. It is based entirely on the use of rtstats for the
> >statistics gathering and monitoring.
> >While not directly involved in the development, I am still monitoring the
> >activities as the project manager, and from experience I know you are the
> >"answer man"!
> >The monitor tool is trying to track all 133 sites currently sending via LDM.
> >All 133 being received into a single LDM system as "CRAFT" feed type.
> >"rtstats" does not seem to be able to keep track of more than 99-100
> >sources. In alphabetical order listing, see attached, after the first 99 
> >(knqa)
> >only 1 product is recorded even though 200+ are actually received. Is this
> >a known limitation, or perhaps a configurable parameter?
> >I know that IRADS only receives from one region (28 to 40 sites each) into
> >a single system so for them a single "rtstats" instance would probably never
> >experience this problem.
> >Thanks for the help.
> >Warren
> >
> >
> >kabr 1118417377 20050610152935 20050610152935 389 2803416 0 0.51 1@1824
> >kabx 1118414510 20050610144102 20050610144102 320 2871377 0 0.28 1@1105
> >kakq 1118417377 20050610152949 20050610152935 231 2626657 -14 0.00 0@0000
> >kama 1118417377 20050610152933 20050610152933 474 5636931 0 0.37 2@2835
> >kamx 1118417377 20050610152933 20050610152934 446 16960636 1 1.05 3@1203
> >kapx 1118417377 20050610152934 20050610152935 227 2613826 1 0.84 3@0842
> >karx 1118417377 20050610152935 20050610152936 477 6475905 1 1.03 4@0029
> >katx 1118417377 20050610152934 20050610152935 229 2606013 1 0.36 1@1633
> >kbbx 1118417377 20050610152926 20050610152927 88 1133976 1 0.86 2@2405
> >kbgm 1118417377 20050610152928 20050610152928 229 2697553 0 0.38 2@2040
> >kbhx 1118417377 20050610152936 20050610152936 230 1787468 0 0.29 2@2010
> >kbis 1118417377 20050610152933 20050610152934 473 3803288 1 0.38 1@2917
> >kblx 1118417377 20050610152933 20050610152934 457 5620270 1 0.42 2@2421
> >kbmx 1118417377 20050610152844 20050610152933 367 4572327 49 48.44 49@0254
> >kbox 1118417377 20050610152933 20050610152933 231 1963780 0 0.23 2@1007
> >kbro 1118417377 20050610152930 20050610152930 230 2530469 0 0.28 1@0841
> >kbuf 1118417377 20050610152934 20050610152935 461 5893924 1 0.40 2@0155
> >kbyx 1118417377 20050610152929 20050610152930 454 16008197 1 0.75 3@2128
> >kcae 1118417377 20050610152935 20050610152936 473 6283307 1 0.33 2@1250
> >kcbw 1118417377 20050610152934 20050610152934 229 1998392 0 0.18 1@2456
> >kcbx 1118417377 20050610152930 20050610152930 380 4422051 0 0.49 1@1809
> >kccx 1118417377 20050610152933 20050610152934 473 4487559 1 0.36 3@1751
> >kcle 1118417377 20050610152933 20050610152934 374 4343552 1 0.32 2@1329
> >kclx 1118417377 20050610152932 20050610152932 389 5654239 0 0.43 2@1233
> >kcrp 1118417377 20050610152920 20050610152921 88 1388793 1 0.51 1@1400
> >kcxx 1118417377 20050610152932 20050610152932 230 2654422 0 0.58 3@2559
> >kcys 1118417377 20050610152931 20050610152932 389 4952747 1 2.98 87@0419
> >kdax 1118417377 20050610152913 20050610152913 87 1455933 0 0.57 2@2400
> >kddc 1118417377 20050610152931 20050610152933 478 9550394 2 1.16 4@0117
> >kdfx 1118417377 20050610152933 20050610152933 227 2227353 0 0.55 1@1458
> >kdgx 1118417377 20050610152934 20050610152935 229 2856748 1 0.36 1@1520
> >kdix 1118417377 20050610152929 20050610152929 229 1904124 0 0.18 1@0652
> >kdlh 1118417377 20050610152931 20050610152932 228 4650768 1 1.14 6@0137
> >kdmx 1118417377 20050610152927 20050610152929 228 3283007 2 1.07 3@2327
> >kdtx 1118417377 20050610152934 20050610152934 89 1100773 0 0.83 2@0732
> >kdvn 1118417377 20050610152918 20050610152936 227 2624433 18 17.77 19@0326
> >kdyx 1118417377 20050610152930 20050610152931 226 3060375 1 0.83 2@2354
> >keax 1118417377 20050610152933 20050610152934 374 5372968 1 0.73 3@0802
> >kemx 1118417377 20050610152928 20050610152928 88 815425 0 0.37 1@2330
> >kenx 1118417377 20050610152929 20050610152929 225 2425583 0 0.32 3@0009
> >kepz 1118417377 20050610152922 20050610152922 88 950718 0 0.38 1@2347
> >kesx 1118417377 20050610152930 20050610152930 88 926206 0 0.59 1@2831
> >kevx 1118417377 20050610152912 20050610152935 468 11197697 23 23.22 25@1600
> >kewx 1118417377 20050610152919 20050610152919 88 1479603 0 0.34 2@0437
> >kfcx 1118417377 20050610152935 20050610152936 387 5122102 1 0.25 1@1243
> >kfdr 1118417377 20050610152933 20050610152933 229 2451717 0 0.67 2@0733
> >kfdx 1118417377 20050610152931 20050610152932 230 2406846 1 0.74 2@0131
> >kffc 1118417377 20050610152935 20050610152936 229 3340904 1 0.38 2@0009
> >kfsd 1118417377 20050610152941 20050610152932 227 2353728 -9 0.00 0@0000
> >kfsx 1118417377 20050610152918 20050610152918 87 573740 0 0.31 2@0036
> >kftg 1118417377 20050610152932 20050610152933 228 2662373 1 0.67 2@1025
> >kfws 1118415669 20050610150009 20050610150014 2 11793 5 5.04 5@0009
> >kggw 1118417377 20050610152934 20050610152934 230 2746126 0 0.58 3@2134
> >kgjx 1118417377 20050610152933 20050610152933 230 3052960 0 1.18 3@2645
> >kgld 1118417377 20050610152932 20050610152932 461 6640813 0 1.08 5@0734
> >kgrb 1118417377 20050610152931 20050610152934 229 2544193 3 0.77 3@0614
> >kgrr 1118417377 20050610152931 20050610152931 231 2576132 0 0.90 3@0749
> >kgsp 1118417377 20050610152932 20050610152933 470 5718963 1 0.29 1@2224
> >kgwx 1118416279 20050610150957 20050610151021 1 901 24 23.42 23@0957
> >kgyx 1118417377 20050610152935 20050610152935 195 2162841 0 0.24 2@2632
> >khgx 1118417377 20050610152920 20050610152921 83 1148925 1 1.33 2@1358
> >khnx 1118417377 20050610152918 20050610152919 87 1208289 1 0.52 2@0212
> >khtx 1118417377 20050610152934 20050610152934 216 2442471 0 0.27 1@1953
> >kict 1118417377 20050610152935 20050610152936 462 10026884 1 1.49 4@1829
> >kicx 1118417377 20050610152930 20050610152930 88 337242 0 0.12 1@0852
> >kiln 1118417377 20050610152934 20050610152934 456 8497689 0 0.30 2@2551
> >kilx 1118417377 20050610152932 20050610152933 459 8135813 1 1.16 4@1243
> >kind 1118417377 20050610152929 20050610152930 227 3346858 1 0.87 3@0818
> >kinx 1118417377 20050610152932 20050610152933 229 3175009 1 0.43 2@0254
> >kiwa 1118417377 20050610152916 20050610152917 87 1255836 1 0.46 4@1806
> >kiwx 1118417377 20050610152932 20050610152933 387 3911318 1 0.74 6@2512
> >kjax 1118417377 20050610152934 20050610152935 440 10370513 1 0.45 1@2717
> >kjgx 1118417377 20050610152930 20050610152931 226 3313248 1 0.87 2@1621
> >kjkl 1118417377 20050610152934 20050610152935 112 1458740 1 1.22 4@2844
> >klbb 1118417377 20050610152926 20050610152934 457 6606562 8 8.39 10@0310
> >klch 1118417377 20050610152935 20050610152935 230 2839120 0 0.34 1@0135
> >klix 1118417377 20050610152935 20050610152935 229 3678377 0 0.49 2@1235
> >klnx 1118417377 20050610152931 20050610152931 230 1831297 0 0.52 2@0821
> >klot 1118417377 20050610152933 20050610152934 388 4618363 1 0.69 3@2914
> >klrx 1118417377 20050610152934 20050610152934 229 1916352 0 0.35 1@1103
> >klsx 1118417377 20050610152933 20050610152935 388 3788121 2 0.66 3@0035
> >kltx 1118417377 20050610152933 20050610152934 229 2084027 1 0.21 1@2624
> >klvx 1118417377 20050610152935 20050610152936 230 2973743 1 0.77 2@0147
> >klwx 1118417377 20050610152936 20050610152936 225 2381201 0 0.15 1@0008
> >klzk 1118417377 20050610152928 20050610152929 229 2783496 1 0.34 1@2128
> >kmaf 1118417377 20050610152933 20050610152934 475 11138048 1 0.62 2@1037
> >kmax 1118417377 20050610152919 20050610152920 88 601003 1 0.48 2@2358
> >kmhx 1118417377 20050610152928 20050610152928 226 1976298 0 0.17 1@2534
> >kmkx 1118417377 20050610152935 20050610152936 472 6244855 1 1.01 5@0049
> >kmlb 1118417377 20050610152934 20050610152935 386 8837246 1 0.49 2@2232
> >kmob 1118417377 20050610152935 20050610152936 477 8133587 1 0.40 1@0649
> >kmpx 1118412070 20050610140106 20050610140108 9 197078 2 1.39 2@0009
> >kmqt 1118417377 20050610152934 20050610152934 473 5049214 0 1.08 4@0527
> >kmrx 1118417377 20050610152930 20050610152931 229 2512985 1 0.32 1@1917
> >kmsx 1118408105 20050610125406 20050610125407 419 3674311 1 0.49 2@2848
> >kmtx 1118417377 20050610152932 20050610152932 227 1311913 0 0.18 1@2358
> >kmux 1118417377 20050610152932 20050610152932 89 714288 0 0.34 1@1408
> >kmvx 1118417377 20050610152941 20050610152934 231 2665748 -7 0.00 0@0000
> >knkx 1118417377 20050610152920 20050610152920 88 708091 0 0.26 0@1423
> >knqa 1118417133 20050610152531 20050610152532 1 7466 1 0.34 0@2531
> >koax 1118417316 20050610152834 20050610152835 1 9610 1 0.94 1@2834
> >kohx 1118413290 20050610142128 20050610142129 1 36344 1 0.96 1@2128
> >kotx 1118416950 20050610152228 20050610152228 1 5446 0 0.34 0@2228
> >kpah 1118416889 20050610152127 20050610152128 1 19837 1 0.76 1@2127
> >kpbz 1118417316 20050610152833 20050610152833 1 25363 0 0.69 1@2833
> >kpdt 1118415547 20050610145904 20050610145905 1 28839 1 0.94 1@5904
> >kpux 1118417377 20050610152935 20050610152937 1 12167 2 1.32 1@2935
> >krax 1118417316 20050610152835 20050610152835 1 11230 0 0.44 0@2835
> >krgx 1118417316 20050610152834 20050610152834 1 4527 0 0.04 0@2834
> >kriw 1118416889 20050610152127 20050610152127 1 5862 0 0.24 0@2127
> >krlx 1118417011 20050610152329 20050610152330 1 29410 1 0.82 1@2329
> >krtx 1118417133 20050610152530 20050610152531 1 18973 1 0.52 1@2530
> >ksfx 1118417255 20050610152734 20050610152734 1 17145 0 0.42 0@2734
> >ksgf 1118417377 20050610152936 20050610152936 1 9634 0 0.48 0@2936
> >kshv 1118417255 20050610152734 20050610152734 1 7217 0 0.15 0@2734
> >ksjt 1118417255 20050610152732 20050610152732 1 12402 0 0.33 0@2732
> >ksox 1118387121 20050610070519 20050610070520 1 20795 1 0.90 1@0519
> >ksrx 1118416828 20050610152027 20050610152027 1 4831 0 0.12 0@2027
> >ktbw 1118417255 20050610152732 20050610152732 1 35774 0 0.64 1@2732
> >ktfx 1118417133 20050610152532 20050610152532 1 5165 0 0.20 0@2532
> >ktlh 1118416950 20050610152154 20050610152229 1 11064 35 34.92 35@2154
> >ktlx 1118417377 20050610152935 20050610152935 1 4382 0 0.14 0@2935
> >ktwx 1118416584 20050610151621 20050610151622 1 15910 1 1.21 1@1621
> >ktyx 1118417316 20050610152832 20050610152833 1 20397 1 1.12 1@2832
> >kudx 1118415608 20050610150003 20050610150003 1 9820 0 0.63 1@0003
> >kuex 1118417255 20050610152731 20050610152733 1 15399 2 2.50 3@2731
> >kvnx 1118417377 20050610152936 20050610152936 1 10107 0 0.47 0@2936
> >kvtx 1118416950 20050610152228 20050610152229 1 4931 1 0.13 0@2228
> >kvwx 1118417011 20050610152329 20050610152330 1 5746 1 0.38 0@2329
> >kyux 1118415120 20050610145158 20050610145159 1 22388 1 0.57 1@5158
> >
> >
> >  
> >