IDD Statistics Monitoring System

Robb Kambic
June 1994

Description

The LDM statistics provide information about the reliability of links between source and sink nodes and the overall efficiency of the IDD topology. In addition to reflecting the state of the network between nodes, the statistics are affected by the version of the LDM software a site is running and the sites position in the IDD topology. The two parts of an LDM statistics report provide the following information:

Part 1

the number of products sent by a source by feed type;
the number of products received by sinks by feed type;
the percentage of products received by a sink;
the levels of the nodes in the IDD topology.

Part 2

the number of hours the LDM has been running;
the last time the server disconnected from the sink and the length of the disconnection (in seconds);
the cumulative number of products sent;
the max/min time for a source to complete delivery of a product to a sink;
the number of connections made to a sink;
the number of disconnections from a sink;
the cumulative time of all disconnections (in seconds).

LDM-Statistics Architecture

Part 1

Each node in the IDD topology collects LDM statistics from the local server via log files. The statistics are then saved locally and forwarded to UPC to be included into the Hourly LDM Statistics report, available on the UPC Web server. The report is constructed by extracting from the log files the number of products sent out by the source and the number of products received by the sink by feedtype (HRS, IDS, DDS|PPS, and MCIDAS). The percentage of products received is calculated by the following formula:

Percentage = number of products at sink / number of products at source

Since these calculations are done at the UPC, there is no need to copy logs between sites and the entire IDD topology is combined into one report. The sinks are numbered to indicate their levels in the IDD topology; for example, Sink_2 is a level-two node. The report has a time bin heading; this is the time the product arrived at the source and it remains with the product header throughout the entire routing process. This is a recent report:

1994060317

Source lightning.alden.com IDS 423 100.00 Source lightning.alden.com DDS|PPS 1787 100.00 Source lightning.alden.com HRS 1504 100.00 Source wxdata.ssec.wisc.edu MCIDAS 8 100.00 Sink_0 thelma.ucar.edu IDS 423 100.00 Sink_0 thelma.ucar.edu DDS|PPS 1787 100.00 Sink_0 thelma.ucar.edu HRS 1504 100.00 Sink_1 measun.nrrc.ncsu.edu IDS 423 100.00 Sink_1 measun.nrrc.ncsu.edu DDS|PPS 1794 100.00 Sink_1 measun.nrrc.ncsu.edu MCIDAS 8 100.00 Sink_1 measun.nrrc.ncsu.edu HRS 1509 100.00 Sink_1 drier.atmos.washington.e IDS 423 100.00 Sink_1 drier.atmos.washington.e DDS|PPS 1794 100.00 Sink_1 drier.atmos.washington.e MCIDAS 8 100.00 Sink_1 drier.atmos.washington.e HRS 1505 100.00 Sink_1 isclient.merit.edu IDS 423 100.00 Sink_1 isclient.merit.edu DDS|PPS 1788 100.00 Sink_1 isclient.merit.edu MCIDAS 8 100.00 Sink_1 isclient.merit.edu HRS 1506 100.00 Sink_1 baron.nldn.albany.edu IDS 423 100.00 Sink_1 baron.nldn.albany.edu DDS|PPS 1792 100.00 Sink_1 baron.nldn.albany.edu HRS 1513 100.00 Sink_1 atm.geo.nsf.gov IDS 423 100.00 Sink_1 atm.geo.nsf.gov DDS|PPS 1786 100.00 Sink_1 atm.geo.nsf.gov HRS 1504 100.00 Sink_2 broken.atms.purdue.edu IDS 423 100.00 Sink_2 broken.atms.purdue.edu DDS|PPS 3570 200.00 Sink_2 broken.atms.purdue.edu MCIDAS 8 100.00 Sink_2 broken.atms.purdue.edu HRS 1504 100.00 Sink_2 jed.atmo.arizona.edu IDS 414 98.00 Sink_2 jed.atmo.arizona.edu DDS|PPS 1771 99.00 Sink_2 jed.atmo.arizona.edu MCIDAS 8 100.00 Sink_2 jed.atmo.arizona.edu HRS 1476 98.00 Sink_2 namias.colorado.edu IDS 259 61.00 Sink_2 namias.colorado.edu DDS|PPS 1148 64.00 Sink_2 namias.colorado.edu HRS 960 64.00 Sink_2 sespe.atmos.ucla.edu DDS|PPS 1607 90.00 Sink_2 sespe.atmos.ucla.edu MCIDAS 4 50.00 Sink_2 vision.soest.hawaii.edu DDS|PPS 1548 87.00 Sink_2 vision.soest.hawaii.edu HRS 1146 76.00 Sink_2 quince.rsmas.miami.edu IDS 145 34.00 Sink_2 quince.rsmas.miami.edu DDS|PPS 636 36.00 Sink_2 quince.rsmas.miami.edu MCIDAS 4 50.00 Sink_2 quince.rsmas.miami.edu HRS 342 23.00 Sink_2 metlab12.met.fsu.edu IDS 380 90.00 Sink_2 metlab12.met.fsu.edu DDS|PPS 1629 91.00 Sink_2 metlab12.met.fsu.edu MCIDAS 6 75.00 Sink_2 metlab12.met.fsu.edu MCIDAS 6 75.00 Sink_2 metlab12.met.fsu.edu HRS 1213 81.00 Sink_2 gsosun2.gso.uri.edu HRS 63 4.00

Part 2

The rest of the LDM statistics information is collected by sending a signal (-USR1) to the LDM server. The information is stored internally to the program, therefore it depends on the running time of the program to put the statistics into perspective. The statistical information is stored locally in a file called node_to_node.log. In the future, a report will be developed to use extracted information to create a reliability rating for each of the nodes in the topology. The header of the report is:

                       LDM Statistics Report
                                    for
 
               Data Source sourcename to Data Sink sinkname
 
 
Report Time     Up    Last            Secs  Total     Latency   Con  Dis Total
                Hrs   Disconnect      Down  Prods     Min  Max            Down
--------------------------------------------------------------------------------

Summary

The LDM statistics system is a valuable tool for assessing the routing for all the nodes in the IDD topology, and it will be an important diagnostic in maintaining a high level of reliability for the IDD system. The statistics are being modified to use the least amount of resources, therefore the mechanisms for collection will probably change in the future. We may, for example, include enhancements that will alert LDM maintainers to the status of the LDM servers.