NLDM Status Report

 Anne Wilson

 September 25, 2003

Software Development

We are now testing two versions of NNTP based data relay software. One is based on INN, a popular open source implementation of the NNTP protocol, written in C. The other is a Java implementation initially developed by a team of students but since delegated to our visitor, Mike Linck (who has written a status report about his work). The Java version, JNLDM, has been running successfully on a UNIX box, and is now being tested on a Windows machine.  Although originally intended to be a receive only client, Mike added relay out capability so that it can send statistics.  It is possible that this relay capability could support data relay, although we have not yet tested that.

Statistics

Version 1 of the NLDM statistics packages, including both user and developer documentation, is nearly complete. The statistics have been expanded to handle multiple reporting hosts and multiple feed types. The web page showing these statistics is "Current NLDM Statistics" .  On that page viewers can select among statistics for a variety of machines, including the Windows machine, domain.unidata.ucar.edu.

Statistics are calculated for a five different window and bin size combinations:


The statistics show the following information:

Like the products, statistics are relayed using news server technology.

The statistics will be extremely valuable in configuring and testing this network.  For example, we'll be able to tune local configurations, such as the minimum and maximum number of connections between two peers.   (In both versions of the software connections are created and destroyed dynamically in response to some queue parameters.)  Also, we'll be able to evaluate peering as the path statistics will illustrate relative speed between two hosts.

Performance

We have been relaying CONDUIT data to atm.geo.nsf.gov in Washington, D.C., since January.  As of this writing only the CONDUIT feed type is being relayed, but it is expected that the full feed will be relayed soon.

We had a successful although unplanned demonstration of the automated routing provided by the NNTP flooding algorithm on the night of the 18th of September (when Isabelle hit Washington, D.C.). At that time atm.geo.nsf.gov lost Internet2 connectivity, although it maintained connectivity to the commodity internet.  Normally the vast majority of products, 98 percent or better, take a direct path from imogene.unidata.ucar.edu to atm.geo.nsf.gov. But during this difficulty, the statistics clearly showed that the percentage of products taking the direct path dropped to approximately 50% to 80%, while the other paths available rose in frequency. Significantly, there was no observable change in product latencies.  Some relevant graphs from the statistics page at that time are shown here.
 

Maximum Latencies

Maximum Latencies

Average Latencies

Average Latencies

Percent of Products Taking Path imogene.unidata.ucar.edu -> atm.geo.nsf.gov

Percent of Products Taking Path imogene.unidata.ucar.edu -> canoe.uregon.edu -> atm.geo.nsf.gov

Percent of Products Taking Path imogene.unidata.ucar.edu -> arclight.uoregon.edu -> atm.geo.nsf.gov

We are building a prototype network using imogene as the ingest machine.  Currently all hosts in the network are Unidata machines, although installation on a machine at Texas A & M is underway at the time of this writing.  We will be soliciting volunteers to be beta sites for this network.  If anyone on the User Committee or the Policy Committee is interested in participating in this prototype, please contact Anne Wilson.