NLDM Update: Using NNTP-based Software to Relay Data

Anne Wilson
May 13, 2003



The CONDUIT data stream, which constitutes about 75% of the total IDD volume, has been relayed continuously for several months from Boulder to Washington, D.C., using NNTP-based NLDM.   Statistics regarding data relay are updated every 5 minutes and posted on the new Unidata web site.  The statistics  now include product count, byte volume, maximum latencies, average latencies, path information, and number of connections binned across a variety of time windows and bin sizes.

Statistics are showing timely data delivery with maximum latencies generally under fifteen seconds, and average latencies generally under 1.5 seconds.   Using the NNTP flooding algorithm for routing, data products can travel either directly from Boulder to Washington, or from Boulder to the University of Oregon, possibly out to Usenet, then to Washington.  The vast majority of products take the direct path, but a small percentage of products regularly take alternative paths, arriving at their destination more quickly than via the direct path.

In May, the students involved in the University of Colorado Senior Software Engineering project to build a receive only NNTP and Java based data management package completed the project and presented it to us.  The software, currently called LeafNLDM, has run successfully under Linux, Solaris, FreeBSD and a Windows laptop.  The software kept up with the CONDUIT data rate with acceptable latencies.  It also provides functionality to file the data or pipe the data to another process like our current LDM.

We have hired one of these students to work with us for the summer in improving LeafNLDM, with the goals of improving the user interface, providing relay out capability so that statistics can be sent, and demonstrating an end-to-end receive and display system using the IDV for visualization.  This student, Mike Linck, will start on May 23 and will be funded for up to four months as a visitor.