LDM and IDD Status Report

Steve Chiswell
Steve Emmerson
Mike Schmidt
Jeff Weber
Tom Yoksas

March 24, 2006

Real-time, self-managing data flows -- Unidata will foster and support the existence of real-time data flows that encompass a broad range of Earth-system phenomena, can be accessed with ease by all constituents, and are self managing in respect to changing contents and user needs.

--A goal of Unidata 2008: Shaping the Future of Data Use in the Geosciences

LDM Status Report

LDM Versions 6.4.2, 6.4.3, 6.4.4, and 6.4.5 have been released since the last report.
Release highlights:
- Improved support for "field project" LDM-s whose host can receive a different IP address every time it connects to the Internet (e.g., the UCAR HIAPER jet)
- Greatly reduced CPU load on startup if an upstream host is unavailable
- Reduced the frequency of transfer-mode switching between PRIMARY and ALTERNATE by downstream LDM-s
- Improved readability of log messages by fully-qualifing the identifier of upstream hosts, if possible
- Improved rtstats(1) by enabling it to report more than 2^32 bytes of received data
- Added support for 4 GB product-queues on 32-bit systems running Free BSD 4
- Improved efficiency of product-queue access on 32-bit systems whose LDM-s have been configured to support large files but that, nevertheless, use a small product-queue
- Improved efficiency of pqact(1)'s PIPE action
- Improved log messages

IDD Relay Cluster idd.unidata.ucar.edu

The cluster approach to toplevel IDD relay, first reported in the Spring 2005 status report, has been operational at the UPC since early summer 2005. The cluster, described in the June 2005 CommunitE-letter article Unidata's New IDD Cluster, routinely relays data to between 250 and 290 downstream connections at an average rate of approximately 170 Mbps (~1.8 TB/day), with peak rates routinely exceeding 290 Mbps (~3 TB/day). Live stress testing of the cluster demonstrated that the limiting factor for data relay was the local network bandwidth which is 1 Gbps at UCAR!

We consider the cluster setup to still be experimental. Configurations change as we learn more about how the system performs.

The cluster approach to IDD relay has been adopted by NOAA/GSD (formerly FSL) and is being implemented at Penn State with funds provided by the Unidata-administered Equipment Awards program. Recently, Unidata staff have fielded questions on implementing similar clusters at Unidata community sites (Texas A&M and University of Nebraska-Lincoln) that have expressed an interest in functioning as toplevel IDD relay nodes.

Next Generation LDM Update

Steve Emmerson is working on a replacement system for the LDM (codenamed "Hermes") that will combine the best features of the LDM, INN (i.e., netnews), and more modern peer-to-peer systems such as BitTorrent. The work is currently in the design stage. Design features include

Support for multiple operating-systems via Java
Reduced bandwidth usage by allowing every site to receive only desired data
Support for dynamic and redundant connection topology
Download initiation via web browsers
Support for both static and dynamic datasets (e.g., case-studies, streaming data)
Infinitely-extensible product-namespace
Built-in (but optional) security via secure-socket connections and cryptographic validation of data requests
Graphical user interface
Scalable to the world

IDD Status Report

328 machines at 173 sites are running LDM-6 and reporting real time statistics. Unidata staff routinely assist in the installation of LDM-6 at user sites to help evolve the IDD as rapidly as possible.
Real time LDM statistics can be found in the Real Time IDD Statistics web page. The topology and data flow information reflects changes in feed requests within a few minutes of changes. rtstats have proven to be most useful in diagnosing reception problems at sites, especially when universities have implemented restrictions on packets volume in and out of the campus without regard for their user needs.
NB: In order to correctly gauge real-time status of the IDD, it is important that all participating sites accurately maintain their system clocks. This is easily done through use of a Network Time Protocol daemon run on the local machine.
NWS continues to use LDM-6 to collect and relay NEXRAD Level II data. LDM-6.0.14 was used in the initial NWS build, but is slowly being replaced at participating nodes. Results of performance monitoring studies being conducted by NOAA personnel (Blanchard) are shared with the UPC, and information gained is being used to guide code modifications that will be included in future releases.
The IDD seamlessly transitioned to the use of DVB-S NOAAPORT ingest at the end of March, 2005. Currently, there are 10 sites known to be running the Unidata NOAAPORT reception software. 6 of these sites are acting as IDD injection nodes for NOAAPORT data.
Internet2 bandwidth usage by the LDM protocol reached a new high of approx. 21 TB/week the week of August 15. Weekly I2 usage statistics can be seen at Internet NetFlow: Weekly Reports. Search for 'UNIDATA LDM' usage statistics in the Advanced Applications portion of Table 7, Detailed Application Types (Full Data Set).
After extensive data relay testing by personnel at the ECMWF (Reading, England), NCAR/SCD (Boulder, Colorado), and the UPC, the LDM-6 was selected for data collection activities in the THORPEX Interactive Grand Global Ensemble (TIGGE). During the testing, the LDM-6 was demonstrated to be able to relay over 17 GB/hr in a a single datastream originating at the ECMWF and being received at NCAR/SCD. The TIGGE stress testing activities were reported in the February, 2006 CommunitE-letter article THORPEX/TIGGE and the LDM.

For previous information, please refer to the April 2005 Users Committee status report.