This report contains information on four aspects of the IDD:
This report updates the Internet Data Distribution (IDD) Status and Progress Presentation IDD Status from the October, 2001, Unidata Policy Committee Meeting.
There has been a slight increase in the number of hosts and institutions running LDM servers:
| Date | Number of Institutions | Number of hosts |
|---|---|---|
| October, 2001 | 131 | 244 |
| February, 2002 | 155 | 268* |
* This number includes 19 new *.wunderground.com hosts that may or may not be at distinct institutions.
Currently 84% of reporting sites are running LDM 5.1.x, 7% are still running 5.0.x or earlier, and 9% are unknown.
The most popular feeds of 133 sites reporting are:
108 IDS|DDPLUS 100 HDS 91 MCIDAS 72 NEXRAD 60 PROFILER 50 NLDN 33 FNEXRAD 30 DIFAX 20 GPSSRC 18 ACARS 17 WSI 13 CONDUIT 11 NOGAPS
This is very similar to the October ranking, except that
Data feed volumes have risen significantly since last October, both due to the doubling of the number of NEXRAD products, and tripling the volume of CONDUIT model output:
Averages for UCAR LDM server
Feed Prods %Prods Feed Mbytes % of
per hr per hr Mbytes
NEXRAD 16044 34.3 CONDUIT 443.6 68.7
CONDUIT 14373 30.7 HDS 93.8 14.5
HDS 9181 19.6 NEXRAD 80.6 12.5
IDS|DDPLUS 6646 14.2 IDS|DDPLUS 6.7 1.0
FNEXRAD 429 0.9 DIFAX 5.4 0.8
WSI 32 0.1 MCIDAS 5.2 0.8
MCIDAS 15 0.0 FNEXRAD 3.6 0.6
NLDN 10 0.0 ACARS 3.4 0.5
PROFILER 11 0.0 PROFILER 1.9 0.3
DIFAX 7 0.0 WSI 1.9 0.3
ACARS 6 0.0 NLDN 0.0 0.0
total 46754 100.0 total 646.1 100.0
Steve Chiswell has created a short summary of results from the new rtstats program and IDD topology available at Information Regarding Real Time Statistics.
LDM Version 5.1.5 is currently under development. Source code is expected to be released by the end of March. Binary code is expected to be released by the end of May.
At the AMS Conference in January, Anne presented the results of a study using Network News Transport Protocol (NNTP) and Usenet, the Netnews network, to relay data.
Usenet is huge, consisting of tens of thousands of participating sites and millions of participants. It is completely decentralized, very heterogeneous, subject to attack, and yet it successfully delivers millions of articles per day. A large Usenet site may receive over 13 million articles per day (of which it probably only accepts a small percentage due to filtering and redundancy), and may deliver almost 700 gigabytes of articles.
In addition to the robustness of Usenet, the NNTP protocol supports several features that mitigate and possibly eliminate problems with the current LDM. For example, the current LDM allows only 31 feed types. In contrast, NNTP supports a virtually unlimited number of hierarchically structured newsgroups. This would allow us to taxonimize the data with a much finer granularity. Additionally, cross posting allows multiple views of the data.
Also, while the LDM requires that network topologies be determined manually, NNTP uses a "flooding" algorithm. In this approach, articles are redundantly relayed to multiple downstream sites. Sites are intentionally highly interconnected, with the result that an article reaches its destination by the fastest route possible. Also with this algorithm, no site is a single point of failure in the network. Thus, there is no notion of a "fail over" feed.
We decided to test relaying data over Usenet. Servers were installed in Boulder and in Washington D.C. Also, a server at the University of Oregon was engaged. We relayed Level II radar data from Boulder to Washington. The results were very encouraging. When the Boulder server peered only with the Washington server, i.e., the University of Oregon site was not included as a peer, 95% of products arrived at their destination within two seconds. When the University of Oregon site was included, 99% of the products arrived at their destination within two seconds. Also, in this second test, 5% of products passed through the University of Oregon. This means that for that 5%, the path via the University of Oregon was faster than the direct path from Boulder to Washington.
We intend to pursue this work further by peering with more sites and stressing the system by sending larger volumes of data.
The complete AMS presentation in the original Powerpoint format is available here. The complete AMS presentation in slightly scrambled htm format (due to automatic conversion) is available here.
Progress is being made on installing a NOAAport dish at NSF in Washington, D.C. The dish company has performed a site survey and NSF Facilities and the building management have given preliminary approval. We are currently awaiting formal approval from Cliff Jacobs before making the purchase request.
When installed, the ingest system will have a four channel Performance Technology card. There will be some savings over the cost of installing of a complete ingest system because a machine, atm, is already in place to relay the data. NSF recently became connected to Internet2, so the bandwidth is very good. It will be very helpful to have another downlink in the East, where congestion can be problematic.