IDD Status

Russ Rew, Steve Chiswell, Anne Wilson, Mike Schmidt
February 14, 2002


This report contains information on four aspects of the IDD:


IDD Size and Data Feed Volumes

This report updates the Internet Data Distribution (IDD) Status and Progress Presentation IDD Status from the October, 2001, Unidata Policy Committee Meeting.

Participation

There has been a slight increase in the number of hosts and institutions running LDM servers:

Date Number of Institutions Number of hosts
October, 2001 131 244
February, 2002 155 268*

* This number includes 19 new *.wunderground.com hosts that may or may not be at distinct institutions.

Currently 84% of reporting sites are running LDM 5.1.x, 7% are still running 5.0.x or earlier, and 9% are unknown.

Data Feed Volumes

The most popular feeds of 133 sites reporting are:

 108 IDS|DDPLUS
 100 HDS
  91 MCIDAS
  72 NEXRAD
  60 PROFILER
  50 NLDN
  33 FNEXRAD
  30 DIFAX
  20 GPSSRC
  18 ACARS
  17 WSI
  13 CONDUIT
  11 NOGAPS

This is very similar to the October ranking, except that

Data feed volumes have risen significantly since last October, both due to the doubling of the number of NEXRAD products, and tripling the volume of CONDUIT model output:

                Averages for UCAR LDM server
                                                      
     Feed    Prods  %Prods          Feed    Mbytes     % of  
             per hr                         per hr    Mbytes 
                                                      
    NEXRAD   16044   34.3         CONDUIT  443.6       68.7  
   CONDUIT   14373   30.7             HDS   93.8       14.5  
       HDS    9181   19.6          NEXRAD   80.6       12.5  
IDS|DDPLUS    6646   14.2      IDS|DDPLUS    6.7        1.0  
   FNEXRAD     429    0.9           DIFAX    5.4        0.8  
       WSI      32    0.1          MCIDAS    5.2        0.8  
    MCIDAS      15    0.0         FNEXRAD    3.6        0.6  
      NLDN      10    0.0           ACARS    3.4        0.5  
  PROFILER      11    0.0        PROFILER    1.9        0.3  
     DIFAX       7    0.0             WSI    1.9        0.3  
     ACARS       6    0.0            NLDN    0.0        0.0  
                                                      
     total   46754  100.0           total  646.1      100.0  


Real Time Statistics

Steve Chiswell has created a short summary of results from the new rtstats program and IDD topology available at Information Regarding Real Time Statistics.


LDM Update

LDM Version 5.1.5 is currently under development. Source code is expected to be released by the end of March. Binary code is expected to be released by the end of May.

At the AMS Conference in January, Anne presented the results of a study using Network News Transport Protocol (NNTP) and Usenet, the Netnews network, to relay data.

Usenet is huge, consisting of tens of thousands of participating sites and millions of participants. It is completely decentralized, very heterogeneous, subject to attack, and yet it successfully delivers millions of articles per day. A large Usenet site may receive over 13 million articles per day (of which it probably only accepts a small percentage due to filtering and redundancy), and may deliver almost 700 gigabytes of articles.

In addition to the robustness of Usenet, the NNTP protocol supports several features that mitigate and possibly eliminate problems with the current LDM. For example, the current LDM allows only 31 feed types. In contrast, NNTP supports a virtually unlimited number of hierarchically structured newsgroups. This would allow us to taxonimize the data with a much finer granularity. Additionally, cross posting allows multiple views of the data.

Also, while the LDM requires that network topologies be determined manually, NNTP uses a "flooding" algorithm. In this approach, articles are redundantly relayed to multiple downstream sites. Sites are intentionally highly interconnected, with the result that an article reaches its destination by the fastest route possible. Also with this algorithm, no site is a single point of failure in the network. Thus, there is no notion of a "fail over" feed.

We decided to test relaying data over Usenet. Servers were installed in Boulder and in Washington D.C. Also, a server at the University of Oregon was engaged. We relayed Level II radar data from Boulder to Washington. The results were very encouraging. When the Boulder server peered only with the Washington server, i.e., the University of Oregon site was not included as a peer, 95% of products arrived at their destination within two seconds. When the University of Oregon site was included, 99% of the products arrived at their destination within two seconds. Also, in this second test, 5% of products passed through the University of Oregon. This means that for that 5%, the path via the University of Oregon was faster than the direct path from Boulder to Washington.

We intend to pursue this work further by peering with more sites and stressing the system by sending larger volumes of data.

The complete AMS presentation in the original Powerpoint format is available here. The complete AMS presentation in slightly scrambled htm format (due to automatic conversion) is available here.


Status of NOAAport dish at NSF

Progress is being made on installing a NOAAport dish at NSF in Washington, D.C. The dish company has performed a site survey and NSF Facilities and the building management have given preliminary approval. We are currently awaiting formal approval from Cliff Jacobs before making the purchase request.

When installed, the ingest system will have a four channel Performance Technology card. There will be some savings over the cost of installing of a complete ingest system because a machine, atm, is already in place to relay the data. NSF recently became connected to Internet2, so the bandwidth is very good. It will be very helpful to have another downlink in the East, where congestion can be problematic.