LDM and IDD Status Report
Summer/Fall 2009

LDM Status Report

LDM version of previous report was 6.7.0. Current LDM is 6.8.1.

LDM release highlights since previous report:

• Improved installation:
  • The ldmadmin(1) configuration-file is copied, read, and rewritten with additional user-configurable parameters.
  • Improved configure(1) script:
    • Added checks for ar(1), rpcgen(1), netstat(1), ntpdate(8), and version 5 of perl(1).
    • Documented more significant environment variables listed by "configure --help".
  • Replaced use of $(INSTALL) in makefiles with cp(1) because that was always the value of the macro.
• Improved ldmadmin(1):
  • Added the ability to check the accuracy of the system clock. This new ability is highly user-configurable. The defaults are to check the clock but not to abort if it's off by too much.
  • Added removal of product-information (~/.*.info) files to the "delqueue" command if "$deleteInfoFiles" is true in the ldmadmin(1) configuration-file.
  • Added the new option "-M max_clients".
  • Added the new commands "printmetrics", "addmetrics", and "plotmetrics" (to graphically display LDM metrics).
  • Moved variables "$os" and "$release" to the configuration-file.
  • Improved responsiveness by changing default ntpdate(1) timeout from 20 seconds to 5.
• Improved LDM Server (rpc.ldmd):
  • Improved vetting (i.e., access-checking) of a client to prevent delays due to a time-out on the reverse DNS lookup, which is used to convert the client IP address into a hostname:
    • Moved vetting from the LDM server before fork(2)ing a child process to the fork(2)ed child process.
    • Made vetting a two-phase procedure: first with the client IP address and then, if and only if that fails, with the client hostname.
  • Added the new option "-M max_clients" (default: 256).
• scripts/plotMetrics:
  • Created this gnuplot(1) script for plotting the accumulated output of "ldmadmin addmetrics".

The NWS's ultimate plans for using the LDM to distribute NEXRAD-II data will likely require a new feature to be added to the LDM to support the low-bandwidth lines out of each radar site. In preparation for this and some other more user-friendly features, Steve is currently working on upgrading the LDM package to use more modern version-control, build procedures (including support for shared libraries), and user-configuration mechanisms. Among other things, this will permit more automatic installation and operation of the LDM.

IDD Status Report

• Approximately 505 machines at 250+ sites are running LDM-6 and reporting real time statistics. Unidata staff routinely assist in the installation of LDM-6 at user sites as a community service.

  There are several organizations/projects that use the LDM to move substantial amounts of data that do not report statistics to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private compaines, etc.).

• IDD toplevel relay node, idd.unidata.ucar.edu

  The cluster approach to toplevel IDD relay, has been operational at the UPC since early summer 2005. The cluster, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more than 600 downstream connections. Data input to the cluster nodes is approx. 5.5 GB/hr (0.13 TB/day); average data output is approx. 430 Mbps (~4.4 TB/day); peak rates routinely exceed 750 Mbps (~6.2 TB/day).

  The following lists the breakdown by feedtype of the data being relayed by Unidata's toplevel IDD relay, idd.unidata.ucar.edu.

  Data Volume Summary for idd.unidata.ucar.edu on 20090928.2218 [UTC]
  
  Maximum hourly volume  10851.011 M bytes/hour
  Average hourly volume   5479.081 M bytes/hour
  
  Average products per hour     198390 prods/hour
  
  Feed                           Average             Maximum     Products
                       (M byte/hour)            (M byte/hour)   number/hour
  NEXRAD2                1929.062    [ 35.208%]     2648.456    50122.583
  CONDUIT                1736.064    [ 31.685%]     5294.970    48805.458
  NGRID                   663.526    [ 12.110%]     1297.465    15715.250
  NEXRAD3                 383.731    [  7.004%]      480.972    30765.375
  HDS                     228.595    [  4.172%]      427.752    19239.896
  FNMOC                   183.238    [  3.344%]     1100.668     1895.292
  NIMAGE                  151.217    [  2.760%]      310.875      170.542
  FNEXRAD                  93.694    [  1.710%]      117.048       53.458
  EXP                      46.240    [  0.844%]       84.510      416.375
  IDS|DDPLUS               34.262    [  0.625%]       40.823    30984.292
  UNIWISC                  21.619    [  0.395%]       29.457       25.792
  DIFAX                     3.600    [  0.066%]       12.547        4.958
  GEM                       2.456    [  0.045%]       29.473      168.417
  FSL2                      1.777    [  0.032%]        2.012       21.875
  

  Currently seven real server nodes operating in two separate locations on the UCAR campus (in the UPC offices and in FL-2) and one director comprise idd.unidata.ucar.edu. A second and third director have been configured and will be added to the cluster to increase redundancy.

  The cluster approach to IDD relay has been adopted by NOAA/GSD and Penn State (using funds provided by the Unidata-administered Equipment Awards program). The cluster operated by Penn State is scheduled to assume toplevel relay responsibilities for CONDUIT data in early October.

• Real time LDM statistics can be found in Real Time IDD Statistics web page (http://www.unidata.ucar.edu/projects/idd/rtstats). The topology and data flow information reflects changes in feed requests within a few minutes of changes. rtstats continues to be most useful in diagnosing reception problems at sites, especially when sites have implemented restrictions on packets volume in and out of their campus without regard for their user needs.

• Internet2 (I2) bandwidth usage by the LDM protocol is routinely exceeding 30 TB/week (peak usage was approx. 33 TB/week the week of September 9, 2009). Weekly I2 usage statistics can be seen at Internet NetFlow: Weekly Reports (search for 'UNIDATA LDM' usage statistics in the Advanced Applications portion of Table 7).

• NWS use of LDM-6 to collect and relay NEXRAD Level II data

  NOAA is rearchitecting the way that NEXRAD Level II data will be collected and disseminated. Current plans for a Level II Refresh include collection of all Level II data at a primary site (NWSTG in Silver Springs, MD) with an active backup site (ROC in Norman, OK). All connections would be over the NOAAnet MPLS backbone. Data collected at the active site (primary or backup) would then be relayed to the MAX GigaPop (University of Maryland), which, in turn, will relay the data to the current set of top level IDD relay nodes (IRADS (OK), Purdue (IN), and the ERC (NC)) over Internet2 and to a small number of additional sites to be designated by the NWS.

  UPC staff (Emmerson, Schmidt, Yoksas) conducted a series of LDM-6 stress tests in support of the Level II Refresh. These tests were aimed at determining if an LDM at a single site could be used to collect and relay all Level II products from all NEXRADs now and in the future when Level II data volumes are expected to increase substantially. The tests demonstrated that a moderate machine (5 year old dual 2.8 Ghz Xeon PC with 6 GB of RAM) can serve in this capacity when running current distributions of LDM-6 and Linux (Unidata tests were conducted using LDM-6.8.0 on machines running Fedora 10).

  Besides providing expert advice on implementing the proposed Level II Refresh, the UPC staff also devised two alternative approaches and presented a cost/benefit analysis of all approaches. The NWS adopted one of the alternative designs.

  NB: It is intended that Unidata sites will continue to receive the data in the same way that they are currently: from one of the top level relay sites or from the top level relay nodes in Boulder, CO (UPC) or Vienna, VA (NSF/AGS).

• The NWS finished modifying the format of NEXRAD Level II products through deployment of Build 10.0 adding message type 31 in support of higher resolution 1/4 km x 0.5 degree radar operations, as well as future dual polarization mode data. The LDM product identifiers were modified to contain additional information about the data products. The new Level II data format allows for variable length records to preserve bandwith.

  The additional data in NEXRAD Level II products measurably increased the total volume of the IDD NEXRAD2 data feed.

• The LDM-6 continues to be used for data collection activities in the THORPEX Interactive Grand Global Ensemble (TIGGE). The UPC continues to provide support as needed in the use or the LDM by TIGGE participants. TIGGE stress testing activities were reported in the February, 2006 CommunitE-letter article THORPEX/TIGGE and the LDM.

• Currently, there are 16 sites known to be running the Unidata NOAAPORT ingestion software including Raytheon which is doing so in support of its use of Unidata GEMPAK which is informing their development of AWIPS-2. There is are potentially more sites using the code given that UW/SSEC makes it available to SSEC-licensed sites running McIDAS-XCD (McIDAS package that decodes NOAAPort-delivered data int McIDAS compatible formats). 7 community sites are acting as IDD injection nodes for NOAAPORT data.
  
  
• The Unidata Program Center has been granted permission from BOTH FNMOC and the CMC to deliver their model output directly to our community. FNMOC provides the COAMPS and NOGAPS model output and the CMC provides the GEM model output. Unidata has provided access to these data for the past 8 years, but on a "point-to-point" basis. This has led to delays for set up and undue hardship and resource use at both FNMOC and the CMC. To help insure we continue to get this data via the LDM-IDD we pursued the option of allowing the UPC to fan out these data and asking the CMC and FNMOC to solely feed the UPC. This took a small amount of convincing, but we gained success over the summer to fan out the model output from both centers to our community. This has made access to these models output easier and more efficient.

Next Generation LDM Status Report

Work has been proceeding, but there are still no tires to kick.

For previous information, please refer to the Spring 2009 User Committee status report.