LDM and IDD Status Report
April 2009

Steve Emmerson
Mike Schmidt
Jeff Weber
Tom Yoksas

March 23, 2009

---

LDM Status Report

LDM version of previous report was 6.6.5. Current LDM is 6.7.0.

LDM release highlights since previous report:

• Installation:
  • Added rpcgen(1) output to distribution (e.g., ldm6_clnt.c, ldm_xdr.c) and removed unconditional use of rpcgen(1) from makefiles. This was done because rpcgen(1) on a Mac OS/X 10.5.1 (Darwin 9.1.1) system generates output that's incompatible with the "rpc" subpackage.
  • Added the typedef "prod_class_t" to the header-file "ldm.h" and changed all code and documentation to use it instead of the typedef "prod_class". This was done to allow "ldm.h" to be processed by a C++ compiler. The typedef "prod_class" still exists for backward compatibility -- though it is no longer used in the code. Both C and C++ programmers should now use "prod_class_t" instead of "prod_class".
  • Added "typedef" for "rpc_inline_t" to "rpc/types.h" to support output of rpcgen(1) on SunOS 5.11 systems.
  • Ported to Fedora 8. The log file of the system log daemon is "/var/run/rsyslogd.pid".
  • Ported to Darwin 9: corrected misdefinition of OPEN_MAX as 10240 by ensuring that the XOPEN_SOURCE and XOPEN_SOURCE_EXTENDED macros are defined on that platform.
• Feedtypes:
  • Changed the primary name of the FT27 feedtype from "NNEXRAD" to "NEXRAD3" to accomodate such data regardless of source. "NNEXRAD" is an understood alias for that feedtype.
• ldmadmin(1):
  • Added checking of LDM configuration-file syntax to "start" command.
  • Modified the "pqactcheck" command. By default, it now checks the syntax of every pqact(1) configuration-file that is associated with an EXEC entry for pqact(1) in the LDM configuration-file. LDM configuration-file entries like "EXEC pqact -f CONDUIT" (i.e., with options but with no explicit configuration-file) are correctly handled.
  • Changed the default size of the product-queue from 400 megabytes to 500 megabytes.
  • Added initialization of local variables to functions in the ldmadmin(1) script to accomodate Perl interpreters that no longer perform default initialization.
• rpc.ldmd(1) (LDM server):
  • Adapted parsing of timestamps in .*.info files to Darwin 9.1.1: replaced use of strptime() with sscanf() (strptime() conversion specification must be separated by non-alphanumeric characters).
• pqact(1):
  • Made each decoder its own process-group leader. This should improve the performance of decoders like GEMPAK's dcgrib2(1).
  • Added "-metadata" and "-nodata" options to the PIPE action of pqact(1).
  • Changed behavior if a corrupt state-file is read. Before, the process would exit. Now, it continues with the most recent data-product (same as if the state-file didn't exist).
  • Adapted pqsurf(1) to handle multiple BOYC reports in composite SHIP bulletins.
  • Added code to the parser of feedtype expressions to prevent an infinite loop upon encountering two consecutive UNION (|) operators.
• pq(3):
  • Removed retry attempt from write-locking fcntl() call in file "pq/pq.c" because only fatal signals could occur.

---

IDD Status Report

• Approximately 470 machines at 240+ sites are running LDM-6 and reporting real time statistics. Unidata staff routinely assist in the installation of LDM-6 at user sites to help evolve the IDD as rapidly as possible. There are several organizations/projects that use the LDM to move substantial amounts of data that do not report statistics to Unidata: NASA, Spain, South Korea, private compaines, etc.).
• Real time LDM statistics can be found in the Real Time IDD Statistics web page (http://www.unidata.ucar.edu/projects/idd/rtstats). The topology and data flow information reflects changes in feed requests within a few minutes of changes. rtstats continue to be most useful in diagnosing reception problems at sites, especially when sites have implemented restrictions on packets volume in and out of their campus without regard for their user needs.
• IDD toplevel relay node, idd.unidata.ucar.edu

  The cluster approach to toplevel IDD relay, has been operational at the UPC since early summer 2005. The cluster, described in the June 2005 CommunitE-letter article Unidata's New IDD Cluster, routinely relays data to more than 550 downstream connections. Data input to the cluster nodes is approx. 3.6 GB/hr (0.086 TB/day); average data output by the cluster is approx. 406 Mbps (~4.2 TB/day); peak rates routinely exceeding 750 Mbps (~6.2 TB/day).

  The following lists the breakdown by feedtype of the data being relayed by Unidata's toplevel IDD relay, idd.unidata.ucar.edu.

  Data Volume Summary for idd.unidata.ucar.edu on 20090322.2120 [UTC]
  
  Maximum hourly volume   8158.960 M bytes/hour
  Average hourly volume   4349.157 M bytes/hour
  
  Average products per hour     186288 prods/hour
  
  Feed                           Average             Maximum     Products
                       (M byte/hour)            (M byte/hour)   number/hour
  CONDUIT                1805.118    [ 41.505%]     4886.608    52606.213
  NEXRAD2                1166.688    [ 26.826%]     1525.352    44972.213
  NGRID                   650.972    [ 14.968%]     1092.804    15356.596
  HDS                     218.344    [  5.020%]      440.370    18361.894
  NEXRAD3                 210.531    [  4.841%]      247.040    25140.170
  NIMAGE                  151.581    [  3.485%]      240.715      176.851
  FNEXRAD                  75.659    [  1.740%]       99.451       52.340
  IDS|DDPLUS               33.528    [  0.771%]       39.944    29377.511
  UNIWISC                  20.827    [  0.479%]       28.017       18.979
  EXP                      10.296    [  0.237%]       13.108      188.532
  DIFAX                     3.713    [  0.085%]       12.869        5.191
  FSL2                      1.889    [  0.043%]        2.081       22.149
  LIGHTNING                 0.010    [  0.000%]        0.029        9.638
  

  Currently six real server nodes operating in two separate locations on the UCAR Foothills Lab campus (in the UPC offices and in FL-2) and one director comprise idd.unidata.ucar.edu. A second and third directors have been configured for added redundancy.

  The cluster approach to IDD relay has been adopted by NOAA/GSD and Penn State (using funds provided by the Unidata-administered Equipment Awards program). The cluster operated by Penn State is scheduled to assume toplevel relay responsibilities for CONDUIT data in early October.

• Internet2 (I2) bandwidth usage by the LDM protocol continues at about 19.5 TB/week (peak usage was approx. 21 TB/week the week of August 15, 2005). Weekly I2 usage statistics can be seen at Internet NetFlow: Weekly Reports (search for 'UNIDATA LDM' usage statistics in the Advanced Applications portion of Table 7). The amount of IDD traffic being reported by I2 netflow statistics fell when UCAR and other organizations began routing preferentially over National Lambda Rail (NLR) instead of I2. More recently, a movement back to use of I2 has resulted in increased LDM netflow statistics.
• NWS continues to use LDM-6 to collect and relay NEXRAD Level II data. Results of performance monitoring studies being conducted by NOAA personnel (Blanchard) are shared with the UPC, and information gained is being used to guide code modifications that will be included in future releases.
• The NWS has finished modifying the format of NEXRAD Level II products through deployment of Build 10.0 adding message type 31 in support of higher resolution 1/4 km x 0.5 degree radar operations, as well as future dual polarization mode data. The LDM product identifiers were modified to contain additional information about the data products. The new Level II data format allows for variable length records to preserve bandwith.

  The additional data in NEXRAD Level II products measurably increased the total volume of the IDD NEXRAD2 data feed.

• The LDM-6 continues to be used for data collection activities in the THORPEX Interactive Grand Global Ensemble (TIGGE). The UPC continues to provide support as needed in the use or the LDM by TIGGE participants. TIGGE stress testing activities were reported in the February, 2006 CommunitE-letter article THORPEX/TIGGE and the LDM.
• Currently, there are 13 sites known to be running the Unidata NOAAPORT reception software including Raytheon which is doing so in support of its use of Unidata GEMPAK which is informing their development of AWIPS-2. 7 community sites are acting as IDD injection nodes for NOAAPORT data.

---

Next Generation LDM Status Report

Much of the work on the next-generation LDM since the last report consited of the following:

• Documenting and conducting two architectural reviews.
• Documenting the major decisions, rationale, and consequences in the design of the new system.

Steve Emmerson also worked on releasing the next-generation UDUNITS package, UDUNITS-2.

---

For previous information, please refer to the Spring 2008 User Committee status report.