LDM and IDD Status Report
May, 2007

Steve Chiswell
Steve Emmerson
Mike Schmidt
Jeff Weber
Tom Yoksas

Real-time, self-managing data flows -- Unidata will foster and support the existence of real-time data flows that encompass a broad range of Earth-system phenomena, can be accessed with ease by all constituents, and are self managing in respect to changing contents and user needs.

--A goal of Unidata 2008: Shaping the Future of Data Use in the Geosciences

---

LDM Status Report

LDM

Previous LDM version was 6.4.6. Current LDM version is 6.6.3

Highlights since previous report:

• LDM:
  • Performance of upstream and downstream LDM-s during startup and termination has been improved.
  • Skipping data-products during reconnection due to invalid data-product creation-time has been eliminated.
  • Downstream LDM process now starts exactly where it left-off from previous LDM session.
  • LDM configuration-file now supports "include" statements.
• "pqact" utility:
  • Starts where it left-off from one LDM session to the next.
  • Logs warning when it processes oldest product in queue.
  • Identifies slow decoders by decoder command-line.
  • Allows up to 2048 arguments in decoder command-line (increased from 15).
  • Correctly handles "-overwrite" option, in absence of "-close" option, in FILE and STDIOFILE actions
  • Has new "-log" option for FILE and STDIOFILE actions.
• "pqinsert" utility now has "-i" option to compute MD5 signature from product-identifier rather than from product data (much faster for large products and necessary for TIGGE project).
• Steve Emmerson conducted an LDM training workshop at the southern region headquarters of the National Weather Service.

Next Generation LDM

Steve Emmerson's work on the LDM replacement was interrupted by necessary work on the current LDM and also by creation of a replacement for the UDUNITS package, which will be both a stand-alone package and incorporated into the next-generation netCDF package.

---

IDD Relay Cluster idd.unidata.ucar.edu

The cluster approach to toplevel IDD relay, first reported in the Spring 2005 status report, has been operational at the UPC since early summer 2005. The cluster, described in the June 2005 CommunitE-letter article Unidata's New IDD Cluster, routinely relays data to more than 400 downstream connections. Data input to the cluster nodes is approx. 4 GB/hr (0.97 TB/day); average data output by the cluster is approx. 246 Mbps (~2.7 TB/day); peak rates routinely exceeding 450 Mpbs (~4.9 TB/day).

We still consider the cluster to be an experiment in progress. Configurations change as we learn more about how the system performs, and new operating system releases are tested for performance.

The cluster approach to IDD relay has been adopted by NOAA/GSD (formerly FSL) and Penn State (using funds provided by the Unidata-administered Equipment Awards program). Unidata staff have fielded questions on implementing similar clusters at Unidata community sites (Texas A&M) that have expressed an interest in functioning as toplevel IDD relay nodes.

---

IDD Status Report

• Approximately 400 machines at some 210+ sites are running LDM-6 and reporting real time statistics. Unidata staff routinely assist in the installation of LDM-6 at user sites to help evolve the IDD as rapidly as possible.

• Additional CONDUIT data was added to data stream on August 9, 2006 increasing the daily volume of CONDUIT from 30 to 40 GB per day. GFS 0.5 degree grids were extended from 84 hours to 180 hours, and additional RUC2 forecast hours were added. Both data sets are provided in GRIB2. A new top level LDM server has been established at NOAA to accommodate CONDUIT.

• Real time LDM statistics can be found in the Real Time IDD Statistics web page. The topology and data flow information reflects changes in feed requests within a few minutes of changes. rtstats have proven to be most useful in diagnosing reception problems at sites, especially when universities have implemented restrictions on packets volume in and out of the campus without regard for their user needs.

  NB: In order to correctly gauge real-time status of the IDD, it is important that all participating sites accurately maintain their system clocks. This is easily done through use of a Network Time Protocol daemon run on the local machine.

• NWS continues to use LDM-6 to collect and relay NEXRAD Level II data. Results of performance monitoring studies being conducted by NOAA personnel (Blanchard) are shared with the UPC, and information gained is being used to guide code modifications that will be included in future releases.

• The IDD seamlessly transitioned to the use of DVB-S NOAAPORT ingest at the end of March, 2005. Currently, there are 10 sites known to be running the Unidata NOAAPORT reception software. 6 of these sites are acting as IDD injection nodes for NOAAPORT data.

• Internet2 bandwidth usage by the LDM protocol reached a high of approx. 21 TB/week the week of August 15, 2005. Weekly I2 usage statistics seen at Internet NetFlow: Weekly Reports are being updated sporadically (search for 'UNIDATA LDM' usage statistics in the Advanced Applications portion of Table 7, Detailed Application Types (Full Data Set)). The amount of IDD traffic being reported by I2 netflow statistics has been seen to fall as UCAR and other organizations are routing over the National Light Rail (NLR) instead of I2.

• The LDM-6 was selected for data collection activities in the THORPEX Interactive Grand Global Ensemble (TIGGE). The UPC continues to provide support as needed in the use or the LDM by TIGGE participants. The TIGGE stress testing activities were reported in the February, 2006 CommunitE-letter article THORPEX/TIGGE and the LDM.

---

For previous information, please refer to the April 2006 Users Committee status report.