LDM

Status Report: Oct 2010 - May 2011

Steve Emmerson, Tom Yoksas, Mike Schmidt

Strategic Focus Areas

The LDM group's work supports five of the funding proposal's six focus areas:

  1. Broadening participation and expanding community services
    Dispite the existence of "pull" clients, (e.g., the IDV), people still want the ability to subscribe to meteorological data and have it automatically processed as soon as it's available. The LDM is one of the few mechanisms for doing this with high-volume data flows.
  2. Advancing data services
    As institutions adopt the LDM to receive IDD data and to distribute their own locally-generated data, more data becomes available to the larger community (quid pro quo).
  3. Developing and deploying useful tools
    The LDM is, apparently, useful and is being deployed more and more.
  4. Providing leadership in cyberinfrastructure
    You know you're providing leadership in cyberinfrastructure when organizations like the NWS and Raytheon start coming to you for advice on using the LDM.
  5. Promoting diversity by expanding opportunities
    Even small colleges and high-schools can use the LDM to receive and process massive amounts of data.

Activities Since the Last Status Report

Released Version 6.9

  • Supports automatic reconciliation of the size of the product-queue and the maximum acceptable latency parameter to ensure that duplicate products are not transmitted. The LDM system can now be configured to
    • Transparently increase the size of the queue; or
    • Transparently decrease the size of the maximum acceptable latency parameter; or
    • Report the problem.
  • Has the ldmadmin check command, which
    • Checks that the LDM is running;
    • Checks the accuracy of the system clock;
    • Checks for a recent insertion of a product into the queue; and
    • Reconciles the size of the queue with the maximum acceptable latency parameter.
  • Has the wasReceived program, which checks for a data-product of a given feedtype and pattern being received in the last given amount of time.
  • Uses an XML-based registry rather than the ldmadmin(1) configuration-file to store dynamic configuration parameters.
  • Contains bug fixes and documentation improvements.

Investigated Slow readnoaaport(1) Problem

We were contacted by Raytheon employees working on the AWIPS-II project for the National Weather Service about a problem they saw with our (partially supported) NOAAPORT ingest package. I say "partially supported" because 1) it's not part of the LDM package; 2) while it's both publicly and freely available, we haven't promoted it; and 3) its release-engineering was poor.

Raytheon uses both our NOAAPORT ingest package and our LDM package to get data into the AWIPS-II system and intends to bundle both into the AWIPS-II package that is delivered to all the Weather Forecast Offices.

The problem they saw is that, under certain circumstances, the insertion of data-products into the LDM product-queue couldn't keep up with the reception of data-products from the satellite broadcast. The circumstances under which this occurs were both highly idiosyncratic (occurred on one system but not a seemingly identical one), and non-intuitive (the problem didn't occur on slow systems but did on faster ones, etc.).

We've since discovered that the problem lay in a change to the IP fragment reassembly mechanism in Red Hat Linux. Subsequent modification of an IP tuning parameter appears to fix the problem for the most part.

There's likely to be more to the story, however, because some systems still show slight gaps in reception -- even after the IP parameter is adjusted.

Planned Activities

Ongoing Activities

We plan to continue the following activies:

  • Support LDM users
    • Email, phone, etc.
    • Training workshops
  • Incrementally improve the LDM as necessary

New Activities

We plan to organize or take part in the following:

  • Fully support the Unidata NOAAPORT package.
  • Improve the efficiency of the Unidata NOAAPORT package in an attempt to eliminate the slight gaps in reception that remain and in anticipation of significant increases to the data-rate of the NOAPORT broadcast.
  • Investigate incorporating changes to the LDM that Raytheon made to enable it to provide data to AWIPS-II into the current LDM so that our AWIPS-II users won't have to manage two different LDM-s.