LDM

Status Report: Oct 2010 - Apr 2011

Steve Emmerson, Tom Yoksas, Mike Schmidt

Strategic Focus Areas

The LDM group's work supports five of the funding proposal's six focus areas:

  1. Broadening participation and expanding community services
    Dispite the existence of "pull" clients, (e.g., the IDV), people still want the ability to subscribe to meteorological data and have it automatically processed as soon as it's available. The LDM is one of the few mechanisms for doing this with high-volume data flows.
  2. Advancing data services
    As institutions adopt the LDM to receive IDD data and to distribute their own locally-generated data, more data becomes available to the larger community (quid pro quo).
  3. Developing and deploying useful tools
    The LDM is, apparently, useful and is being deployed more and more.
  4. Providing leadership in cyberinfrastructure
    You know you're providing leadership in cyberinfrastructure when organizations like the NWS and Raytheon start coming to you for advice on using the LDM.
  5. Promoting diversity by expanding opportunities
    Even small colleges and high-schools can use the LDM to receive and process massive amounts of data.

Activities Since the Last Status Report

Released Version 6.9

  • Supports automatic reconciliation of the size of the product-queue and the maximum acceptable latency parameter to ensure that duplicate products are not transmitted. The LDM system can now be configured to
    • Transparently increase the size of the queue; or
    • Transparently decrease the size of the maximum acceptable latency parameter; or
    • Report the problem.
  • Has the ldmadmin check command, which
    • Checks that the LDM is running;
    • Checks the accuracy of the system clock;
    • Checks for a recent insertion of a product into the queue; and
    • Reconciles the size of the queue with the maximum acceptable latency parameter.
  • Has the wasReceived program, which checks for a data-product of a given feedtype and pattern being received in the last given amount of time.
  • Uses an XML-based registry rather than the ldmadmin(1) configuration-file to store dynamic configuration parameters.
  • Contains bug fixes and documentation improvements.

Investigating Slow readnoaaport(1) Problem

We've been contacted by Raytheon employees working on the AWIPS-II project for the National Weather Service about a problem they're seeing with our NOAAPORT ingest package that we (sort of) support. I say "sort of" because 1) it's not part of the LDM package; 2) while both publicly and freely available, we don't promote it; and 3) its release-engineering is poor. Raytheon uses both our NOAAPORT ingest package and our LDM package to get data into the AWIPS-II system and intends to bundle both into the AWIPS-II package that is delivered to all the Weather Forecast Offices.

The problem they're seeing is that, under certain circumstances, the insertion of data-products into the LDM product-queue can't keep up with the reception of data-products from the satellite broadcast. The circumstances under which this occurs are both highly idiosyncratic (occurs on one system but not a seemingly identical one), and non-intuitive (the problem doesn't occur on slow systems but does on faster ones, etc.).

We're working on it.

Planned Activities

Ongoing Activities

We plan to continue the following activies:

  • Support LDM users
    • Email, phone, etc.
    • Training workshops
  • Incrementally improve the LDM as necessary

New Activities

We plan to organize or take part in the following:

  • Resolve the slow readnoaaport(1) problem that the AWIPS-II project at Raytheon has seen on some of its systems.
  • Investigate incorporating changes to the LDM that Raytheon made to enable it to provide data to AWIPS-II into the current LDM so that our AWIPS-II users won't have to manage two different LDM-s.