Real-time, self-managing data flows -- Unidata will foster and support the existence of real-time data flows that encompass a broad range of Earth-system phenomena, can be accessed with ease by all constituents, and are self managing in respect to changing contents and user needs.
--A goal of Unidata 2008: Shaping the Future of Data Use in the Geosciences
Previous LDM version was 6.6.4. Current LDM version is 6.6.5
LDM release highlights since previous report:
Corrected the "pqact" utility's determination of the month associated with a data-product from the creation-time of the data-product and the day-of-the-month field in the product-identifier. This modification is tested extensively by executing the command "make check" in the pqact/ subdirectory.
Removed extraneous carriage returns from file pqact/wmo_header.c.
Modified the function surf_split() in the "pqsurf" program so that it uses a dynamically allocated buffer instead of a statically allocated one. This means that "pqsurf" can now handle arbitrarily large composite bulletins.
Modified the "pqact" utility's determination of the month associated with a data-product from the creation-time of the data-product and the day-of-the-month field in the product-identifier.
Changed the behavior of a downstream LDM upon reception of a COMINGSOON message whose data-product has zero length. Before, this would cause the downstream LDM to exit; now the data-product is simply rejected. Both LDM-6 and LDM-5 code were modified.
Improved the performance of the "scour" utility.
Added a call to exitIfDone() after the call to pq_sequence() in file "up6.c".
Corrected command that removes old ".*.info" files from the LDM user's home-directory in the "ldmadmin" script. The "-prune" option wasn't preventing the "find" process from descending into subdirectories.
Modified algorithm that determines when an upstream LDM should flush the connection to its downstream LDM. The modification of the algorithm introduced in version 6.6.1 appears to not flush the connection often enough --- resulting in bursts of data which can cause problems (e.g., the ORPG apparently has problems keeping up with bursts of NEXRAD Level II data). The modified algorithm will flush the connection the first time the end of the queue is hit and every 30 seconds thereafter if no relevant data is added to the queue.
Corrected (again) the "pqact" utility's determination of the month associated with a data-product from the creation-time of the data-product and the day-of-the-month field in the product-identifier. This is the primary reason for this release.
Added "-prune" option to execution of "find" in "ldmadmin" when removing old ".*.info" files from the LDM user's home directory.
Corrected the logic behind an upstream LDM sending something (at least a NULLPROC) every 30 seconds.
Demoted the "Exiting" message from "rpc.ldmd" from log-level NOTE to log-level INFO unless it's in response to a TERM signal (such as by "ldmadmin stop").
Modified the "flushing" algorithm of an upstream LDM. It used to flush the connection more often than every 30 seconds; now it flushes the connection no more often than every 30 seconds.
Added a persistent-state file for a downstream LDM. This file saves the metadata of the last, successfully-received data-product so that the next downstream LDM process that requests the same data from the same source can start where the previous process stopped. The files reside in the LDM user's home-directory and have the pattern ".*.info". This increases the startup performance of a downstream LDM and will greatly benefit LDM's with many REQUEST entries.
Changed format of distribution file from ".tar.Z" to ".tar.gz" because the "compress" utility is not available on my development workstation (due to IP restrictions) and the (now necessary) "gunzip" utility appears to be ubiquitous.
Corrected the "pqact" utility's determination of the month associated with a data-product from the creation-time of the data-product and the day-of-the-month field in the product-identifier.
Steve Emmerson's work on the LDM replacement was interrupted by necessary work on the current LDM and also by creation of a replacement for the UDUNITS package, which will be both a stand-alone package and incorporated into the next-generation netCDF package.
Even though GOES-South imagery has successfully been incorporated into the half-hourly GOES-East image sectors in the Unidata-Wisconsin IDD datastream, it is believed that inclusion of the available 15-minute scans would be of use to Unidata community members, especially those in South America. The additions of GOES-South imager sectors to the Unidata-Wisconsin datastream was endorsed by the Unidata User's Committee at its May, 2007 meeting.
Proposed Unidata-Wisconsin Datastream Changes (http://www.unidata.ucar.edu/committees/polcom/2007summer/statusreports/uniwisc_prop.html) presents examples of GOES-10 imagery that will be added to the Unidata-Wisconsin IDD datastream and discusses potential impacts to end-users.
NB: In order to correctly gauge real-time status of the IDD, it is important that all participating sites accurately maintain their system clocks. This is easily done through use of a Network Time Protocol daemon run on the local machine.
The cluster approach to toplevel IDD relay, first reported in the Spring 2005 status report, has been operational at the UPC since early summer 2005. The cluster, described in the June 2005 CommunitE-letter article Unidata's New IDD Cluster, routinely relays data to more than 400 downstream connections. Data input to the cluster nodes is approx. 4 GB/hr (0.97 TB/day); average data output by the cluster is approx. 246 Mbps (~2.7 TB/day); peak rates routinely exceeding 450 Mpbs (~4.9 TB/day).
We still consider the cluster to be an experiment in progress. Configurations change as we learn more about how the system performs, and new operating system releases are tested for performance.
The cluster approach to IDD relay has been adopted by NOAA/GSD (formerly FSL) and Penn State (using funds provided by the Unidata-administered Equipment Awards program). Unidata staff have fielded questions on implementing similar clusters at Unidata community sites (Texas A&M) that have expressed an interest in functioning as toplevel IDD relay nodes.