September 20, 2007
Real-time, self-managing data flows -- Unidata will foster and support the existence of real-time data flows that encompass a broad range of Earth-system phenomena, can be accessed with ease by all constituents, and are self managing in respect to changing contents and user needs.
-- A goal of Unidata 2008: Shaping the Future of Data Use in the Geosciences
LDM version of previous report was 6.6.3. Current LDM version is 6.6.5. (Beta LDM version is 6.7.0.0.)
LDM release highlights since previous report:
Modified "ldmadmin pqactcheck". By default, it now checks the syntax of every "pqact" configuration-file that is associated with an EXEC entry for "pqact" in the LDM configuration-file.
Corrected the "pqact" utility's determination of the month associated with a data-product from the creation-time of the data-product and the day-of-the-month field in the product-identifier. This modification is tested extensively by executing the command "make check" in the pqact/ subdirectory.
Modified the function surf_split() in the "pqsurf" program so that it uses a dynamically allocated buffer instead of a statically allocated one. This means that "pqsurf" can now handle arbitrarily large composite bulletins.
Modified the "pqact" utility's determination of the month associated with a data-product from the creation-time of the data-product and the day-of-the-month field in the product-identifier.
Changed the behavior of a downstream LDM upon reception of a COMINGSOON message whose data-product has zero length. Before, this would cause the downstream LDM to exit; now the data-product is simply rejected. Both LDM-6 and LDM-5 code were modified.
Improved the performance of the "scour" utility.
After being diverted by development of a new UDUNITS package (with an XML database, support for logarithmic units, and support for Latin-1 and UTF-8 characters) Steve Emmerson has returned to the development of a next-generation LDM.
Even though GOES-South imagery has successfully been incorporated into the half-hourly GOES-East image sectors in the Unidata-Wisconsin IDD datastream, it is believed that inclusion of the available 15-minute scans would be of use to Unidata community members, especially those in South America. The additions of GOES-South imager sectors to the Unidata-Wisconsin datastream was endorsed by the Unidata User's Committee at its May, 2007 meeting.
Proposed Unidata-Wisconsin Datastream Changes (http://www.unidata.ucar.edu/committees/polcom/2007summer/statusreports/uniwisc_prop.html) presents examples of GOES-10 imagery that are to be added to the Unidata-Wisconsin IDD datastream and discusses potential impacts to end-users. The target of adding GOES-South imagery to the Unidata-Wisconsin datastream by the end of Summer, 2007 was delayed until the GOES-South dish (located on the southeast side of the Mesa Lab) gets upgraded to be able to track GOES-10. Motorization of the dish should be completed late this fall.
NB: In order to correctly gauge real-time status of the IDD, it is important that all participating sites accurately maintain their system clocks. This is easily done through use of a Network Time Protocol daemon run on the local machine.
The cluster approach to toplevel IDD relay, first reported in the Spring 2005 status report, has been operational at the UPC since early summer 2005. The cluster, described in the June 2005 CommunitE-letter article Unidata's New IDD Cluster, routinely relays data to more than 440 downstream connections. Data input to the cluster nodes is approx. 4 GB/hr (0.97 TB/day); average data output by the cluster is approx. 246 Mbps (~2.7 TB/day); peak rates routinely exceeding 500 Mpbs (~5.4 TB/day).
Currently seven real server nodes operating in two separate locations on the UCAR Foothills Lab campus (in the UPC offices and in FL-2) and one director comprise idd.unidata.ucar.edu. A second, director will be installed in the coming months for added redundancy. In addition, the cluster is also being used for Spam Assassin processing on approximately 800,000 inbound email messages to the UPC each month.
The cluster approach to IDD relay has been adopted by NOAA/GSD (formerly FSL) and Penn State (using funds provided by the Unidata-administered Equipment Awards program). Unidata staff have fielded questions on implementing similar clusters at Unidata community sites (Texas A&M) that have expressed an interest in functioning as toplevel IDD relay nodes.