IDD and NOAAPort
Status Report: October 2011 - March 2012
Mike Schmidt, Jeff Weber, Tom Yoksas
Strategic Focus Areas
The IDD/NOAAPort group's work supports the following Unidata funding proposal focus areas:
- Broadening participation and expanding community
services
Providing access to data in real-time is a fundamental Unidata activity. - Advancing data services
The community-driven IDDs provide push data services to users an ever increasing community of global educators and researchers - Developing and deploying useful tools
The IDD is powered by the Unidata LDM-6 which is made freely available to all. Our NOAAPort ingest package is being used by a variety of university and non-university community members. Both the LDM and NOAAPort ingest packages will be bundled by Raytheon with AWIPS-II. - Providing leadership in cyberinfrastructure
A project like the IDD demonstrates how sites can employ the LDM to move data in their own environments. - Promoting diversity by expanding opportunities
The IDD-Brasil, the South American peer of the North American IDD operated by the UPC, is helping to extend real-time data delivery outside of the U.S. to countries in South America and Africa.
IDD Activities Since the Last Status Report
- NEXRAD Level II datastream includes an increasing amount of dual polarization data as NEXRADs become upgraded. This has significantly increased the data volume in the IDD NEXRAD Level II (NEXRAD2) and Level III (NEXRAD3) data feeds.
- Fire Weather products from parallel NAM model runs were added to CONDUIT on September 20. The additional products increased the average CONDUIT volume by approx. 800 MB/hour. The addition did not require increased LDM queue sizes on WOC or Unidata community toplevel relay machines because the peaks in volume for the fire weather products did not coincide with the existing volume peak that is dominated by global GFS model output.
- Rapid Refresh (RAP) grids are scheduled to replace RUC2 grids in the NOAAPort SBN and in CONDUIT on or about April 3 (the transition date has moved over time)
- Unidata has assumed a toplevel role in NEXRAD Level II data distribution. The other toplevel relay sites for Level II data are the ERC (Education and Research Consortium), IRaDS (Integrated Robust Assured Data Services), and Purdue University
- Unidata is now receiving High Resolution Rapid Refresh (HRRR) grids in an LDM/IDD feed from NOAA/GSD. The intention is to evaluate making these products available in a new (EXP feedtype) datastream from the Unidata-operated toplevel IDD relay node, idd.unidata.ucar.edu. The challenge in doing making the data routinely available is its large data volume which is thought to be on the order of 8-10 GB/hour.
- The UPC continues to relay FNMOC and the CMC data model output directly to the community. FNMOC provides the COAMPS and NOGAPS model output and the CMC provides the GEM model output. Unidata has provided access to these data for the past 8 years, but on a "point-to-point" basis. GEM model output was converted from GRIB1 to GRIB2 in January.
NOAAPort Data Ingest
- NOAAPort ingest has been functioning near-flawlessly since the NWS transitioned the SBN from DVB-S to DVB-S2 in April/May 2011. The switchover was smooth enough that end-users should have never noticed the change
- The NOAAPort ingest package has been bundled with the LDM as of LDM version 6.10
- Raytheon is bundling LDM-6 with AWIPS-II and is actively managing NOAAPort ingest at a variety of NOAA offices
Relevant IDD Metrics
- Approximately 505 machines at 230 sites are running LDM-6 and reporting real time statistics. Unidata staff routinely assist in the installation of LDM-6 at user sites as a community service.
A number organizations/projects use the LDM to move substantial amounts of data that do not report statistics to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private compaines, etc.). - IDD toplevel relay node, idd.unidata.ucar.edu
The cluster approach to toplevel IDD relay, has been operational at the UPC since early summer 2005. The cluster, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more than 650 downstream connections. Data input to the cluster nodes is now over 6.5 GB/hr (0.16 TB/day); average data output is approx. 855 Mbps (~9.3 TB/day); peak rates routinely exceed 1.75 Gbps (which would be ~18.8 TB/day if the rate was sustained).
The following shows a snapshot by feedtype of the data being relayed by Unidata's toplevel IDD relay, idd.unidata.ucar.edu.Data Volume Summary for idd.unidata.ucar.edu Maximum hourly volume 14281.280 M bytes/hour Average hourly volume 9055.233 M bytes/hour Average products per hour 263654 prods/hour Feed Average Maximum Products (M byte/hour) (M byte/hour) number/hour CONDUIT 3253.905 [ 35.934%] 6045.153 70664.545 NEXRAD2 3006.657 [ 33.204%] 4078.456 58341.068 NEXRAD3 1002.702 [ 11.073%] 1243.256 55681.727 NGRID 881.130 [ 9.731%] 1646.430 17609.864 HDS 320.313 [ 3.537%] 517.932 19305.068 FNMOC 262.610 [ 2.900%] 1590.909 2325.455 NIMAGE 141.902 [ 1.567%] 255.932 189.841 FNEXRAD 77.223 [ 0.853%] 94.044 71.477 IDS|DDPLUS 46.631 [ 0.515%] 56.780 38488.341 EXP 27.351 [ 0.302%] 60.356 343.159 UNIWISC 24.193 [ 0.267%] 30.273 26.682 DIFAX 3.553 [ 0.039%] 13.163 4.886 LIGHTNING 3.518 [ 0.039%] 6.834 346.136 FSL2 1.770 [ 0.020%] 1.918 22.591 GEM 1.665 [ 0.018%] 22.355 231.705 GPS 0.109 [ 0.001%] 1.263 1.045Currently six real server nodes operating in one location on the UCAR campus (in the UCAR co-location facility in FL-2) and two directors comprise idd.unidata.ucar.edu. The cluster approach to IDD relay has been adopted by NOAA/GSD and Penn State (using funds provided by the Unidata-administered Equipment Awards program). The cluster operated by Penn State assumed toplevel relay responsibilities for CONDUIT data in early March, 2010.