IDD and NOAAPort
Status Report: March 2012 - September 2012
Mike Schmidt, Jeff Weber, Tom Yoksas
Strategic Focus Areas
The IDD/NOAAPort group's work supports the following Unidata funding proposal focus areas:
- Enable widespread, efficient access to geoscience data
A project like the IDD demonstrates how sites can employ the LDM to move data in their own environments. - Develop and provide open-source tools for effective use of
geoscience data
The IDD is powered by the Unidata LDM-6 which is made freely available to all. The Unidata NOAAPort ingest package is being used by a variety of university and non-university community members. Both the LDM and NOAAPort ingest packages will be bundled by Raytheon with AWIPS-II. - Provide cyberinfrastructure leadership in data discovery,
access, and use
The community-driven IDDs provide push data services to users an ever increasing community of global educators and researchers - Build, support, and advocate for the diverse geoscience
community
Providing access to data in real-time is a fundamental Unidata activity.
The IDD-Brasil, the South American peer of the North American IDD operated by the UPC, is helping to extend real-time data delivery outside of the U.S. to countries in South America and Africa. The Universidad de Costa Rica is experimenting with relaying data received in the IDD to Colombia.
IDD Activities Since the Last Status Report
- Rapid Refresh (RAP) grids replaced RUC2 grids in the NOAAPort SBN and in CONDUIT in April
- NOAA implemented changes to the CONDUIT datastream on August 21. Briefly,
the changes removed the Eta and RSM members (which are no longer available) and
added the new NMMB members. The changes made resulted in CONDUIT users
continuing to get all of the individual members available.
While working with NOAA on the CONDUIT upgrade, it was learned that the great majority of GRIB2 messages being transmitted were originally created as GRIB1 messages and then converted to GRIB2 format. Over the years, we (Unidata and its CONDUIT user community) has observed that a number of the GRIB2 products had PDS values that do not accurately represent the actual field in the GRIB2 message. This problem is now understood to have been introduced by the routine being to convert GRIB1 messages to GRIB2. NOAA has agreed to work to fix errant GRIB2 fields when we can provide documentation for the problems we are encountering. We view this as being a great opportunity to fix some things that have plagued a variety of Unidata projects for years.
- Unidata is acting as a toplevel relay in NEXRAD Level II data distribution for university sites and others that were receiving data from the MAX GigaPoP that was decommissioned by the NWS. The other toplevel relay sites for Level II data are the ERC (Education and Research Consortium), IRaDS (Integrated Robust Assured Data Services), and Purdue University.
- Unidata is now receiving High Resolution Rapid Refresh (HRRR) grids (surface only) in an LDM/IDD feed from NOAA/GSD. The intention is to evaluate making these products available in a new (EXP feedtype) datastream from the Unidata-operated toplevel IDD relay node, idd.unidata.ucar.edu. The challenge in making the data routinely available is its large data volume which is on the order of ~8GB for the pressure level output and ~10 GB/hour for the sigma level output.
- The UPC continues to relay FNMOC and the CMC data model output directly to the community. FNMOC provides the COAMPS and NOGAPS model output and the CMC provides the GEM model output. Unidata has provided access to these data for the past 8 years, but on a "point-to-point" basis. GEM model output was converted from GRIB1 to GRIB2 in January. The CMC has offered to relay output of there new hi-resolution (15km) GEM model to Unidata by the end of September. The 60 km output will go away permanently by end of year, and likely off Unidata systems once we are getting the 15 km output.
NOAAPort Data Ingest
- NOAAPort ingest has been functioning near-flawlessly since the NWS transitioned the SBN from DVB-S to DVB-S2 in April/May 2011. The switchover was smooth enough that end-users should have never noticed the change.
- The NOAAPort ingest package has been bundled with the LDM as of LDM version 6.10+.
- Raytheon is bundling LDM-6 with AWIPS-II and is actively managing NOAAPort ingest at a variety of NOAA offices using the Unidata NOAAPort ingest package.
Relevant IDD Metrics
- Approximately 510 machines at 220 sites are running LDM-6 and
reporting real time statistics to Unidata. Unidata staff routinely assist in the
installation of LDM-6 at user sites as a community service.
A number organizations/projects use the LDM to move substantial amounts of data that do not report statistics to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private compaines, etc.). - IDD toplevel relay node, idd.unidata.ucar.edu
The cluster approach to toplevel IDD relay, has been operational at the UPC since early summer 2005.The cluster, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more than 650 downstream connections. Data input to the cluster nodes now averages over 12.5 GB/hr (0.3 TB/day); average data output is approx. 1.23 Gbps (~13.3 TB/day); peak rates routinely exceed 2.27 Gbps (which would be ~24.5 TB/day if the rate was sustained).
The following shows a snapshot by feedtype of the data being relayed by Unidata's toplevel IDD relay, idd.unidata.ucar.edu.Data Volume Summary for idd.unidata.ucar.edu Maximum hourly volume 18386.051 M bytes/hour Average hourly volume 12791.791 M bytes/hour Average products per hour 300999 prods/hour Feed Average Maximum Products (M byte/hour) (M byte/hour) number/hour NEXRAD2 5143.794 [ 40.212%] 6945.801 69965.250 CONDUIT 3225.701 [ 25.217%] 5019.001 70255.200 NEXRAD3 1839.069 [ 14.377%] 2357.928 81251.050 FSL2 877.396 [ 6.859%] 1643.279 1027.550 NGRID 743.965 [ 5.816%] 1276.658 16729.900 HDS 338.387 [ 2.645%] 531.282 18853.000 FNMOC 275.755 [ 2.156%] 1095.363 2960.300 NIMAGE 137.472 [ 1.075%] 254.370 179.850 FNEXRAD 86.099 [ 0.673%] 108.016 69.700 IDS|DDPLUS 46.155 [ 0.361%] 52.688 38724.600 EXP 45.278 [ 0.354%] 87.771 415.650 UNIWISC 22.536 [ 0.176%] 29.113 20.200 LIGHTNING 4.699 [ 0.037%] 9.468 337.100 DIFAX 3.829 [ 0.030%] 11.884 5.150 GEM 1.535 [ 0.012%] 8.447 203.900 GPS 0.119 [ 0.001%] 0.955 1.050Currently five real server nodes operating in one location on the UCAR campus (in the UCAR co-location facility in FL-2) and two directors comprise idd.unidata.ucar.edu. The cluster approach to IDD relay has been adopted by NOAA/GSD, Penn State and Texas A&M (using funds provided by the Unidata-administered Equipment Awards program).