IDD and NOAAPort
Status Report: October 2013 - March 2014
Mike Schmidt, Jeff Weber, Tom Yoksas
Strategic Focus Areas
The IDD/NOAAPort group's work supports the following Unidata funding proposal focus areas:
- Enable widespread, efficient access to geoscience data
A project like the IDD demonstrates how sites can employ the LDM to move data in their own environments. - Develop and provide open-source tools for effective use of
geoscience data
The IDD is powered by the Unidata LDM-6 which is made freely available to all. The Unidata NOAAPort ingest package is being used by a variety of university and non-university community members. Both the LDM and NOAAPort ingest packages are being bundled by Raytheon in AWIPS-II. - Provide cyberinfrastructure leadership in data discovery,
access, and use
The community-driven IDDs provide push data services to users an ever increasing community of global educators and researchers - Build, support, and advocate for the diverse geoscience
community
Providing access to data in real-time is a fundamental Unidata activity.
The IDD-Brasil, the South American peer of the North American IDD operated by the UPC, is helping to extend real-time data delivery outside of the U.S. to countries in South America and Africa. The Universidad de Costa Rica is experimenting with relaying data received in the IDD to Colombia.
Activities Since the Last Status Report
Internet Data Distribution (IDD)
- Unidata continues to act as a toplevel relay in NEXRAD Level II data distribution for university sites and others that were receiving data from the MAX GigaPoP that was decommissioned by the NWS. The other toplevel relay sites for Level II data are the ERC (Education and Research Consortium), IRaDS (Integrated Robust Assured Data Services), and Purdue University.
- Unidata is receiving High Resolution Rapid Refresh (HRRR) grids (both 2D and 3D fields) in an LDM/IDD feed from NOAA/GSD. These products are available currently from the Unidata-operated toplevel IDD relay, idd.unidata.ucar.edu. The challenge in making the data routinely available is its large data volume which is on the order of ~8GB for the pressure level output and ~10 GB/hour for the sigma level output. The HRRR is being experimentally served at: http://lead.unidata.ucar.edu/thredds/catalog.html (.xml for machines)
- The UPC continues to relay FNMOC and the CMC data model output
directly to the community. FNMOC provides the COAMPS and
NAVGEM model output and the CMC provides the GEM model
output. Unidata has provided access to these data for the
past 8 years, but on a "point-to-point" basis. GEM model
output was converted from GRIB1 to GRIB2 in January. The
CMC is now relaying output of there new hi-resolution
(15km) GEM model to Unidata.
NOAAPort Data Ingest
- NOAAPort ingest has been functioning well since the NWS transitioned the SBN from DVB-S to DVB-S2 in April/May 2011.
- The NOAAPort ingest package was bundled with the LDM starting in version 6.10. The current LDM releases is 6.11.6.
- Raytheon bundles a modified version LDM-6 with AWIPS-II and is actively managing NOAAPort ingest at a variety of NOAA offices using the Unidata NOAAPort ingest package. Raytheon's LDM modifications are evaluated by the UPC LDM developer and incorporated into Unidata releases when possible
Relevant IDD Metrics
- Approximately 540 machines at 230 sites are
running LDM-6 and reporting real time statistics to
Unidata. Unidata staff routinely assist in the installation
of LDM-6 at user sites as a community service.
A number organizations/projects use the LDM to move substantial amounts of data that do not report statistics to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private compaines, etc.).
- IDD toplevel relay node, idd.unidata.ucar.edu
The cluster approach to toplevel IDD relay, has been operational at the UPC since early summer 2005.
The cluster, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more than 700 downstream connections. Data input to the cluster nodes now routinely averages about 15 GB/hr (~0.36 TB/day); average data output from the entire cluster exceeds 1.1 Gbps (~13 TB/day); peak rates routinely exceed 2.2 Gbps (which would be ~24 TB/day if the rate was sustained).
The following shows a snapshot by feedtype of the data being received on one node of the Unidata toplevel IDD relay, idd.unidata.ucar.edu.
Data Volume Summary for uni16.unidata.ucar.edu Maximum hourly volume 17903.967 M bytes/hour Average hourly volume 9582.558 M bytes/hour Average products per hour 247931 prods/hour Feed Average Maximum Products (M byte/hour) (M byte/hour) number/hour CONDUIT 3109.129 [ 32.446%] 5508.893 70263.791 NEXRAD2 1624.647 [ 16.954%] 2591.167 36151.930 NGRID 1330.851 [ 13.888%] 2275.660 20318.209 FNMOC 1129.902 [ 11.791%] 6434.368 2839.651 FSL2 989.430 [ 10.325%] 1555.097 1240.767 NEXRAD3 703.909 [ 7.346%] 1069.840 61259.419 HDS 318.323 [ 3.322%] 597.553 16219.093 NIMAGE 130.804 [ 1.365%] 248.942 178.535 GEM 64.435 [ 0.672%] 415.164 694.070 FNEXRAD 56.945 [ 0.594%] 77.799 66.698 EXP 50.449 [ 0.526%] 93.573 403.535 IDS|DDPLUS 47.557 [ 0.496%] 61.330 37956.140 UNIWISC 22.538 [ 0.235%] 30.999 27.302 DIFAX 2.858 [ 0.030%] 12.712 4.349 LIGHTNING 0.441 [ 0.005%] 0.938 306.070 GPS 0.340 [ 0.004%] 7.674 1.116Currently six real server nodes operating in one location on the UCAR campus (in the UCAR co-location facility in FL-2) and two directors comprise idd.unidata.ucar.edu. The cluster approach to IDD relay has been adopted by NOAA/GSD, Penn State and Texas A&M.