IDD and NOAAPort

Status Report: October 2012 - April 2013

Mike Schmidt, Jeff Weber, Tom Yoksas

Strategic Focus Areas

The IDD/NOAAPort group's work supports the following Unidata funding proposal focus areas:

  1. Enable widespread, efficient access to geoscience data
    A project like the IDD demonstrates how sites can employ the LDM to move data in their own environments.
  2. Develop and provide open-source tools for effective use of geoscience data
    The IDD is powered by the Unidata LDM-6 which is made freely available to all. The Unidata NOAAPort ingest package is being used by a variety of university and non-university community members. Both the LDM and NOAAPort ingest packages will be bundled by Raytheon with AWIPS-II.
  3. Provide cyberinfrastructure leadership in data discovery, access, and use
    The community-driven IDDs provide push data services to users an ever increasing community of global educators and researchers
  4. Build, support, and advocate for the diverse geoscience community
    Providing access to data in real-time is a fundamental Unidata activity.
    The IDD-Brasil, the South American peer of the North American IDD operated by the UPC, is helping to extend real-time data delivery outside of the U.S. to countries in South America and Africa. The Universidad de Costa Rica is experimenting with relaying data received in the IDD to Colombia.

IDD Activities Since the Last Status Report

  • Unidata continues to act as a toplevel relay in NEXRAD Level II data distribution for university sites and others that were receiving data from the MAX GigaPoP that was decommissioned by the NWS. The other toplevel relay sites for Level II data are the ERC (Education and Research Consortium), IRaDS (Integrated Robust Assured Data Services), and Purdue University.
  • Unidata is receiving High Resolution Rapid Refresh (HRRR) grids (both 2D and 3D fields) in an LDM/IDD feed from NOAA/GSD. The intention is to make these products available very soon from the Unidata-operated toplevel IDD relay, idd.unidata.ucar.edu. The challenge in making the data routinely available is its large data volume which is on the order of ~8GB for the pressure level output and ~10 GB/hour for the sigma level output. We have a test ingest of this data and soon the FIM output going to our LDM on gale.unidata.ucar.edu
  • The UPC continues to relay FNMOC and the CMC data model output directly to the community. FNMOC provides the COAMPS and NAVGEM model output and the CMC provides the GEM model output. Unidata has provided access to these data for the past 8 years, but on a "point-to-point" basis. GEM model output was converted from GRIB1 to GRIB2 in January. The CMC is now relaying output of there new hi-resolution (15km) GEM model to Unidata. The 60 km output is going away permanently once the CMC is confident with the 15 km output.

NOAAPort Data Ingest

  • NOAAPort ingest has been functioning near-flawlessly since the NWS transitioned the SBN from DVB-S to DVB-S2 in April/May 2011. The switchover was smooth enough that end-users should have never noticed the change.
  • The NOAAPort ingest package has been bundled with the LDM as of LDM version 6.10. The current LDM releases is 6.11.4.
  • Raytheon bundles a modified version LDM-6 with AWIPS-II and is actively managing NOAAPort ingest at a variety of NOAA offices using the Unidata NOAAPort ingest package. Raytheon's LDM modifications are evaluated by the UPC LDM developer and incorporated into Unidata releases when possible

Relevant IDD Metrics

  • Approximately 520 machines at 230 sites are running LDM-6 and reporting real time statistics to Unidata. Unidata staff routinely assist in the installation of LDM-6 at user sites as a community service.

     

    A number organizations/projects use the LDM to move substantial amounts of data that do not report statistics to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private compaines, etc.).
  •  

  • IDD toplevel relay node, idd.unidata.ucar.edu

     

    The cluster approach to toplevel IDD relay, has been operational at the UPC since early summer 2005.

    The cluster, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more than 680 downstream connections. Data input to the cluster nodes now routinely averages over 12 GB/hr (~0.3 TB/day); average data output from the entire cluster is approx. 1.1 Gbps (~13 TB/day); peak rates routinely exceed 2.2 Gbps (which would be ~24 TB/day if the rate was sustained).

     

    The following shows a snapshot by feedtype of the data being received one node of the Unidata toplevel IDD relay, idd.unidata.ucar.edu.
    Data Volume Summary for uni18.unidata.ucar.edu
    
    Maximum hourly volume  19828.723 M bytes/hour
    Average hourly volume  11667.719 M bytes/hour
    
    Average products per hour     281928 prods/hour
    
    Feed                           Average             Maximum     Products
                         (M byte/hour)            (M byte/hour)   number/hour
    CONDUIT                3328.114    [ 28.524%]     5269.075    73736.773
    NEXRAD2                3077.665    [ 26.378%]     3714.055    48861.636
    NEXRAD3                1256.441    [ 10.769%]     1478.044    76113.500
    NGRID                  1118.158    [  9.583%]     2086.498    20125.886
    FNMOC                  1082.579    [  9.278%]     6351.885     3116.750
    FSL2                   1057.268    [  9.061%]     2539.983     1254.886
    HDS                     329.436    [  2.823%]      507.528    18251.182
    NIMAGE                  148.282    [  1.271%]      253.169      189.318
    GEM                      76.773    [  0.658%]      466.329      977.568
    FNEXRAD                  73.161    [  0.627%]       91.272       71.818
    IDS|DDPLUS               48.819    [  0.418%]       61.076    38494.500
    EXP                      38.953    [  0.334%]       66.279      351.341
    UNIWISC                  25.664    [  0.220%]       33.948       29.568
    DIFAX                     3.593    [  0.031%]       12.966        4.977
    LIGHTNING                 2.687    [  0.023%]        6.380      347.636
    GPS                       0.126    [  0.001%]        0.992        1.068
    
    Currently six real server nodes operating in one location on the UCAR campus (in the UCAR co-location facility in FL-2) and two directors comprise idd.unidata.ucar.edu. The cluster approach to IDD relay has been adopted by NOAA/GSD, Penn State and Texas A&M.