IDD and NOAAPort

Status Report: October 2014 - March 2015

Mike Schmidt, Jeff Weber, Tom Yoksas

Strategic Focus Areas

The IDD/NOAAPort group's work supports the following Unidata funding proposal focus areas:

  1. Enable widespread, efficient access to geoscience data
    A project like the IDD demonstrates how sites can employ the LDM to move data in their own environments.

  2. Develop and provide open-source tools for effective use of geoscience data
    The IDD is powered by the Unidata LDM-6 which is made freely available to all. The Unidata NOAAPort ingest package is being used by a variety of university and non-university community members. Both the LDM and NOAAPort ingest packages are being bundled by Raytheon in AWIPS-II.

  3. Provide cyberinfrastructure leadership in data discovery, access, and use
    The community-driven IDDs provide push data services to users an ever increasing community of global educators and researchers

  4. Build, support, and advocate for the diverse geoscience community
    Providing access to data in real-time is a fundamental Unidata activity.
    The IDD-Brasil, the South American peer of the North American IDD operated by the UPC, is helping to extend real-time data delivery outside of the U.S. to countries in South America and Africa. The Universidad de Costa Rica is experimenting with relaying data received in the IDD to Colombia.

Activities Since the Last Status Report

Internet Data Distribution (IDD)

  • The UPC has been evaluating ingest and relay of the 0.25 degree GFS data that became operational in NCEP on January 14, 2015. Testing has shown that peak CONDUIT data volumes would increase from about 8 GB/hr to about 21 GB/hr if all forecast hours for the 0.25 degree GFS were added to the current set of products being delivered.

    The increase in data aggregate volume that results from the addition of the 0.25 degree GFS and HRRR data from NOAA/GSD can be seen by comparing the volume on our IDD test leaf node, lead.unidata.ucar.edu with that on one of the idd.unidata.ucar.edu real server backends shown below:

    Data Volume Summary for lead.unidata.ucar.edu
    
    Maximum hourly volume  44003.174 M bytes/hour
    Average hourly volume  21199.138 M bytes/hour
    
    Average products per hour     322785 prods/hour
    
    Feed                           Average             Maximum     Products
                         (M byte/hour)            (M byte/hour)   number/hour
    CONDUIT                7756.879    [ 36.591%]    21449.892    91513.640
    FSL2                   3718.160    [ 17.539%]     7435.481     6485.200
    NGRID                  3523.378    [ 16.620%]     5434.958    24026.240
    NEXRAD2                2878.497    [ 13.578%]     3605.088    53754.120
    FNMOC                  1146.801    [  5.410%]     5302.426     3131.360
    NEXRAD3                1075.070    [  5.071%]     1262.917    79334.720
    HDS                     393.330    [  1.855%]      604.352    19163.000
    NOTHER                  244.889    [  1.155%]      735.924     1177.960
    NIMAGE                  152.517    [  0.719%]      247.598      198.840
    FNEXRAD                  78.796    [  0.372%]       88.861      104.760
    GEM                      74.915    [  0.353%]      467.891      803.200
    UNIWISC                  70.412    [  0.332%]      116.423       46.000
    IDS|DDPLUS               55.050    [  0.260%]       66.344    42427.640
    EXP                      29.312    [  0.138%]       51.651      304.600
    LIGHTNING                 1.054    [  0.005%]        1.707      312.920
    GPS                       0.079    [  0.000%]        0.405        1.000
    

    A working group composed of User Committee members and Unidata staff prepared and distributed a questionaire that is primarily aimed gauging the community's interest in getting the 0.25 degree GFS data added to CONDUIT. Responses to the questionaire will be collected for review during the March User Committee meeting.

Planned Activities

Ongoing Activities

We plan to continue the following activies:

  • Unidata continues to receive High Resolution Rapid Refresh (HRRR) grids (both 2D and 3D fields) in an LDM/IDD feed from NOAA/GSD and feed these products to a small number (6) of university sites on hrrr.unidata.ucar.edu. Since HRRR and ESTOFS data were added to the NOAAPort Satellite Broadcast Network (SBN) in late September, 2014, continuing to relay the HRRR ingested from NOAA/GSD is considered to be of lesser importance and will eventually be discontinued.

    The HRRR is being experimentally served at: http://thredds-jumbo.unidata.ucar.edu/thredds/modelsHrrr.html (.xml for machines)

  • Other data sets we are actively exploring with NOAA/GSD/ESRL are:
     

  • HRRR and ESTOFS products were added to NOAAPort in late September, 2014. The following TINs announced these additions:

    http://www.nws.noaa.gov/os/notification/tin14-28hrrr-cca.htm
    http://www.nws.noaa.gov/os/notification/tin13-43estofs_noaaport_aaa.htm

    Briefly, these additions are comprised of:

    • HRRR: 81 products, hourly F00-15 each hour. CONUS 2.5km grid184. ~44 GB/day
    • ESTOFS: 3 products, hourly F00-F180, 00, 06, 12, 18z runs. CONUS 2.5km grid, Puerto Rico 1.25 km grid. ~2 GB/day

    HRRR fields and forecasts times that are not included in the NOAAPort expansion will be evaluated as additions to the CONDUIT IDD datastream.

  • The UPC continues to relay FNMOC and CMC data model output directly to the community. FNMOC provides the COAMPS and NAVGEM model output and the CMC provides the GEM model output. Unidata has provided access to these data for the past 8 years, but on a "point-to-point" basis. GEM model output was converted from GRIB1 to GRIB2 in January. The CMC is now relaying output of there new hi-resolution (15 km) GEM model to Unidata.

NOAAPort Data Ingest

  • The NOAAPort SBN, which transitioned from DVB-S to DVB-S2 in April/May 2011, was upgraded to support much higher throughput in August, 2014. Ingestion of the broadcast as been working at the UPC since the upgade, but we routinely experienced high numbers of missed frames.

    Comparison of our ingest metrics with other sites running our software (e.g., UW/SSEC, NOAA/GSD, LSU/SRCC, and a Northrup Grumman office in Northern Virginia) strongly suggested that signal quality was a major contributing factor in the problems were being experienced. We expended considerable effort to understand the data ingest problems being experienced. Experimentation demonstrated that that use of an older version of firmware on out Novra S300N receivers (V2R7 the version recommended by the manufacturer for our hardware) would produce errors in the UDP output UDP when the S300Ns were interrogated for status information, and this effect was, in turn, a function of signal quality. This problem was verified by one of the commercial vendors of NOAAPort receipt systems who is currently working with Novra to correct this and other problems being experienced by S300N receivers. It was learned very recently that Raytheon was not aware of this problem, but they were aware of other problems related to the S300N receiver, and they too are working with Novra to get the problems corrected.


  • Unidata's NOAAPort ingest package is bundled with current versions of the LDM. The current LDM release is v6.12.6, and v6.12.7 is being readied for release.

  • Raytheon bundles a version LDM-6 with AWIPS-II and is actively using Unidata's NOAAPort ingest code at a variety of NOAA offices. Raytheon has provided the UPC code modifications and GRIB table updates needed to support new data to be added to in the NOAAPort expansion. when possible

Relevant IDD Metrics

  • Approximately 550 machines at 245 sites are running LDM-6 and reporting real time statistics to the UPC. Unidata staff routinely assist in the installation and tuning of LDM-6 at user sites as a community service.

    A number organizations/projects continue use the LDM to move substantial amounts of data that do not report statistics to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private compaines, etc.).

  • IDD toplevel relay node, idd.unidata.ucar.edu

    The cluster approach to toplevel IDD relay, has been operational at the UPC since early summer 2005.

    The cluster, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more than 700 downstream connections. Data input to the cluster nodes routinely averages up to 20 GB/hr (~0.5 TB/day); average data output from the entire cluster exceeds 1.3 Gbps (~14 TB/day); peak rates routinely exceed 2.2 Gbps (which would be ~24 TB/day if the rate was sustained).

    The following shows a snapshot by feedtype of the data being received on one real server backend node of the Unidata toplevel IDD relay, idd.unidata.ucar.edu.

    Data Volume Summary for uni19.unidata.ucar.edu
    
    Maximum hourly volume  23982.273 M bytes/hour
    Average hourly volume  13282.657 M bytes/hour
    
    Average products per hour     297389 prods/hour
    
    Feed                           Average             Maximum     Products
                         (M byte/hour)            (M byte/hour)   number/hour
    CONDUIT                3653.337    [ 27.505%]     8376.741    73213.320
    NGRID                  3523.492    [ 26.527%]     5434.958    24064.880
    NEXRAD2                2863.788    [ 21.560%]     3605.088    53421.200
    FNMOC                  1146.801    [  8.634%]     5302.426     3131.360
    NEXRAD3                1073.337    [  8.081%]     1262.917    79176.360
    HDS                     393.425    [  2.962%]      604.352    19170.920
    NOTHER                  244.889    [  1.844%]      735.924     1177.960
    NIMAGE                  151.668    [  1.142%]      247.598      197.920
    GEM                      74.915    [  0.564%]      467.891      803.200
    IDS|DDPLUS               54.982    [  0.414%]       66.344    42352.160
    UNIWISC                  39.047    [  0.294%]       90.063       19.760
    FNEXRAD                  32.550    [  0.245%]       53.054       40.480
    EXP                      29.301    [  0.221%]       51.651      304.520
    LIGHTNING                 1.047    [  0.008%]        1.707      313.720
    GPS                       0.079    [  0.001%]        0.405        1.000
    
    Currently six real server nodes operating in one location on the UCAR campus (in the UCAR co-location facility in FL-2) and two directors comprise idd.unidata.ucar.edu. The cluster approach to IDD relay has been adopted by NOAA/GSD, Penn State and Texas A&M.