IDD and NOAAPort

Status Report: May - September 2011

Mike Schmidt, Jeff Weber, Tom Yoksas

Strategic Focus Areas

The IDD/NOAAPort group's work supports the following Unidata funding proposal focus areas:

  1. Broadening participation and expanding community services
    Providing access to data in real-time is a fundamental Unidata activity.
  2. Advancing data services
    The community-driven IDDs provide push data services to users an ever increasing community of global educators and researchers
  3. Developing and deploying useful tools
    The IDD is powered by the Unidata LDM-6 which is made freely available to all. Our NOAAPort ingest package is being used by a variety of university and non-university community members. Both the LDM and NOAAPort ingest packages will be bundled by Raytheon with AWIPS-II.
  4. Providing leadership in cyberinfrastructure
    A project like the IDD demonstrates how sites can employ the LDM to move data in their own environments.
  5. Promoting diversity by expanding opportunities
    The IDD-Brasil, the South American peer of the North American IDD operated by the UPC, is helping to extend real-time data delivery outside of the U.S. to countries in South America and Africa.

IDD Activities Since the Last Status Report

  • NEXRAD Level II datastream will include an increasing amount of dual polarization data as NEXRADs become upgraded. This will significantly increase the data volume in the NEXRAD IDD feed.
  • Fire Weather products from parallel NAM model runs were added to CONDUIT on September 20. The additional products increased the average CONDUIT volume by approx. 800 MB/hour. The addition did not, however, require increased LDM queue sizes on WOC or Unidata community toplevel relay machines because the peaks in volume for the fire weather products did not coincide with the existing volume peak that is dominated by global GFS model output.
  • Unidata is assuming a toplevel role in NEXRAD Level II data distribution. The other toplevel relay sites for Level II data are the ERC (Education and Research Consortium), IRaDS (Integrated Robust Assured Data Services), and Purdue University.
  • The UPC continues to relay FNMOC and the CMC data model output directly to the community. FNMOC provides the COAMPS and NOGAPS model output and the CMC provides the GEM model output. Unidata has provided access to these data for the past 8 years, but on a "point-to-point" basis. CMC GEM model output is being broadcast in BOTH GRIB1 and GRIB2 currently, with a cut over to GRIB2 in January

NOAAPort Data Ingest

  • The NWS transitioned the NOAAPort SBN from DVB-S to DVB-S2 in April/May. The switchover was smooth enough that end-users should have never noticed the change.

  • The NOAAPort ingest package has been bundled with the LDM as of LDM version 6.10.

Relevant IDD Metrics

  • Approximately 480 machines at 215 sites are running LDM-6 and reporting real time statistics. Unidata staff routinely assist in the installation of LDM-6 at user sites as a community service.

     

    A number organizations/projects use the LDM to move substantial amounts of data that do not report statistics to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private compaines, etc.).

    •  

  • IDD toplevel relay node, idd.unidata.ucar.edu

     

    The cluster approach to toplevel IDD relay, has been operational at the UPC since early summer 2005. The cluster, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more than 650 downstream connections. Data input to the cluster nodes is now over 6.5 GB/hr (0.16 TB/day); average data output is approx. 525 Mbps (~5.7 TB/day); peak rates routinely exceed 1.15 Gbps (which would be ~12.4 TB/day if the rate was sustained).

     

    The following shows a snapshot by feedtype of the data being relayed by Unidata's toplevel IDD relay, idd.unidata.ucar.edu. 
    Data Volume Summary for idd.unidata.ucar.edu

Maximum hourly volume  14126.159 M bytes/hour
Average hourly volume   7764.707 M bytes/hour

Average products per hour     241135 prods/hour

Feed                           Average             Maximum     Products
                     (M byte/hour)            (M byte/hour)   number/hour
CONDUIT                3430.226    [ 44.177%]     6683.558    70604.312
NEXRAD2                1940.325    [ 24.989%]     2874.110    50809.146
NGRID                   759.792    [  9.785%]     1493.868    16385.583
NEXRAD3                 724.156    [  9.326%]     1003.710    44242.062
FNMOC                   300.142    [  3.865%]     1795.255     2590.667
HDS                     263.983    [  3.400%]      421.706    18046.604
NIMAGE                  140.712    [  1.812%]      237.801      174.250
FNEXRAD                  88.378    [  1.138%]      102.509       73.083
IDS|DDPLUS               41.729    [  0.537%]       51.005    37093.917
EXP                      39.987    [  0.515%]       75.847      370.625
UNIWISC                  23.854    [  0.307%]       30.150       26.688
GEM                       3.628    [  0.047%]       39.015      338.333
DIFAX                     3.597    [  0.046%]       12.513        4.958
LIGHTNING                 2.597    [  0.033%]        4.886      351.792
FSL2                      1.600    [  0.021%]        1.720       22.521
    Currently siz real server nodes operating in two locations on the UCAR campus (in the UPC offices and in FL-2) and one director comprise idd.unidata.ucar.edu. A second director is scheduled to be added to the cluster to increase redundancy. The cluster approach to IDD relay has been adopted by NOAA/GSD and Penn State (using funds provided by the Unidata-administered Equipment Awards program). The cluster operated by Penn State assumed toplevel relay responsibilities for CONDUIT data in early March, 2010.