IDD and NOAAPort Status Report
April 2011

Mike Schmidt
Jeff Weber
Tom Yoksas

March 25, 2011

Strategic Focus Areas

This effort supports five of the funding proposal's six focus areas:

1. Broadening participation and expanding community services
2. Advancing data services
3. Developing and deploying useful tools
5. Providing leadership in cyberinfrastructure
6. Promoting diversity by expanding opportunities

Internet Data Distribution (IDD)

• In August, 2010 the flow of NLDN data from Vaisala Corporation to UAlbany was interrupted by the cessation of the satellite broadcast of the data. David Knight of UAlbany forged new agreements with Vaisala that allow for resumption of delivery of the NLDN data to SUNY Albany and free distribution to the Unidata university community. Unidata has been assisting UAlbany by updating NLDN processing code to work with the new, socket-based delivery mechanism. Flow of the NLDN data to the community resumed on February 22, but some additional coding needs to be done before this effort can be considered to be complete.

• In February, 2010 higher resolution WSR-88D products began to be added NOAAPort SBN. The staggered activation began with the lowest 2 elevations and the addition of the remaining levels has been completed. These products were automatically included in the IDD NEXRAD3 datastream.

  NB: The NWS stopped disseminating most of the low resolution Level III products on January 12, 2011.

  The NWS is proceeding with distribution of dual polarization WSR88D Level 3 products via the NOAAPort SBN and RPCCDS. Implementation is expected to begin in a test mode has begun. This addition to the NOAAPort SBN required code modifications to the Unidata NOAAPort ingest package; coordinated deployment of the package; and modifications of existing end-user LDM pattern-action file actions. The code modifications needed were included in an update to the NOAAPort ingest package and has been adopted by all sites know to be using the package.

• NWS use of LDM-6 to collect and relay NEXRAD Level II data

  NOAA rearchitected its NEXRAD Level II data collection and dissemination. The current Level II Refresh includes collection of all Level II data at a primary site (NWSTG in Silver Springs, MD) with an active backup site (ROC in Norman, OK). All connections are being made over the NOAAnet MPLS backbone. Data collected at the active site (primary or backup) is relayed to the current set of top level IDD relay nodes (IRADS (OK), Purdue (IN), and the ERC (NC)) over Internet2 and to a small number of additional sites designated by the NWS.

  NB: Unidata sites continue to receive the Level II data in the same way that they always have: from one of the top level relay sites or from the top level relay nodes in Boulder, CO (UPC) or Vienna, VA (NSF/AGS).

  The IDD toplevel relay nodes operated by Unidata redundantly feed Level II data from all of the toplevel relay nodes in order to provide as failsafe a system as possible.

• The LDM-6 continues to be used for data collection activities in the THORPEX Interactive Grand Global Ensemble (TIGGE). The UPC continues to provide support as needed in the use of the LDM by TIGGE participants. TIGGE stress testing activities were reported in the February, 2006 CommunitE-letter article THORPEX/TIGGE and the LDM.

• The Unidata Program Center continues to relay FNMOC and the CMC data model output directly to the community. FNMOC provides the COAMPS and NOGAPS model output and the CMC provides the GEM model output. Unidata has provided access to these data for the past 8 years, but on a "point-to-point" basis.

• For previous information, please refer to the Fall 2010 User Committee status report

NOAAPort Data Ingest

• In preparation for the NOAAPort SBN conversion from DVB-S to DVB-S2 technologies and associated tripling of potential bandwidth, the NOAAPort ingest package design has been undergoing a performance review. It is currently believed that no code changes will be needed, but we will not be confident of this view until the period of dual DVB-S/DVB-S2 broadcast begins on April 17.

  NB: All sites ingesting the NOAAPort SBN feed will need to upgrade their receivers. The UPC is following the Weather Service's lead by adopting use of the Novra S-300 DVB-S2 receiver. The challanges involved in use of the new receiver are 1) availability -- the new units will not be available from Novra until mid-April at the earliest, and 2) cost -- the Novra S-300 costs approx. $649 plus shipping from Canada.

• Unidata's NOAAPort ingest package continues to provide NOAAPort SBN content to the IDD HDS, IDS|DDPLUS, NGRID, NIMAGE, and NEXRAD3 feeds. The Unidata NOAAPort ingest package is known to be used by 16 sites. The number of actual users is undoubtedly higher since SSEC makes the package available to its McIDAS-XCD users.

• Currently, there are 17 sites known to be running the Unidata NOAAPORT ingestion software. There are undoubtedly more sites using the code since UW/SSEC makes it available to SSEC-licensed sites running McIDAS-XCD (McIDAS package that decodes NOAAPort-delivered data int McIDAS compatible formats).

• Very recently we learned that Raytheon is planning on bundling our NOAAPort ingest packge and the LDM into the AWIPS-II baseline release. We learned of this while working with Raytheon on issues they have experienced when using the NOAAPort ingest package on newer workstations.

• 7 community sites are acting as IDD injection nodes for NOAAPORT data.

Relevant IDD Metrics

• Approximately 467 machines at 219 sites are running LDM-6 and reporting real time statistics. Unidata staff routinely assist in the installation of LDM-6 at user sites as a community service.

  There are several organizations/projects that use the LDM to move substantial amounts of data that do not report statistics to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private compaines, etc.).

• IDD toplevel relay node, idd.unidata.ucar.edu

  The cluster approach to toplevel IDD relay, has been operational at the UPC since early summer 2005. The cluster, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more than 650 downstream connections. Data input to the cluster nodes is now over 6.5 GB/hr (0.16 TB/day); average data output is approx. 525 Mbps (~5.7 TB/day); peak rates routinely exceed 1.15 Gbps (which would be ~12.4 TB/day if the rate was sustained).

  The following shows a snapshot by feedtype of the data being relayed by Unidata's toplevel IDD relay, idd.unidata.ucar.edu.

  Data Volume Summary for idd.unidata.ucar.edu
  
  Maximum hourly volume  12898.705 M bytes/hour
  Average hourly volume   5756.894 M bytes/hour
  
  Average products per hour     227670 prods/hour
  
  Feed                           Average             Maximum     Products
                       (M byte/hour)            (M byte/hour)   number/hour
  CONDUIT                2468.451    [ 42.878%]     7214.778    65778.822
  NEXRAD2                1312.950    [ 22.807%]     1943.615    48699.889
  NGRID                   708.782    [ 12.312%]     1523.854    15412.422
  NEXRAD3                 459.403    [  7.980%]      652.756    40539.711
  HDS                     254.359    [  4.418%]      427.890    17985.156
  FNMOC                   231.792    [  4.026%]     1521.107     2026.578
  NIMAGE                  140.420    [  2.439%]      232.029      182.489
  FNEXRAD                  68.213    [  1.185%]       83.200       72.311
  IDS|DDPLUS               41.875    [  0.727%]       51.148    35980.178
  EXP                      38.806    [  0.674%]       78.491      428.111
  UNIWISC                  23.145    [  0.402%]       32.259       26.200
  DIFAX                     3.690    [  0.064%]       12.396        5.044
  GEM                       2.562    [  0.045%]       19.542      179.644
  FSL2                      1.788    [  0.031%]        1.976       22.289
  LIGHTNING                 0.586    [  0.010%]        1.593      327.067
  WMO                       0.071    [  0.001%]        0.074        3.889
  

  Currently eight real server nodes operating in two locations on the UCAR campus (in the UPC offices and in FL-2) and one director comprise idd.unidata.ucar.edu. A second director was added to the cluster to increase redundancy. The cluster approach to IDD relay has been adopted by NOAA/GSD and Penn State (using funds provided by the Unidata-administered Equipment Awards program). The cluster operated by Penn State assumed toplevel relay responsibilities for CONDUIT data in early March, 2010.

• Real time LDM statistics can be found in Real Time IDD Statistics web page (http://www.unidata.ucar.edu/software/idd/rtstats). The topology and data flow information reflects changes in feed requests within a few minutes of changes. rtstats continues to be most useful in diagnosing reception problems at sites, especially when sites have implemented restrictions on packets volume in and out of their campus without regard for their user needs.

• Internet2 (I2) bandwidth usage by the LDM protocol routinely exceeds 30 TB/week (peak usage was approx. 33 TB/week the week of September 9, 2009).

  Weekly I2 usage statistics can be seen at Internet NetFlow: Weekly Reports (search for 'UNIDATA LDM' usage statistics in the Advanced Applications portion of Table 7).

  NB: We are unsure why the Internet2 statistics are not being routinely updated of late!

• IDD Inquiry Metrics (as separate items in the general support metrics list)

---

by Tom Yoksas and Jeff Weber.