IDD and NOAAPort Status Report
October 2010

Mike Schmidt
Jeff Weber
Tom Yoksas

October 1, 2010

Strategic Focus Areas

This effort supports five of the funding proposal's six focus areas:

1. Broadening participation and expanding community services
2. Advancing data services
3. Developing and deploying useful tools
5. Providing leadership in cyberinfrastructure
6. Promoting diversity by expanding opportunities

Internet Data Distribution (IDD)

• In February, higher resolution WSR-88D products began to be added NOAAPort SBN. The staggered activation began with the lowest 2 elevations and the addition of the remaining levels has largely been completed. These products were automatically included in the IDD NEXRAD3 datastream.

  NB: The NWS plans to stop diseminating the low resolution products on December 1, 2010.

  The NWS is proceeding with its plan to distribute dual polarization WSR88D Level 3 products via the NOAAPort SBN and RPCCDS. Implementation is expected to begin in a test mode in the November 2010 timeframe. This addition to the NOAAPort SBN required code modifications to the Unidata NOAAPort ingest package; coordinated deployment of the package; and modifications of existing end-user LDM pattern-action file actions. The code modifications needed have already included in an update to the ingest package and has been adopted by all sites know to be using the package.

• NWS use of LDM-6 to collect and relay NEXRAD Level II data

  NOAA rearchitected the way that NEXRAD Level II data is collected and disseminated. The current Level II Refresh includes collection of all Level II data at a primary site (NWSTG in Silver Springs, MD) with an active backup site (ROC in Norman, OK). All connections are being made over the NOAAnet MPLS backbone. Data collected at the active site (primary or backup) is be relayed to the current set of top level IDD relay nodes (IRADS (OK), Purdue (IN), and the ERC (NC)) over Internet2 and to a small number of additional sites designated by the NWS.

  NB: Unidata sites continue to receive the Level II data in the same way that they always have: from one of the top level relay sites or from the top level relay nodes in Boulder, CO (UPC) or Vienna, VA (NSF/AGS).

• In August, the flow of NLDN data from Vaisala Corporation to SUNY Albany was interrupted by the cessation of the satellite broadcast of the data. David Knight of SUNY Albany has recently forged new agreements with Vaisala that allow for resumption of delivery of the NLDN data to SUNY Albany and free distribution to the Unidata university community. Unidata will be assisting SUNY Albany by updating NLDN processing code to work with the new, socket-based delivery mechanism in the coming weeks. It is expected that the flow of NLDN data to the community will resume by the end of October, 2010.

• The LDM-6 continues to be used for data collection activities in the THORPEX Interactive Grand Global Ensemble (TIGGE). The UPC continues to provide support as needed in the use of the LDM by TIGGE participants. TIGGE stress testing activities were reported in the February, 2006 CommunitE-letter article THORPEX/TIGGE and the LDM.

• The Unidata Program Center continues to relay FNMOC and the CMC data model output directly to the community. FNMOC provides the COAMPS and NOGAPS model output and the CMC provides the GEM model output. Unidata has provided access to these data for the past 8 years, but on a "point-to-point" basis.
• For previous information, please refer to the Spring 2010 User Committee status report

NOAAPort Data Ingest

• Unidata's NOAAPort ingest package continues to provide NOAAPort content in the IDD's HDS, IDS|DDPLUS, NGRID, NIMAGE, and NEXRAD3 feeds. An update to the package to accommodate new "high resolution" and new dual-polarization products Level III products was made in spring, 2010.

  The Unidata NOAAPort ingest package is known to be used by 16 sites. The number of actual users is undoubtedly higher since SSEC makes the package available to its McIDAS-XCD users.

• Currently, there are 16 sites known to be running the Unidata NOAAPORT ingestion software including Raytheon which is doing so in support of its use of Unidata GEMPAK which is informing their development of AWIPS-2. There are undoubtedly more sites using the code since UW/SSEC makes it available to SSEC-licensed sites running McIDAS-XCD (McIDAS package that decodes NOAAPort-delivered data int McIDAS compatible formats).

• 7 community sites are acting as IDD injection nodes for NOAAPORT data.

Relevant IDD Metrics

• Approximately 500 machines at 250 sites are running LDM-6 and reporting real time statistics. Unidata staff routinely assist in the installation of LDM-6 at user sites as a community service.

  There are several organizations/projects that use the LDM to move substantial amounts of data that do not report statistics to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private compaines, etc.).

• IDD toplevel relay node, idd.unidata.ucar.edu

  The cluster approach to toplevel IDD relay, has been operational at the UPC since early summer 2005. The cluster, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more than 650 downstream connections. Data input to the cluster nodes is now over 6.5 GB/hr (0.16 TB/day); average data output is approx. 525 Mbps (~5.7 TB/day); peak rates routinely exceed 1.15 Gbps (which would be ~12.4 TB/day if the rate was sustained).

  The following lists the breakdown by feedtype of the data being relayed by Unidata's toplevel IDD relay, idd.unidata.ucar.edu.

  Data Volume Summary for idd.unidata.ucar.edu
  
  Maximum hourly volume  14155.751 M bytes/hour
  Average hourly volume   6973.663 M bytes/hour
  
  Average products per hour     249825 prods/hour
  
  Feed                           Average             Maximum     Products
                       (M byte/hour)            (M byte/hour)   number/hour
  CONDUIT                2471.870    [ 35.446%]     7245.773    64061.738
  NEXRAD2                2175.907    [ 31.202%]     2917.550    62035.357
  NEXRAD3                 820.510    [ 11.766%]     1048.914    51436.000
  NGRID                   690.929    [  9.908%]     1406.051    15980.810
  HDS                     253.852    [  3.640%]      393.416    17838.524
  FNMOC                   226.990    [  3.255%]     1312.805     1960.190
  NIMAGE                  134.981    [  1.936%]      241.821      168.571
  FNEXRAD                  87.758    [  1.258%]      109.766       72.405
  IDS|DDPLUS               40.365    [  0.579%]       55.577    35461.214
  EXP                      37.829    [  0.542%]       65.342      330.786
  UNIWISC                  22.599    [  0.324%]       29.568       26.786
  DIFAX                     3.305    [  0.047%]       12.090        4.833
  GEM                       2.807    [  0.040%]       26.388      192.476
  LIGHTNING                 2.129    [  0.031%]        4.094      232.762
  FSL2                      1.830    [  0.026%]        2.076       22.357
  

  Currently seven real server nodes operating in two separate locations on the UCAR campus (in the UPC offices and in FL-2) and one director comprise idd.unidata.ucar.edu. A second director has been added to the cluster to increase redundancy. The cluster approach to IDD relay has been adopted by NOAA/GSD and Penn State (using funds provided by the Unidata-administered Equipment Awards program). The cluster operated by Penn State assumed toplevel relay responsibilities for CONDUIT data in early March, 2010.

• Real time LDM statistics can be found in Real Time IDD Statistics web page (http://www.unidata.ucar.edu/projects/idd/rtstats). The topology and data flow information reflects changes in feed requests within a few minutes of changes. rtstats continues to be most useful in diagnosing reception problems at sites, especially when sites have implemented restrictions on packets volume in and out of their campus without regard for their user needs.

• Internet2 (I2) bandwidth usage by the LDM protocol routinely exceeds 30 TB/week (peak usage was approx. 33 TB/week the week of September 9, 2009).

  Weekly I2 usage statistics can be seen at Internet NetFlow: Weekly Reports (search for 'UNIDATA LDM' usage statistics in the Advanced Applications portion of Table 7).

• IDD Inquiry Metrics (as separate items in the general support metrics list)

---

by Tom Yoksas and Jeff Weber.