Background

To address the compelling need for real time access to the WSR-88D Level II digital base data, Unidata joined forces in 1998 with the Center for Analysis and Prediction of Storms (CAPS) at the University of Oklahoma, the University of Washington, the National Severe Storms Laboratory (NSSL), and the WSR-88D Operational Support Facility (now the Radar Operations Center - ROC). The data, which are in native radar coordinates at full resolution, are useful for a variety of research and commercial applications, including severe storms research and data assimilation in numerical weather prediction.

Under the auspices of the Collaborative Radar Acquisition Field Test (CRAFT), the partners developed the principle goals of testing real-time compression using NSSL’s Bzip2; Internet-based data transmission using the Unidata Local Data Manager (LDM) technology; and delivery of the data to the National Climatic Data Center (NCDC) more efficiently for transition from 8MM tapes to Web accessible archive availability within minutes rather than days. Both the USWRP Science Steering Committee and the Unidata Policy Committee approved the goals. The initial test bed of six radars located in and around Oklahoma began delivery of real-time Level II data in early 1999 has evolved to the distribution of data from 62 radars today.

By every measure, the CRAFT project has been a remarkably successful example of close partnership with a growing number of stakeholders from academia, government, and the private sector. With funding from the National Science Foundation, the CRAFT Level II data recently were made available to the Unidata community, and 17 universities are now receiving the data. In addition, Level II data are also being distributed to eight private sector firms for use in commercial applications. Not only has the project demonstrated the high-degree of reliability, flexibility, and scalability of the LDM technology, it has also demonstrated the value of real-time access to radar base data. The partners have agreed that the ultimate success would be a technology transfer from CRAFT to the National Weather Service (NWS).

National Weather Service Data Distribution

Affirmation of the effectiveness of the LDM data-relay technology came when the NWS announced in February 2003 that it plans to use the LDM to deliver real-time Level II data from 123 NWS radars in the continental U.S. to the National Centers for Environmental Prediction (NCEP) for numerical weather prediction use and the data will also be made available to all interested academic, private, and government customers by the end of 2003. In 2004 the NWS will add nine more DOD radars to the Level II network and transition to the full operational capability and distribution architecture. The NWS made arrangements with the Internet2 consortium to use the Internet 2/Abilene network for moving the data.

Since the NWS announcement, the NWS staff and the LDM developers at the Unidata Program Center (UPC), in consultation with Internet2/Abilene, have worked closely to design the architecture for the new distribution system. Unidata was also invited to develop a plan for distributing the Level II data to education and research communities, and the plan was presented by the UPC during the Public Meeting to discuss the NWS plans to collect, distribute, and archive WSR-88D Level II Data, held in Silver Spring, MD on June 18, 2003 (meeting summary available at: http://www.roc.noaa.gov/NWS_Level_2/ResultsFrom061803Mtg/).

The Private Sector Issues

The CRAFT project successfully involved private sector customers almost since its inception, with eight customers subscribed to receive the Level II data through OU/CAPS. Despite their competitive business interests, the private sector stakeholders had managed to work harmoniously and in partnership with the other stakeholders in gaining access to the Level II data from the CRAFT project. However, extending the CRAFT model to the proposed NWS Level II data distribution system is not straightforward for the following reasons:

1. Many private sector customers have a more stringent requirement for data availability than the NWS. For example, the private sector requirements call for a maximum of 10 second latency and near 100% reliability in the reception of data. In contrast, the NWS requirement calls for a maximum of 60 second latencies and 95% reliability.
2. The private sector customers need 7x24 operational support.
3. There is a significant disparity between the private and the academic sector in technology, especially in regard to access to advanced networking infrastructure. For example, the high-speed Internet2/Abilene network is primarily meant for promoting research and education, and, therefore, is not intended to carry commercial or proprietary traffic. As a result, the private sector customers cannot directly use the Internet2/Abilene network. This technology gap makes it even harder to guarantee the stringent, 10-second maximum latency that the private sector requires.
4. Because of their competitive business interests, the private sector stakeholders have had difficulty speaking with one voice to the NWS in proposing a parallel distribution plan to the one planned by Unidata for the academic community.
5. The NWS maintains the position that the proposed architectural specifications and resource provisions will be driven largely by their internal requirements for Level II data, which do not necessarily meet the needs of the academic or private sector communities.

Considerations related to Private Sector Issues

Discussions regarding the needs of the private sector have included several concepts:

• consortia to manage and operate its own distribution system
• university established not-for-profit consortium where private sector customers would join and pay membership and service fees to the consortium to provide service to the private customers
• informal “quid-pro-quo” between a university and private sector company might also be used for data distribution
• NWS service much like the current Family of Services (FOS) to provide external users access to the data via dedicated telecommunication access lines, and the FOS customers, who are mostly in the private sector, obtain data and/or products from NWS for a one-time connection charge and an annual user fee that is based on the cost of equipment and operation of the service. The service would be provided with 7x24 support. A caveat is that with the advent of NOAAPort, the customer base for FOS has shrunk considerably, so the NWS is examining ways to eliminate the service.

An option that pertains to Unidata which has only been informally mentioned by the NWS is the possibility of Unidata providing a parallel service to the private sector community, similar to the one it provides for the academic community. In this scenario, the UPC would manage, operate and provide support for a private sector distribution system, with all incremental costs being absorbed by the commercial customers. While this option has been mentioned only as a possibility and may never materialize into a formal proposal by either the NWS or the private sector, it is worth examining its merits and discussing the implications for Unidata. Even if another option is pursued for the distribution of Level II radar data to the private sector, the issue of Unidata’s involvement in the delivery of other types of data (e.g., high-resolution operational model data) to the private sector may not go away.

Analysis of Unidata providing service to the Private Sector

The concept of Unidata’s developing, managing and supporting a private sector distribution system raises several fundamental issues. Unidata recognizes the overall importance of the services provided by the private sector and its needs for timely access to Level II data. Unidata tools are available freely to everybody so many private sector customers can already use the LDM without Unidata’s involvement in the administration and support of a private sector distribution system. Unidata has maintained a long-standing relationship with the private sector and the NWS, and it is in the best interest of the Unidata community to maintain this strong partnership with the other stakeholders.

Fundamental Issues

• Unidata’s mission is to “Provide data, tools, and community leadership for enhanced Earth-system education and research.”
• Unidata’s funding comes primarily from the NSF to execute the mission and facilitate education and research.
• Providing operational support (7x24) and data distribution to private industry would be a major departure from Unidata’s mission. This raises concerns of “mission creep” and may distract from focusing on its primary users in academia.
• Significant amount of additional resources and cost recovery mechanisms would be required for staff, space and equipment for a 7x24 operation.
• Establishing clear walls of separation between academic and commercial operations would be difficult.
• Even though the IDD/LDM system is highly reliable and efficient, Unidata could not guarantee a 10 second maximum latency and near 100% reliability, which are the stated requirements by the private sector.
• Unlike the current Unidata community, which has nearly ten years of experience with the IDD and LDM, many of the commercial users will be new to the Unidata distribution system and its guiding principles. The new user base, coupled with the competitive business pressures could result in a potential mine field for the UPC to navigate around. As a result, if problems arise with data reception and quality of service issues, the UPC could face credibility concerns by the private sector clients.
• The UPC does not have much experience in operating a distribution system for the private sector. For example, the support demands from a commercial customer base that expects a near-100% operational and time-critical data reception will be much more rigorous than the current support burden from the academic community.
• Real-time radar base data are used in severe weather warning decision support systems and emergency management operations, which raise potential liability issues for Unidata and UCAR.
• The technology gap between the academic and private sector communities are always likely to exist. For instance, the commercial customers cannot directly use the Internet 2/Abilene network and Unidata cannot address this issue.
• Given the prohibitive costs associated with using private networks and the intertwined nature of the Internet and IDD topologies, a clear bifurcation of the academic and private sector distribution systems seems unlikely.
• Long term sustainability is questionable even if adequate start-up funds are made available.

Any discussion of Unidata’s involvement in the distribution of Level II data to the private sector is not complete without referencing the seminal report Fair Weather: Effective Partnerships in Weather and Climate Services by the National Academy of Sciences’ Board on Earth Sciences and Resources (NAS/BESR) and the Committee on Partnerships in Weather and Climate Services. Several recommendations from that report are pertinent to this discussion. For example, one of the recommendations in the NAS/BESR report calls for the NWS and relevant academic, state, and private organizations to seek a neutral host, such as the AMS, to provide a venue to discuss issues related to such public-private partnerships. Another recommendation asks the NWS to provide unrestricted access to all publicly funded data and products in a timely manner and at the lowest cost to all users. This is an important recommendation because the private sector stakeholders have expressed concern about the significant costs involved in setting up their own network and distribution system for the Level II data. The report also recommends that the NWS make its data and products available in Internet-accessible digital form, using widely recognized standards, formats, and metadata description. Clearly, the NWS is adhering to this recommendation in the proposed plans for distribution of Level II data.

A recent development that is worth mentioning is the formation of Peak Weather Inc., a new technology transfer company that has been established by the UCAR Foundation. Peak Weather is planning to provide an array of services to its customers, including operating data centers. In exchange for providing commercially marketable tools and products, UCAR divisions may get some funding to sponsor their research and development activities. In this scenario, Unidata would spin-off the private operations to Peak Weather Inc. and the UPC staff would be shielded from those responsibilities.

Concluding Remarks

The most critical factor in undertaking such an endeavor is that the responsibility would fundamentally alter Unidata’s mission and character. Further discussion of this important matter amongst the Unidata Policy Committee, UCAR President, and cognizant NSF program officials seems appropriate. Preliminary discussions with Dr. Cliff Jacobs, NSF, have already taken place. Those discussions led to the development of this white paper.