In 1998, researchers at a number of leading universities expressed a need for improved access to high-resolution data sets from NCEP, similar to the form of access Unidata provides for other (operational) data flows from NOAA/NWS, including NOAAport. The Unidata methodology is dubbed IDD, for Internet Data Distribution, and it is used in most meteorology departments nationwide. To address the aforementioned need, a joint USWRP, NOAA, NASA, and Unidata project was formed under the name CONDUIT (the Cooperative Opportunity for NCEP Data Using IDD Technology). A specific CONDUIT objective is for model-generated NCEP data sets of interest to USWRP researchers—including those which have been available only through FTP servers at NCEP and NWS/OSO— to be accessible via IDD. The premise for this objective is that IDD (which is built on Unidata's LDM software) would:
Currently, model data are prepared at NCEP and distributed to NWS/OSO using NCEP's DBNet. The data are then inserted into an LDM at NWS/OSO and sent to NASA Goddard via a dedicated T1 (1.544 megabits per second) link. From NASA, the data are relayed via Internet to three additional sites running LDMs (see Fig 1), which in turn relay the data to all other recipients. This IDD topology scales without limit; specifically the number of recipients can grow without ever increasing congestion, merely by adding additional relays. The only impact is that newer recipients encounter additional delay, but experience indicates that the delay remains acceptable. This is as expected, because IDD delay grows as the log of the number of sites, whereas delay introduced by FTP (or DBNet) congestion grows exponentially with the number of recipients, in accordance with queueing theory.
Unfortunately, the OSO-Goddard T1 link is seriously undersized in relation to the current demands being placed on CONDUIT; hence large delays are being introduced at the head of the IDD distribution tree. (The same delay would be incurred with FTP or DBNet, because the data volume greatly exceeds the line capacity.) The reasons for the mismatch derive from changes in the expectations being placed on CONDUIT since the original concept was articulated. Specifically:
Compounding the T1 bottleneck from October 1999 to February 2000 was a severe problem in an NWS router. This problem went unsolved for 5 months because its onset coincided with a fire that destroyed one of the main computers at NCEP. Hence the problem with CONDUIT was thought to be related to the destruction of NCEP's computer. CONDUIT has been functioning flawlessly since February, except for delays introduced by the T1 link.
NOAA is now establishing Next Generation Internet (NGI) connectivity in the DC area. CONDUIT would benefit immensely by utilizing this development in the following ways:
This upgrade is consistent with the new capabilities being tested and provided through the Internet2/NGI. Indeed, enhancing CONDUIT via Abilene (which is an academically oriented component of the NGI) could be "...a key way to turn the potential created by the emerging Internet2 infrastructure into value for the nation's scientists," in the opinion of Guy Almes, Acting Director of Abilene.
We believe the NGI/Internet2 connections in the Silver Spring and Camp Springs areas would be ideal for CONDUIT. Specifically, LDMs at OSO and/or NCEP could be configured to send data via NGI to two or three Abilene sites (thus providing redundancy). This would eliminate the present T1 bottleneck and have zero impact on the loading of the NWS computers now running LDM.
Many gains would derive from this new method of distribution for CONDUIT. These include:
