On Friday, March 28, 2014 the Unidata Program Center (UPC)
updated the demonstration THREDDS Data Server (TDS) running on
thredds.ucar.edu to version 4.5 of the TDS software. After
encountering some issues with the updated version, including
problems with community members' IDV bundles not loading as
expected from thredds.ucar.edu, UPC staff reverted the
demonstration server to version 4.3 on April 1st.
What happened?
UPC staff had became aware that TDS version 4.3 running on
thredds.ucar.edu was not keeping some dataset indices up to
date automatically. This situation required manual
intervention by UPC staff to keep the indices updated, which, coupled
with changes in UPC security procedures following the
intrusion on February 20, 2014, increased the urgency of
releasing a new TDS version that fixed the problem. After internal testing
convinced us that TDS version 4.5 was functioning properly,
we followed the normal procedure of updating UPC
servers (specifically thredds.ucar.edu) before
releasing the package to other sites. This staged release process
normally allows us to find and fix "real world" issues that
are not caught by internal testing.
In the case of TDS 4.5, however, the update brought to light
a difference between our internal test servers and the
public demonstration server thredds.ucar.edu. These
differences caused the demonstration server to crash entirely,
which resulted in several
problems, including the inability of some IDV bundles to
access relevant data.
Additionally, there was a problem with the way the new GRIB indexer
handled GRIB1 timeType=10 and some changes to GRIB tables from
NCEP were not handled in TDS 4.5.
What did Unidata do?
After receiving problem reports on Monday, March 31, UPC staff
reverted the thredds.ucar.edu demonstration server to TDS version 4.3
on Tuesday, April 1. The problems in TDS version 4.5 have been
rectified, but thredds.ucar.edu has not yet been re-updated.
What will Unidata do in the future?
While the UPC is not funded as an operational data center, and we do not have resources available to ensure 24/7 access to all of our demonstration systems, we realize that many in our community do rely on access to UPC servers for teaching and research. As a result, we strive to balance the daily needs of community members who use servers managed by the program center with development and testing activities aimed at creating tools that can be usefully installed at community members' home sites. (See Unidata's Use Policy.)
As a result of this upgrade situation, we have decided to implement the following changes to our system configuration, software, and procedures.
- We have created a new mailing list named "data-service-announce@unidata.ucar.edu" that will serve as a conduit for UPC staff to announce upcoming changes to data servers operating at the UPC (IDD, TDS, RAMADDA, ADDE) to our community. This mailing list will carry brief synopses of upcoming changes, information about the testing process (including how community members can test the new services), and provide links to additional information on the Unidata web site. The new list will be the conduit for announcements about data servers run by the UPC. We anticipate that this mailing list will have very low traffic, and we encourage anyone who uses data servers run by the UPC to subscribe by clicking this link.
-
We will augment the UPC's core data servers (
thredds.ucar.edu,adde.ucar.edu, etc.) with "backup" servers that will allow users to revert to using the most recent stable version if problems are found during an upgrade or other transition. Backup servers will only operate during the transition, but will remain available until reported problems are addressed. IDV version 5.0, due to be released later this Spring, will include a way for users to switch to the "backup" TDS server if needed. We expect this to be used for critical functions such as teaching classes. - We will institute a mechanism whereby community members can submit IDV bundles to be included in Unidata's test suite, allowing us to determine immediately when a change causes the bundles to "break." Note that while being able to include community members' IDV bundles in our in-house automated testing of new releases will alert us to some problems, it is not a full substitute for interactive user testing. We will provide more information about the bundle-testing program as we figure out the details.
We plan to re-update thredds.ucar.edu with TDS 4.5 in mid-May 2014; we will
provide at least one week's notice and testing period (announced via the data-service-announce
mailing list) prior to the update.
If you have questions about any of these ideas, please send e-mail to support-datastream@unidata.ucar.edu.
