Summary of the
Unidata Users Committee
1 - 2 October 1998 at the
Unidata Program Center (UPC)
Boulder, Colorado

Members:
Jennie Moody , Chair
Rich Clark
Dave Dempsey
Steve Koch
Michael Morgan (MetApps Rep.)
Mohan Ramamurthy
Tony Rockwood
Clint Rowe
Jim Steenburgh
Doug Yarger

Ex-Officio Members:
Jim Moore, Policy Committee
Tom Whittaker, ATAC

UPC Staff:
Sally Bates
Steve Chiswell
Ethan Davis
Ben Domenico
Steve Emmerson
Dave Fulker
Joanne Graham
Robb Kambic
Linda Miller
Don Murray
Russ Rew
Donna West
Tom Yoksas

Administrative Matters:

New committee members were welcomed.
UPC's new program administrator, Joanne Graham, was introduced.
Outgoing chairman, Mohan Ramamurthy, was warmly thanked for his long and exemplary service. The bell was passed to Jennie Moody.
The action items 4 and 5 from the last meeting have not been completed. Action 4 will be continued to the next meeting; Action 5 is tabled pending NWS developments. Action 3 has been superseded by UPC's MOU with Alden.
The next meeting of the committee will be: 15-16 April 1999.

Director's Report:

Fulker gave a preview of the report he is preparing for the upcoming Policy Committee:

The 5-year NSF proposal "Unidata 2003" was reviewed very positively, but the budgets had to be resubmitted at levels much lower than requested; this was done by defining a minimal "baseline" effort, complemented by a number of potential "advancement projects" that would be undertaken with additional funding.
A Memorandum of Understanding (MOU) with Alden and the U WI, signed 30 September, greatly accelerates the replacement of FOS data with NOAAport data for delivery on Unidata's Internet Data Distribution (IDD) system. Alden will provide NOAAport receivers--built with components from SSEC at the U WI--for use at 2 university sites (probably FL St and U AZ). These will be complemented by NOAAport receivers at Alden, U WI, and some Alden customer to create an IDD-linked network of 5 downlinks, offering high reliability through redundancy. Data from this network will be available to Unidata participants (via IDD) under the same terms and conditions as at present, and the data will flow to Alden and SSEC customers as well. In addition to gaining a downlink network at no cost, Unidata benefits from the immediate cessation of the FOS contract (with Alden). Unidata's role is to adapt and support the LDM software as needed for reliable operation of the downlink network, including the creation of FOS-like products. Wisconsin provides certain downlink hardware and software.
Fulker will report to the Policy Committee on Unidata's general status as follows:

Data sources: yellow; there are risks inherent in dropping FOS so quickly; also, the future of NIDS is uncertain due to pending (but as yet unknown) changes in the NWS relationship to NIDS vendors.
Data distribution: green; IDD now delivers over 80 gbytes/day in the aggregate (calculated from site-reported statistics); NWS funds have been promised to continue the joint case-study effort with COMET; the Distributed Oceanographic Data System (DODS) project is off to a good start (with NASA funding received via U RI).
Current Software: yellow; WXP is increasingly oriented toward commercial use; future NWS support for GEMPAK is unclear; Wisconsin is discontinuing support for OS/2-McIDAS.
Scheduling: green; a new schedule was defined and submitted to NSF along with the revised budgets.
Financial: red; the NSF for now cannot commit even to the proposed baseline level of funding; indeed, the discrepency exceeds the savings achieved via the Alden MOU; Unidata will receive no KDI funds; compensating measures include depleting carryover reserves, decreasing Web support, having no booth at the AMS meeting, and working to implement the Alden MOU.
Staff: yellow; Unidata now has no Webmaster, and this puts pressure on the documentation staff; the rapid shift to NOAAport puts pressure on the LDM and IDD support staff; however, Unidata's recent reorganization seems to be positive.

Discussion:

A common reason for lack of NIDS subscriptions (via IDD from WSI) is that people can get current NIDS products via the Web.
There is no specific contingency plan in case NWS ceases GEMPAK support. If Unidata use is to continue under such a scenario, UPC staff would need unlimited access to source code (which also would enable the borrowing of ideas and approaches for Java development). However, UPC staff cannot compensate for cessation of NWS-funded advancements in GEMPAK or GARP
Support for AWIPS at universities is not defined as a "baseline" Unidata activity. Exploration of options is planned, but any official support posture would require additional funding, and there are significant practicality questions because the inexpensive (Java-based) AWIPS workstations require the services of a full-up AWIPS site in order to gain the data they display.
The UPC proposes to cope with the OS/2 dilemma by officially undertaking support for Red Hat Linux and Solaris x86.

Policy Committee Report

Moore noted that the last meeting was dominated by the NSF funding issues. Other topics included access to Level II data, the formation of the MetApps Taskforce (which was endorsed by the Policy Committee), the initiative to inject NCEP model data into the IDD, the Russ DeSouza Award, and questions about maintaining staff at UCAR in the face of high-tech growth along the Front Range.

MetApps Report

Summaries of the MetApps meeting are in the notebook. Morgan distributed an example use-case (C. Murphy's use-case for an interactive skew-T). He noted that the Users Committee will be a source of ideas for software development and suggested that the committee begin thinking about how effective communication among the broader community, the Users Committee, and MetApps can be established.

Discussion:

The motivation for building Java applications was discussed: Java reduces Unidata's platform support problems and moves Unidata software toward current technologies and languages.
UPC staff were asked about the value of the use cases. While it's early to evaluate this, Emmerson said he found the skew-T use cases very useful in identifying unusual needs.
Use cases will include research needs as well as educational ones.
Use cases will become part of the applications documentation.
Use cases are expected to be helpful in ensuring new applications are effective in changing environments (hardware, software, research needs, teaching needs).
The philosophy behind Unidata's shift to Java needs to be fully explained to the community. Need to stress that the community can help shape the development of these tools. And there need to be milestones established so that everyone can evaluate progress in this effort.
The proper balance between supporting current packages and undertaking new development is an ongoing topic of discussion in the UPC and the Policy Committee. Generally, support received priority.

Action 1:

Fulker will describe Unidata's Java transition in his presentation to the AMS Head & Chairs October 8th; there will also be an article on this in the next issue of the newsletter.

NOAAport:

Fulker briefly summarized the design of NOAAport, including the NWS use of terrestrial networks to add reliability. This is especially important, e.g., during periods of solar occultation by the data-broadcasting satellite. The LDM can provide reliable NOAAport reception by utilizing geographically distributed injection sites that experience solar occultations at different times. Duplicate products that may arise from such redundancy are eliminated in the LDM through the use of unique product signatures. The data stream will not include DIFAX; we believe it will include the "additional" international data (which currently is a hot political issue).

Discussion:

We've yet to read fully the NWS request for comments on NIDS.
The exclusion of DIFAX won't affect Unidata users, since Alden will continue to offer DIFAX to its subscribers. In addition, the NWS makes DIFAX products available on the Web.
A single site could be the injection source on IDD for DIFAX-like products.
The switch to NOAAport should be largely transparent. UPC will create a separate feed-type for NOAAport, and will not strive to make NOAAport exactly like FOS, so users may see a few changes.
NOAAport will include some gridded data, but not RUC2, 1-degree global, or ensemble data, for example. Our NCEP initiative will complement (not duplicate) NOAAport products.

Platform Issues

The UPC's proposal concerning platform support is in the notebook.

Discussion:

The staff was questioned about the impact of adding Linux and Solaris x86 support. The staff felt the impace would be minimal. The UPC already supports Solaris, so there is no additional support needed for x86; the Red Hat work has already been done, and the staff won't be testing other variants of Linux (though they expect the software to run on all/most variants without significant problems).
There were questions about the future of Solaris x86. The staff believes Sun will be supporting it for the foreseeable future.
There were questions about systems administration requirements of Linux and Solaris x86, compared to OS/2. The staff believes that administering these is harder, but on every campus it will be easier for individuals to find technical support for these than for OS/2.
There was considerable discussion about Windows NT. Wisconsin is investigating Interix and many campuses are installing NTs. This is precisely why the UPC wants to transition to Java--to avoid these operating system issues. Porting the LDM to NT would require significant effort, since the LDM depends on UNIX features such as shared memory-mapped files.
Rockwood reported on his transition from OS/2 to x86: Murray helped with the installation, and he trained Rockwood. Once everything was installed, both Rockwood and his students were very happy with the results--a much faster system. These success stories need to be shared with others in the community.
There are some limitations to x86: it doesn't come with compilers or perl. Users need to download and install perl to use LDM perl scripts, and would need to download the free gcc compiler in order to compile source releases. We can provide binary releases that need no compiler.
There was considerable discussion on how to sunset OS/2 support.

Resolution 1 (passed unanimously):

The Unidata Users Committee endorses the following changes in Unidata's platform support:
A. Immediately sunset support for ULTRIX and SunOS 4.x.
B. Add support for Red Hat Linux and Solaris for Intel (x86).
C. Sunset support for OS/2 on July 31, 1999.

Action 2:

Rockwood will work with Bates to publish an article in the next newsletter on his transition to x86.

Action 3:

The Users Committee will keep an eye on NT developments.

Java Discussion

Russ's presentation is available on the Web at:

http://www.unidata.ucar.edu/staff/russ/jdi/usercomm.html

Discussion:

Solaris x86 doesn't support Java 3-D yet, so users cannot render 3-D displays with VisAD; 2-D displays are possible.
VisAD models data, visualizations, manipulations, and user interfaces in a general way. In order to exploit this, domain-specific frameworks (for atmospheric science) need to be created.
There is now a test of VisAD's 3-D capabilities--you can import a netCDF file into the VisAD spreadsheet and display the data.
There were questions about sources for additional help (development work, money). The component approach may encourage community development. There is also the potential of some involvement with NWS (FSL, for example, is looking into developing Java tools), but the NWS is so intensely focused on AWIPS implementation that no collaboration can be expected in the near term. NCAR Graphics is a copyrighted package so cooperation there is impossible.

Archival Data

Dave Fulker explained that, while Unidata's main focus is real-time data, the program is currently engaged in three efforts involving archival data: (1) SSEC stores the McIDAS data stream on line for 20-30 days, then moves the data off-line; (2) Unidata collaborates with COMET and JOSS to make COMET case studies available on the Web; and (3) With funding from NASA, Unidata is now providing support for the Distributed Ocean Data System (DODS), under development byt the U. Rhode Island. DODS allows applications to access data from a URL as though they were local. While most data on DODS is oceanographic, NCAR's HAO is moving its CEDAR database onto DODS.

Discussion

Some of the COMET case studies have class activities attached. More of these are needed.
Faculty need some help on how these data sets can be used in the classroom. Some felt that Unidata should ask case-study users for lab materials; others felt that this was not an appropriate Unidata task.
COMET selects which case study is to be loaded next onto the system.
There was considerable discussion about sharing data within the community. Many sites are unable to respond to requests for data because of time constraints. Committee members would like to have access to SCD data archives, but one needs an NSF grant to be given a password, and then there is the problem of dealing with data formats. Unidata doesn't have the resources to build translators for SCD data sets. Users will also want similar access to NCDC data.
There was some discussion about UPC creating an ADDE server for filenames other than the McIDAS names, and then having SSEC put this server on the Unidata/Data Recovery machine at SSEC.

Resolution 2 (passed unanimously):

In agreement with the recommendation by the NSF review panel, the Users Committee recommends that the Unidata Policy Committee explore Unidata collaboration with other agencies and organizations (e.g., NCAR SCD) to facilitate access to archived meteorological data in forms compatible with Unidata-supported software.

Outstanding Participant Award

Bates reported that the procedures outlined by the Policy Committee for selecting an award recipient were such that most of the outstanding participants would be inelegible--they are always serving on some committee. She suggested that the committee might request that the Policy Committee change the eligibility requirements.

Discussion:

There was agreement on the existence of a problem and considerable discussion about how to design a fair selection procedure.

Resolution 3 (passed unanimously):

The Users Committee requests the Policy Committee give the Users Committee full authority for making the Russell DeSouza Award. Current Users Committee members will be excluded from consideration. The Users Committee will seek nominations from the community, discuss the nominations in an open meeting, and render its decision in executive session.

Status Reports

GEMPAK

Chiswell reported that the NWS developers are planning to distribute another release to SOOs, so continued N-AWIPS development appears likely for at least a year. Also, N-AWIPS is still used to create products that cannot be generated in AWIPS (such as marine forecasts). We anticipate an unofficial release containing new capabilities, such as N-MAP, that are not fully documented. Chiswell anticipates having to create some documentation for Unidata users. He also noted that COMET was focusing on AWIPS, so there may be little or no development of the GARP interface.

Chiswell reported that the distribution of gridded model data from NCEP would be announced to the community later this fall.

Discussion:

There was considerable discussion about the impact of AWIPS on the university community, particularly if NWS advancement of N-AWIPS (GEMPAK) ceases. Few universities will be installing AWIPS, so a disconnect between the academic and the NWS communities may develop. (AWIPS appears relatively impractical for Unidata adoption, at least for now.) The disconnect will be minimized if NWS support for the use of GEMPAK (N-AWIPS) by SOOs continues, since GEMPAK/GARP is widely used on campuses and serves as an application common to forecasters and academics. The best hope for campus use of AWIPS is manifest in the (Java-based) FX-net system developed at FSL, but a workable strategy for Unidata implementation has yet to be defined.
The development of GARP and N-TRANS has allowed some universities to cease subscribing to DIFAX; they use GARP and N-TRANS to produce higher quality DIFAX-like products.

Action 4:

Clark and Moore, working with Fulker, will consider how best to present the GEMPAK/GARP concerns to the NWS.

Action 5:

A demonstration of the FX-net system will be on the agenda for the next meeting.

McIDAS

Yoksas reported on the release of McIDAS 7.40, which included changes in ADDE capabilities (ADDE will be fully supported by the end of November). McIDAS documentation is now all on line, with SSEC and UPC commands fully integrated; the UPC will no longer distribute paper copies. The training workshops were well attended, and there was a request by some to hold a training workshop sometime during the summer instead of the fall.

During a demonstration of ADDE at the workshop, Yoksas and Murray realized that ADDE server software (which is part of the McIDAS distribution) could be used to create a distributed data archive for McIDAS users. ADDE uses client/server architecture; users can browse holdings on remote servers (it requires simply knowing the IP address of the remote machine and having read permission on that machine) and request a product by data-set name. By sharing address information, and setting appropriate permissions, a community of users could serve data to each other (or a small school with minimal equipment and low bandwidth, could obtain data from somewhere else). The system could be further enhanced if users agreed to common file-naming conventions.

Discussion:

The ADDE data distribution model is similar to DODS, but DODS is not limited to McIDAS users. The UPC's Java development is designed to take advantage of remote access capabilities.
There is always the problem of firewalls.
There was discussion about the possibility of forming a "buddy system" where a large school would set up a data server that small schools, without LDM, could then access to acquire data. With some work, the ADDE serve could be turned into a push system (either by running rsh commands on the clients or by coding "remote file write servers" that would allow the server to write files to the remote client machine (the smaller school). Only two such "file write servers" now exists, and they are both for image data.
The ADDE/DODS servers offer models for implementing the data access resolution (Resolution 2, above).
Yoksas intends to explore the possibility of establishing a satellite data server, possibly at NASA/Marshall.

DODS

Davis reported that DODS and ADDE do have some commonalities. DODS, however, is based on HTTP protocols and allows no writing to the server. The commercial packages MatLab and IDL, if recompiled with the DODS library, can be used to access files (no Unidata applications are supported). The applications access remote data via URLs. The UPC, under a U RI contract funded by NASA, will provide user support for DODS.

Discussion:

There are no plans to date to develop DODS support for Unidata applications.
DODS is compatible with several APIs, including netCDF and JGOFS.
DODS servers now exist for some near-real-time data, but there is no automatic synchronization when new data are added.

Case Studies

Davis reported that the placing of four case studies in the last two months was done with help from PAGE staff and a student assistant.

External Programs

Miller reiterated the status of the NCEP model-data distribution effort. She is coordinating the project with representatives from USWRP, NWS/OM, and NASA/Goddard.

Discussion:

The data sets involved are huge, so bandwidth and disk space are important considerations.
The data sets will be available on the IDD; relaying these will probably involve a relay structure different from FOS data.
On an annual basis, a working group composed of representatives from WSRP, OM, NASA, and Unidata will review which data sets are being distributed.

Summary of the Unidata Users Committee 1 - 2 October 1998 at the Unidata Program Center (UPC) Boulder, Colorado

Administrative Matters:

Director's Report:

Discussion:

Policy Committee Report

MetApps Report

Discussion:

Action 1:

NOAAport:

Discussion:

Platform Issues

Discussion:

Resolution 1 (passed unanimously):

Action 2:

Action 3:

Java Discussion

Discussion:

Archival Data

Discussion

Resolution 2 (passed unanimously):

Outstanding Participant Award

Discussion:

Resolution 3 (passed unanimously):

Status Reports

Discussion:

Action 4:

Action 5:

Discussion:

Discussion:

Discussion:

PAGE

Discussion:

Brainstorming on the Next Workshop

Summary of the
Unidata Users Committee
1 - 2 October 1998 at the
Unidata Program Center (UPC)
Boulder, Colorado