IDV Development Status Report

Don Murray
September 25, 2003

This report updates the status of Unidata's IDV development efforts since the last report.

Objectives

During the 1998-2003 Unidata proposal period, the MetApps project goals were:

Deliver turn-key platform-independent applications for the analysis and visualization of meteorological data. These applications must provide and extend the most important capabilities of currently-available applications such as GEMPAK, GARP, and McIDAS. Use of these applications should require no knowledge of Java or component architectures.
Deliver easily installed executables, user documentation, and test cases for the turn-key applications.
Identify or create a component-oriented framework that supports construction of custom applications from components used to build the turn-key applications. This framework should be useful to create new combinations of components that provide subsets of the capabilities of the turn-key applications, connect them in unanticipated ways, or provide tailored mini-applications for embedding in educational materials. Use of these components may require knowledge of component architectures or Java.
Deliver full source code, class documentation, and test cases for the MetApps components.
Enlist other developers in enhancing components and developing new components for the framework by providing an archive, mailing list, developers' forums, documentation, web site, and support for the components.

The Integrated Data Viewer (IDV) was released in June 2003 and is the culmantation of this effort. The IDV reference application embodies goals #1 and #2 listed above. The IDV framework supports the development of custom applications as specified in #3 such as the specialized IDV based visualization tool in the Visual Geophysical Exploration Environment (VGEE). Source code, documentation and example applications are provided with the IDV release. Other developers are starting to provide enhancements to the IDV framework (VGEE and EPA) and many of the items in #5 are already in place. The UPC is currently investigating ways to make it easier for developers to contribute to all software projects and the IDV development will benefit from such a system.

We feel that we have met the goals of the MetApps project in the timeframe specified and are now moving on to the next phase of IDV development.

In the Unidata 2008 proposal, IDV development will continue under the proposed Endeavor 4 "Software to Analyze and Visualize Geoscience Data". Under this endeavor, the following goals are outlined:

Supporting analysis and visualization of local modeling efforts (e.g., WRF and MM5 output)
Incorporating new datasets and data types
Exploring new approaches to visualizing and interacting with Earth system data, focusing on novel 3-D techniques that fuse data from multiple sources
Expanding IDV capabilities to support the creation of exploration based, interactive, pedagogic materials and integration with digital libraries.
Developing collaborative tools to make effective use of shared visualizations
Developing a framework that allows end-user assembly and integration of IDV components
Adapting to GIS frameworks (OpenGIS and ISO standards), content-based data mining, and other evolving technologies

The IDV development team (Don Murray, Jeff McWhirter, Stuart Wier) will also be involved in Endeavor 1 "Responding to a broader and more diverse community" (e.g., adding support for new and larger datasets) and Endeavor 2 "Comprehensive support services" (e.g., providing IDV support to a growing community of users, IDV training workshops). Under Endeavor 5 "Distributed, organized collections of digital material", we will continue to work towards incorporating network accessible datasets into the IDV.

Progress since last report

In June 2003, the final 1.0 (non-beta) version of the IDV was released to the Unidata community. The IDV can be downloaded at http://www.unidata.ucar.edu/software/idv/.
Features added after the last beta 6 release included:
- Display of NOAA Wind Profiler data from ADDE servers
  - time height displays
  - plan views
  - 3D wind staffs
- Display of WSR-88D Level II Archive data
- Better support for netCDF CF 1.0 convention data including two-dimensional latitude, longitude coordinate variables.
- Support for the "latest" gridded model data.
- numerous bug fixes and performance enhancements.

In July, version 1.0.1 of the IDV was made available. This was largely a maintenance release (i.e. bug fixes) with the following exception:
- During the Unidata Summer 2003 workshop "Expanding Horizons", the need for tools that support GIS data formats was expressed by many participants. This is also a goal of item 7 of the proposed enhancements under Endeavor 4 of the 2008 propsal. With this in mind and the flexible IDV framework, a new control for displaying ESRI Shapefiles was easily added to the IDV for this release. Presently, this allows the users only to overlay shapefile polygons (i.e., roads, railroads, school district boundaries) on top of IDV data depictions. In the future this will be enhanced to provide access to the shapefile attributes as well (e.g., place names).

As part of the continuing collaboration between ATD and Unidata in relation to the IDV, ATD has submitted a proposal for an NCAR Opportunity Grant entitled Collaborative Tools and Displays for Field Projects on the Net (COLT) with Don Murray as a Co-PI. Under COLT, the IDV will be used as a visualization tool for an upcoming field project.

Collaboration has continued with DLESE on the continued development of the Visual Geophysical Exploration Environment (VGEE) using the IDV framework. Dan Bramer at UIUC has extended some of the IDV components to connect the VGEE concept models to the data available through the IDV. Stuart Wier has been working with the Chuck Meertens and Greg Benson of UNAVCO on a project to create a VGEE curriculum focused on geophysical processes. Enhancements to the IDV were made to support displays of upper mantle shear velocity anomolies. This is being done as part of the GEON project. Both of these are examples of how the IDV supports goals 3 and 5 of the MetApps project and item 4 in Endeavor 4 of the 2008 proposal.

The IDV Steering Committee is being formed. The goals of this committee are:
- prioritize the development efforts based on the perceived needs of the Unidata community and other interested parties and input from the IDV developers on feasibility and cost.
- Considers and recommends large focused development efforts in the range of 3-6 months. These would include:
  - types of data to support
  - new types of displays to implement
  - new means of use (publishing, collaboration)
  - organizational collaborations
  - educational content
- test implementations and provide feedback. Releases to the committee will occur on a 1-2 month time frame.
This group will report to the Unidata User's Committee as the MetApps Task Force that proceeded it.
To date, the following members have accepted the invitation to be on the committee:
- Charlie Murphy (Kean University) - Chair
- Tom Whittaker (University of Wisconsin - Madison)
- Bill Fingerhut (Lyndon State College)
- Chris Herbster (Embry Riddle Aeronautical University)
We are looking for 2 or 3 more members (any volunteers?) The plan is for the Committee to meet face-to-face at a kickoff meeting sometime in the next few months.

The first IDV training workshop is scheduled for November 3-4, and has registered participants from a variety of academic institutions, organizations and countries. All training materials will be available online and if possible, the workshop will be web cast. Presently, this workshop has more registered participants than the other scheduled Unidata workshops.

Presentations on the IDV capabilities have been made at the following venues:
- Don Murray provided an overview of the IDV to a UCAR wide audience at the March 17 Unidata Seminar on "IDV, THREDDS and VGEE". Ben Domenico and Rajul Pandya provided overviews of THREDDS and VGEE, respectively.
- Charlie Murphy and Bill Fingerhut led workshop labs on IDV use at the Unidata Summer 2003 Workshop "Expanding Horizons". The labs covered using gridded model data and diagnostic functions. Rajul Pandya led a lab on the use of VGEE.
- Don Murray was invited to present an overview of the IDV at the Second International Workshop on Modelling and Simulation of Geophysical Mass Flows (avalanches and volcanic debris flows) on August 14-15, 2003 in Buffalo, NY. Marcus Bursik at University of Buffalo (UB) had given a talk at AGU in which he lamented the lack of open source 3D visualization tools. Ben mentioned the IDV to him and that set the ball rolling. There was interest in using the IDV to visualize some of their model datasets.
- Don Murray provided an overview and demonstration of the IDV to the OPeNDAP breakout session at the ESIP Partners meeting on July 31.
- Don Murray provided an overview and demonstration of the IDV to a group at COMET at the request of Wendy Abshire. COMET has a contract with the Navy to provide training materials using 3-D visualizations and Wendy was interested in finding more about what the IDV could do. In the past, they've used Vis5D.
- Tom Whittaker gave two seminars on the IDV to audiences at the UW. The first was attended by more than fifty faculty, staff and students from the AOS (Atmostpheric, Oceanic and Space Science Depararment), SSEC, CIMSS, ERSC (Environmental Remote Sensing Center), CCR (Center for Climatic Research) and AMRC (Antarctic Meteorological Research Center). The second was attended by a dozen administrators and staff from SSEC. During each session, a powerpoint presentation of background information was given, followed by a half-hour live demonstration, including showing the remote collaboration capability by connecting to a running IDV display in Boulder.
- Abstracts for talks/papers/posters on the IDV have been submitted and accepted for the AAG East Lakes/West Lakes Regional meeting in Kalamazoo, MI (October 16-18) and the AMS IIPS conference in Seattle (Jan 11-16, 2004). The IDV will be displayed at the UOP booth during the AGU meeting in San Francisco in December.

New development

Testing of IDV 1.1 beta1 is currently underway and this version will be released just before the November workshop. New features (and their relation to the Unidata 2008 endeavors where applicable) include:

Enhancement of displays of Level II Archive Format in the IDV (Endeavor 4, Items 2 and 3). New features include:
- RHI displays in 2- and 3-D
- 3D sweep views
- Isosurfaces of Level II moments
- Nested cone displays
- Time series and data probes

New colloboration facility that allows sharing of displays on remote machines (Endeavor 4, item 5).

Support for displaying Digital Elevation Model (DEM) data from USGS and ArcInfo ASCII Grid files (Endeavor 4, item 7).

New control for displaying profiles of data along a transect (Endeavor 4, item 3)

Initial support for reading GeoTIFF data files (Endeavor 4, item 7).

Support for reading and displaying all valid VisAD data supported data types. If not geo-referenced, these can be displayed in an enhanced version of the VisAD Spreadsheet cell. (Endeavor 4, items 2 and 3)

Preliminary work on a Jython-based display control that allows developers and advanced users to create new types of displays and perform data manipulation via Jython scripts rather than writing and compiling Java code. Jython is a scripting language more like MatLab or IDL and is more familiar to many researchers than Java. (Endeavor 4, item 6)