IDV with RAMADDA

Status Report: September 2014 - March 2015

Yuan Ho, Julien Chastang

This report updates the status of Unidata's Integrated Data Viewer (IDV) development efforts since the last report (September, 2014).

Strategic Focus Areas

The IDV group's work supports the following Unidata funding proposal focus areas:

  1. Enable widespread, efficient access to geoscience data
    The IDV is a state of the art geoscience visualization application. It gives users the ability to view and analyze a rich set of geoscience data, including real time data, in a seamless and integrated fashion. This analysis is captured in IDV bundles. RAMADDA is a content management system and service specifically tailored towards the sharing and distribution of IDV bundles facilitating distribution of scientific data and analysis.
  2. Develop and provide open-source tools for effective use of geoscience data
    The IDV has been an open-source project for several years. The IDV is available on the github version control platform for greater open-source collaboration. The IDV provides users the unparalleled ability to analyze, integrate, and visualize heterogeneous geoscience data in two, three, and four dimensions. The IDV coupled with RAMADDA enables geoscience specialists the capability to share and collaborate their IDV analysis via social scientific networks.
  3. Provide cyberinfrastructure leadership in data discovery, access, and use
    RAMADDA allows geoscience specialists the ability to search and publish their IDV bundles on-line. Unidata's RAMADDA installation enables the IDV team to communicate more effectively to our users concerning their IDV issues. Specifically, during support ticket conversations, the IDV team requests that users upload pertinent data to RAMADDA for analysis. The IDV team also takes advantage of RAMADDA to share instructional IDV screencasts with users.
  4. Build, support, and advocate for the diverse geoscience community
    Unidata offers yearly multi-day training and occasionally regional workshops for IDV and RAMADDA. The IDV coupled with RAMADDA enables our earth science community partners to distribute geoscience data and metadata through web-based technologies thereby fostering scientific collaborations. Moreover, the IDV's ability to share bundles through RAMADDA creates a scientific social and collaborative network for the geoscience community.

Activities Since the Last Status Report

System Changes

Latest netCDF-Java Version: 4.5
The version of the netCDF-Java library currently distributed with the IDV is 4.5. The 4.5 release includes improvements related to I/O, NIO.2, HttpClient 4, improved projections, and updated GRIB support. See netCDF-Java Library for a more details on these changes.

IDV distribution, installation and program execution: There have been several changes related to security and distribution of the IDV. In particular, the Windows, Mac OS X, and Java Webstart versions of the IDV installers are now signed with certificates from certificate authorities (CA). We continue to fix bugs and make improvements to the mechanism that provides an IDV old version warning to users. The Install4J script continues to be maintained and improved. These changes result in a smoother installation and execution of the IDV for our users.

RAMADDA plugin: The RAMADDA IDV plugin has been enhanced to facilitate the publication of movies and images, including images as zip files, to RAMADDA. This change improves the display of movies published from the IDV to RAMADDA.

IDV Webstart: The Java Webstart version of the IDV has been fixed following problems associated with the Java 3D migration to Java 7.


Display Changes

CSV Time Matching: The IDV now has time matching capability for CSV point data files. In this scenario, users first need to create the display, and subsequently specify a driver time. In particular, users need to select View > Times > Uses Time Driver Times in the display control of the CSV display. Reloading the data will create the display with matching driver times.

IDV Webcams: We continue to maintain a list of webcams that can be accessed through the IDV.


Data Changes

CFRadial netCDF Format Support: The IDV now has the capability of viewing CFRadial netCDF format radar data files. We collaborated with engineers from NCAR/EOL to achieve this result and we continue to make changes, improvements, and bug fixes in this area.

AMRC Text Point: IDV can now handle text point data from the Antarctic Meteorological Research Center (AMRC).


IDV and RAMADDA Training and Conference Attendance

  • Produced three new IDV training videos:
    1. Creating a 3D map
    2. Creating a Jython Formula
    3. Combining Data with Time Matching
  • Presented "Using Progressive Resolution to Visualize large Satellite Image dataset" at the 2015 AMS conference in Phoenix.
  • IDV introduction at WRF Tutorial, January 28-30, 2015
  • Participated in the NSF DataViz Hackathon for Polar CyberInfrastructure on November 3-4, 2014

RAMADDA

Cloud Activity: Docker is a new cloud-centric technology that borrows from the notion of containers from the shipping industry to facilitate installation and deployment of server side applications. We are investigating and exploring the possibility of creating Docker containers for easy distribution and installation of RAMADDA in a cloud environment. We have a prototype RAMADDA Docker image we are are experimenting with and intend to build upon in the near future.

Planned Activities

New Activities

Preparing to submit a NASA ROSES ACCESS proposal in collaboration with SSEC.

Preparing for forthcoming WRF workshops and tutorials.

Preparing for Costa Rica IDV/McV training for May, 2015.

Preparing for University of South Florida Regional Workshop in April, 2015.


Relevant Metrics

The IDV team continues to provide the geoscience community with high-quality support through e-support software and idv-users maillist. The volume of e-support remains high and constitutes a large fraction of our daily activities. In the last half year the IDV team has closed ~175 e-support tickets. Each individual ticket may and often does involve many back-and-forth messages.

The number of both casual and regular IDV users is stable or increasing. In February 2015, there were 130 IDV users starting the IDV more than 20 times per month compared with 119 users for the same period in 2014. In the same month, there were 74 IDV users starting the IDV more than 30 times per month compared with 67 users for the same period in 2014. Each month there were about 1700 to 2000 unique IP addresses where IDVs were run at least one time. Notably, the number of the IDV users from the edu domain is the largest among the top level domains (about 10 times the second top level domains). Readers can find the raw metrics at http://www.unidata.ucar.edu/software/idv/logging/left.html.

In the area of greater collaborative development, since the migration of the IDV project to github, we have accepted 85 "pull requests" or code contributions from internal and external collaborators.

In the area of online IDV training, the Youtube IDV instructional videos have been viewed over 4,840 times compared with 3,000 from six months ago.The video on trajectories has generated an especially great interest.