NetCDF

Status Report: April 2014 - September 2014

Russ Rew, Ward Fisher, Dennis Heimbigner

Strategic Focus Areas

The netCDF group's activities support Unidata's strategic goals in the following ways:

  1. Enable widespread, efficient access to geoscience data by developing netCDF and related cyberinfrastructure solutions to facilitate local and remote access to scientific data.
  2. Develop and provide open-source tools for effective use of geoscience data by supporting the use of netCDF and related technologies for analyzing, integrating, and visualizing multidimensional geoscience data; enabling visualization and effective use of very large data sets; and accessing, managing, and sharing collections of heterogeneous data from diverse sources.
  3. Provide cyberinfrastructure leadership in data discovery, access, and use by developing useful data models, frameworks, and protocols for geoscience data; advancing geoscience data and metadata standards and conventions; and providing information and guidance on emerging cyberinfrastructure trends and technologies.
  4. Build, support, and advocate for the diverse geoscience community by providing expertise in implementing effective data management, conducting training workshops, responding to support questions, maintaining comprehensive documentation, maintaining example programs and files, and keeping online FAQs, best practices, and web site up to date; fostering interactions between community members; and presenting community perspectives at scientific meetings, conferences, and other venues.

Activities Since the Last Status Report

Project and Issue Tracking

We use a project tracker tool to manage bug reports, track issues, plan releases, and make our development process more transparent to users. Between 1 April 2014 and 29 August 2014, we created 15 new Jira issues, updated 3 issues, resolved 9 issues, and we currently have 81 open issues. In addition, 17 GitHub issues were resolved, and 8 GitHub issues currently remain open. (Note: issues vary greatly in size and effort required to resolve, so number of issues is not a useful measure of amount of work to do.)

An important milestone during the last six months was completion of CMake support for netCDF-Fortran, which makes it possible to now build netCDF-Fortran libraries on Windows platforms, after installing the netCDF-C library. Another related milestone is prominent use of netCDF data in ESRI visualizations in one of the short opening plenary talks at the recent annual ESRI User Community meeting.

The netCDF-C test dashboard continues to provide results of testing the most recent development code with various configurations on multiple platforms.

Releases

The netCDF-C 4.3.2 release was made available in April, following 2 release candidates. Since then, we announced a release candidate for version 4.3.3 that includes various bug fixes and enhancements to portability and documentation, as described in the latest Release Notes.

A July release of netCDF-Fortran version 4.4.0, the first full Fortran release since October 2011, added support for recent language standard updates, in particular C-compatibility features that have greatly improved portability for various Fortran compilers on a variety of platforms. Use of the C compatibility feature in modern Fortran standards instead of a complex netCDF-specific header file has already lessened our support burden for netCDF-Fortran, as demonstrated by the decline in support questions.

Another important netCDF-related release is version 2.2.17 of the UDUNITS package, adopted over a decade ago by the CF (Climate and Forecast) Conventions for netCDF metadata. Although previous versions of UDUNITS have been easy to install on Unix-based platforms, this is the first version adapted to support building and installing on Windows. Its C library provides for arithmetic manipulation of units and for conversion of numeric values between compatible units. The package also contains an extensive user-extendable units database and a command-line utility for investigating units and converting values.

Collaborations

Increasing collaboration includes continued "pull requests" from community developers contributing fixes, as well as use of the GitHub issue-tracking system.

We continue to work with Jeff Whitaker (NOAA/ESRL), developer of netcdf4-python, a widely used Python interface to netCDF-4 now hosted in the Unidata GitHub repository.

A collaborator at Mississippi State, Associate Research Professor Richard Weed, has contributed much of the new Fortran-2003 code as well as new Fortran-2008 enhancements.

Planned Activities

Ward will be presenting an afternoon session on netCDF in September for a group of visitors from the Chinese Aviation Agency, as part of a two-week workshop associated with a RAL project.

For the October Unidata Training Workshops, we plan to lead sessions on the use of netCDF with Python, in collaboration with other Unidata developers and Jeff Whitaker.

Short- and medium-term development plans are driven by the list of about 90 open Jira and GitHub issues for netCDF libraries, utilities, and documentation.

During the next six months, we also plan to continue efforts to

Ongoing Activities

New Activities

Longer-term plans include addressing a need discussed at the last Strategic Advisory Committee meeting, to survey and help tool providers work towards more complete support of the netCDF-4 enhanced data model. There has already been much recent progress adding netCDF-4 support to NCO and NCL. NCO now supports groups, chunking, and all of the netCDF-4 primitive types. NCL now provides beta-level support for the complete NetCDF 4 data model, including primitive and user-defined netCDF-4 data types: string, variable-length, compound, enumeration, and opaque, as well as features like groups, chunking, compression, and caching. The netcdf4python package also continues to add support for more features of the enhanced data model.

Efforts we have taken to encourage and support use of the enhanced data model include:

Metrics

Detailed metrics, including for netCDF-Java/CDM, are available.

Other metrics, with comparisons from 5 months ago, include number of

Prepared September 2014