NetCDF

Status Report: April - September 2012

Russ Rew, Dennis Heimbigner, Ward Fisher

Strategic Focus Areas

The netCDF group's activities support Unidata's strategic goals in the following ways:

  1. Enable widespread, efficient access to geoscience data
    by developing netCDF and related cyberinfrastructure solutions to facilitate local and remote access to scientific data.
  2. Develop and provide open-source tools for effective use of geoscience data
    by supporting the use of netCDF and related technologies for analyzing, integrating, and visualizing multidimensional geoscience data; enabling visualization and effective use of very large data sets; and accessing, managing, and sharing collections of heterogeneous data from diverse sources.
  3. Provide cyberinfrastructure leadership in data discovery, access, and use
    by developing useful data models, frameworks, and protocols for geoscience data; advancing geoscience data and metadata standards and conventions; and providing information and guidance on emerging cyberinfrastructure trends and technologies.
  4. Build, support, and advocate for the diverse geoscience community
    by providing expertise in implementing effective data management, conducting annual training workshops, responding to support questions, maintaining comprehensive documentation, maintaining example programs and files, and keeping online FAQs, best practices, and web site up to date; fostering interactions between community members; and presenting community perspectives at scientific meetings, conferences, and other venues.

Activities Since the Last Status Report

Project and Issue Tracking

We continue to use the Jira project tracker tool to manage bug reports, track issues, plan releases, and make our development process more transparent to users. Between April 2012 and September 2012, we created 35 issues, updated 58 issues, resolved 34 issues, and we currently have 54 open issues for netCDF-C, 11 open issues for netCDF-Fortran, and 2 open issues for netCDF-C++. (Note: Jira issues vary greatly in size and effort required to resolve, so number of issues is not a useful measure of amount of work to do.)

Releases

Versions 4.2.1 and 4.2.1.1 of netCDF-C software were released in July and August, 2012. These releases included important portability and performance enhancements, bug fixes, and new features.

Release 4.2.1 included two important developments: portability to Windows platforms, and diskless netCDF "files" for all netCDF formats, providing high performance for operations on netCDF data that can fit in memory.

Porting netCDF to Windows included building and making available 12 different binaries for both static and shared, 32- and 64-bit libraries, with various combinations of OPeNDAP client access and netCDF-4/HDF5 support. This should make access to netCDF data easier and increase interoperability for communities that have standardized on Windows platforms (e.g. hydrology). It will also ease porting new versions of netCDF to widely used commercial software, such as ESRI ArcGIS and MATLAB.

The Windows porting work is already making an important impact on netCDF usage and interoperability. It made possible the recent netCDF-Java implementation of a netCDF-4 writing capability, which is needed for enhanced Common Data Model services such as netCDF subsetting. We have also noticed fewer requests for help building netCDF from source on Windows platforms, now that pre-built libraries and executables for the latest released version are available.

NetCDF subject of NSF-funded cyberinfrastructure study

NSF is funding a study that

… seeks to create a software taxonomy framework that will provide data-driven insight on NSF's software environment investments. In addition, the project will develop a set of recommendations, based upon case study observations and software inventory data, for NSF consideration on future studies of software sustainability factors. NetCDF was chosen by the project's advisory group as a case study candidate.

Unidata will be participating in the study by hosting an on-site visit of project researchers on September 20.

Code quality improvement through static analysis

In addition to extensive testing on multiple platforms, we are now regularly making use of a static analysis tool (Coverity) that can reveal bugs, resource leaks, and security problems not easily caught by run-time testing or compiler warnings. In July, netCDF joined other widely used open source projects that are qualified for free analysis under the Coverity SCAN initiative, a joint Coverity and DHS effort to make open source software more secure. Initial use has lowered the measured defects per thousand lines of code from 1.989 to 1.625.


Planned Activities

Ongoing Activities

  • Respond to support questions and help requests from netCDF users.
  • Improve support for netCDF on various platforms.
  • Incorporate successful features of netCDF-Java into C-based libraries.
  • Respond to needs of a growing user community for representing observational data, satellite products, and geoinformatics data.

New Activities

During the next six months, we'll be running new netCDF training workshops, adding Windows support for MS Visual Studio builds using CMake, improving and completing the documentation generated by Doxygen, releasing an updated version of netCDF-Fortran and netCDF-C++ packages, and benchmarking performance improvements made possible with netCDF-4 for the January AMS meeting.

Metrics

During the last year, there were 80,000 downloads from over 116 countries of the C-based netCDF software from Unidata, in addition to downloads from mirror sites, package management systems, and incorporation into other software packages. A record of over 10,300 downloads in one month were recorded for the release of the new netCDF version 4.2 in May 2012. More detailed metrics about downloads for the last year, including affiliation types for institutions, are available.

Other metrics that may be useful include number of

  • Google hits for "netcdf": 1,310,000
  • Google hits for "netcdf-3": 686,000
  • Google hits for "netcdf-4": 612,000
  • Bing hits for "netcdf": 386,000
  • Google images for "netcdf": 42,200
  • forum discussions mentioning netCDF: 32,100
  • blog mentions of netCDF: 24,400
  • videos tagged as netCDF-relevant: 15,700
  • Google scholar entries for "netcdf": 8,320
  • books containing the term "netcdf": 5,250
  • free software packages that can access netCDF data: 106
  • U.S. patents mentioning netCDF: 37
  • commercial software packages that can access netCDF data: 21