NetCDF

Status Report: October 2010 - April 2011

Russ Rew, Ed Hartnett, Dennis Heimbigner

Strategic Focus Areas

The netCDF group's work supports the following Unidata funding proposal focus areas:

  1. Broadening participation and expanding community services
    by providing effective support to a growing netCDF user community, including users of satellite products, GIS, climate, and ocean data.
  2. Advancing data services
    by developing and supporting a key infrastructure element for data providers and users of climate, atmosphere, and ocean data.
  3. Developing and deploying useful tools
    by engineering general-purpose software and utilities supporting the analysis, visualization, and management of model output and observational data.
  4. Enhancing user support services
    by maintaining comprehensive documentation, providing annual workshops, responding to support questions, maintaining example programs and files, and keeping online FAQs, best practices, and the netCDF web site up to date.
  5. Providing leadership in cyberinfrastructure
    by seeking endorsement of standards bodies for netCDF formats and for Climate and Forecast (CF) model metadata conventions based on netCDF.
  6. Promoting diversity by expanding opportunities
    using an open source development process, accepting user contributions, and democratizing access to valuable data archives.

Activities Since the Last Status Report

Workshops

We presented two Unidata netCDF workshops on October 28-29, 2010, in the Damon room of NCAR's Mesa Lab. In addition to presentations by Unidata staff Ed, Russ, Dennis, John, Ethan, Yuan, and Jeff, we also had invited presentations on NCAR Climate Data by Gary Strand (NCAR CGD), HDF5 for netCDF Developers by Quincey Kozial (The HDF Group), and GRIDSPEC by Alex Pletzer (Tech-X). With over fifty participants for both days, things went surprisingly well even though Four Mile Canyon caught fire during the workshop, requiring Ed to evacuate his family between two of the presentations!

The workshops were preceded by a useful one-day "Data Summit" with HDF5 developers as well as grad students and faculty with the Damasc research project at UCSC.


Releases

After several beta test releases, netCDF version 4.1.2 of the C-based software was announced on March 29, 2011. The most important changes in this release were:

  • changes for building cleanly on Windows;
  • building shared libraries by default, instead of static libraries;
  • greatly speeding up opening of large or complex netCDF-4 files, writing a large number of attributes, or lookups by name in files with lots of variables or dimensions;
  • adding security authorization to the C OPeNDAP client for Earth System Grid use and for standard HTTP authorization using passwords; and
  • adding useful compression and chunking capabilities to the nccopy utility.

Windows DLLs are now being built as part of the daily snapshot build/test cycle; separate Visual Studio build files are no longer required. Testing of the resulting DLLs is underway.

The 4.1.2 release also included a number of bug fixes as well as improvements in portability, extensibility, maintainability, and additional new features described in the Release Notes.


Standards progress

A new web page tracks standards endorsements related to netCDF. The most recent is an OGC endorsement for use of netCDF as an encoding standard, which makes it a standard for accessing data with OGC web services.


Presentations

For the AGU Fall Meeting, Ed coauthored a poster on GRIDSPEC/libCF and Russ presented a talk on advances in the netCDF data model, format, and software. Ed also presented a poster on recent netCDF developments at the EGU Spring Meeting, and Russ presented several talks on scientific data management and netCDF at a workshop on high-performance and grid computing hosted by the International Centre for Theoretical Physics in Trieste. All these posters and presentations are available from the NetCDF Papers and Presentations web page.

Planned Activities

Ongoing Activities

We plan to continue the following activities:

  • Respond to C- and Fortran-based netCDF user questions and run netCDF workshops.
  • Incorporate successful features of netCDF-Java into C-based libraries.
  • Improve support for evolving Climate and Forecast (CF) conventions.
  • Improve support for netCDF on Windows platforms.
  • Oversee development of GRIDSPEC in libCF as part of a collaboration with NOAA's Global Interoperability Project, PCMDI, GFDL, and Tech-X, anticipating that this will become part of the CF conventions.
  • Deal with needs of a growing user community for representing observational data, satellite products, and geoinformatics data.
  • Add support for additional security authorization mechanisms as needed.

New Activities

Our netCDF development plans are now being entered into the Jira issue tracking system, where bugs, sub-projects, issues, tasks, and progress may be followed transparently by users and other developers. We're also referring users to specific Jira tickets for tracking progress on bugs or issues in which they have an interest. In the last 30 days, we've created 51 new issues and resolved 16 of them, but we hope to get closer to equilibrium once we have most of the accumulated issues entered into Jira.

We're also drafting a NASA ACCESS proposal with a university partner proposing the development of "CF Metadata Conventions for Satellite Data and Derived Products".