NetCDF Status

Russ Rew, Ed Hartnett, and Dennis Heimbigner
April 6, 2010

This report updates the last status report of Unidata's netCDF development efforts since the Joint Policy Committee and User Committee meeting in October 2009.

Relation to Unidata 2013 proposal

NetCDF efforts are associated with Theme 2: Advancing data services from the Unidata 2013 proposal. NetCDF continues to serve as a key infrastructure element for data providers and users of climate, atmosphere, and ocean data.

The overall goals for the next five years are to

Manage a graceful transition from a simple data model (netCDF-3) to the enhanced Common Data Model of netCDF-4
Provide better support for remote access and server-side data manipulation
Respond to the need to faithfully represent observational data as well as gridded data
Scale up to handle larger volumes of data efficiently
Provide effective support to a larger netCDF user community, including users of satellite products, GIS, climate, and ocean data

Recent progress

Recent netCDF developments improve client access to remote data services, documentation, portability, ease of use, interoperability with other data representations, and performance on high-end platforms.

NetCDF and libcf Development

Enhancement of OPeNDAP client integration through user feedback
Improvement of netCDF 4.1 default chunk size algorithms through user feedback
Release of version 4.1 and 4.1.1 of the netCDF C/Fortran/C++ libraries. These releases include remote data access via built-in OPeNDAP client, ncgen that works with netCDF-4 enhanced data mode, ability to read some HDF4 and HDF5 data files, use of the parallel-netcdf library for parallel I/O to classic format files, a new nccopy utility, bug fixes and portability and performance enhancements.
Inclusion of a new C++ API for the enhanced netCDF-4 data model, contributed by a researcher at the UK Culham Centre for Fusion Energy
Development of experimental new iterator-style interface for C access to variable data of any size, shape, or type
Release of libCF-alpha6, including GRIDSPEC code from GFDL to assist AR-5 data providers
Addition of test for HDF5-1.8 bug writing netCDF-4 data on big-endian platforms in specific circumstances

Presentations and Meetings

At October GO-ESSP meeting in Hamburg, Russ presented information about a proposed new Climate and Forecast (CF) conventions standard and information on status of the proposed CF conventions for point observation data. Ed presented developments in libcf and related software, including new features in netCDF 4.1, the OPeNDAP client library, reading HDF4 and HDF5 archives, inclusion of udunits, developments in parallel I/O, and GridSpec.
In November at GFDL, Ed presented at the NOAA-sponsored Global Interoperability Project (GIP) meeting about new interoperability features in the netCDF C/Fortran/C++ library (including access to existing HDF5 and HDF4 data archives), and libcf, which provides assistance with the creation and use of files that use CF metadata conventions.
In March at the NCDC Workshop on Ensuring Access and Trustworthiness of Climate Observations and Models for Society, Russ and John presented "Enabling technologies for facilitating access and use of data"

Near-term Plans

During the next 6 months, plans include:

Finish port of C-based netCDF 4.1.1 for Windows developers
Write guide and FAQ section for netCDF-4 chunking and compression
Refactor netCDF classic/enhanced/DAP dispatch architecture for additional extensions and improved maintainability
Refactor how configure handles HDF5 and other libraries
Develop and implement the Princeton API for GridSpec for use in libcf
Get approval for CF Conventions extensions for CF observations data (used in IDV)
Move the effort forward to agree on a CF convention for model ensembles
Create netCDF-4 test file collection to support tool developers
Joint work with NCDC for improving and integrating Unidata infrastructure in NCDC's National Climate Model Portal (NCMP)
Implement and test netCDF simple record streaming as specified in standards RFC submission
Submit RFC for netCDF-4 for the NASA Earth Science Standards group
Migrate netCDF file and program examples, contributed programs, and current distribution sources to RAMADDA
Enhance ncdump to support selection of specified groups
Enhance nccopy to support changes to chunking, compression
Provide examples that use new observational conventions

Metrics

Unidata's netCDF software is widely deployed in the NSF research and education community. Making the software freely available without registration means it's difficult to know the full extent to which netCDF is used, because it's available from mirror sites and is distributed with some OS-specific package management systems.

Metrics data: Mar 1 2009 to Feb 28 2010