NetCDF and Unidata

Work in maintaining, supporting, and developing the netCDF data model and software is associated with Endeavor 6: Improved scientific data access infrastructure from the Unidata 2008 proposal. NetCDF has become a key infrastructure element for producers and consumers of atmospheric science data, as well as data in other geosciences.

Recent netCDF development, both at Unidata and at other institutions, aims at improving interoperability with other representations for scientific data, making the netCDF interface more suitable for use on high-end parallel platforms with high-resolution models, and providing netCDF software on a wider range of desktop platforms.

NetCDF/HDF5 Merger

In November, we presented an interim review of the NASA-funded project, budget, and milestones for a NASA Advanced Information Systems Technology (AIST) review team. We also presented a poster on the project at the annual AGU meeting in December.

Ed Hartnett completed a prototype implementation of the netCDF-3 API using HDF5 as a storage layer. This implementation passed all netCDF automatic testing at the beginning of December. The implementation comprises approximately 13,000 lines of C code. The goals for this implementation were to:

1. Establish that HDF5 could be used as a storage layer for the netCDF data model. This has been established.
2. Verify that backward compatibility with netCDF-3 programs and data could be achieved by just recompiling and relinking. This was verified.
3. Identify the desired enhancements to HDF5 to accommodate netCDF-4 more easily. These new features have been documented in a requirements list, and the HDF5 team is working to implement them. They are scheduled to be implemented some time in the 3rd quarter of 2004.
4. Prove that read/write times and file sizes would not grow prohibitively using HDF5. This has been proved.

We presented a paper and poster on the project at the annual AMS meeting in January. At the same time, Ed worked on the merger between netCDF-3.x and netCDF-4. This entails testing to promote the 3.5.1 beta release of netCDF to a full-fledged release, completing two release cycles with current netCDF-3.x enhancements, and then merging in the netCDF-4 implementation.

As extensive testing on multiple platforms is required for each release, Ed has begun to more fully automate the test process with a free-ware Python tool called BuildBot. Using BuildBot will save testing time for each release, and also, by making it easier to test netCDF on all supported platforms, will ensure that netCDF is fully tested any time the code changes. This will allow us to detect portability problems immediately.

A tentative netCDF release schedule can be found at http://www.unidata.ucar.edu/software/netcdf/release_schedule.html. A partial list of supported platforms can be found at http://www.unidata.ucar.edu/software/netcdf/supported_platforms_3_5_1.html.

Parallel netCDF

We continue to work with researchers at Argonne National Laboratory and Northwestern University who have developed a parallel interface for writing and reading netCDF data, tailored for use on high performance platforms with parallel I/O. Their implementation builds on the MPI-IO interface, providing portability to most platforms in use and allowing users to leverage the many optimizations built into MPI-IO implementations. We advised Greg Sjaardema of Sandia Labs in implementing a new version of the netCDF format that uses 64-bit offsets, then convinced the Argonne/Northwestern group to adopt the 64-bit offset changes into their parallel netCDF implementation. We intend to release the same changes in version 3.6 and migrate that support into netCDF-4.