THREDDS

Status Report: September-April 2011

John Caron, Ethan Davis, Dennis Heimbigner, and Robb Kambic

Strategic Focus Areas

The THREDDS group's work supports the following Unidata funding proposal focus areas:

 

Advancing data services

 

The work of the THREDDS group is comprised of two main areas: the THREDDS Data Server (TDS) and the Common Data Model (CDM) / netCDF-Java library. The TDS provides catalog and data access services for scientific data using OPeNDAP, OGC WCS and WMS, HTTP, and other remote data access protocols. The CDM provides data access through the netCDF-Java API to a variety of data formats (e.g., netCDF, HDF, GRIB). Layered above the basic data access, the CDM uses the metadata contained in datasets to provide a higher-level interface to geoscience specific features of datasets, in particular, providing geolocation and data subsetting in coordinate space. The CDM also provides the foundations for all the services made available through the TDS.

 

The data available from the IDD is a driving force on both the TDS and netCDF-Java development. The ability to read all the IDD data through the netCDF-Java library allows the TDS to serve that data and provide services on/for that data.

 

Developing and deploying useful tools

 

Unidata's Integrated Data Viewer (IDV) depends on the netCDF-java library for access to local data, and on the THREDDS Data Server (TDS) for remote access to IDD data. At the same time, the CDM depends on the IDV to validate and test CDM software. Many other tools build on the CDM / netCDF-Java library (eg ERDDAP, Panolpy, VERDI, etc) and on the TDS (LAS, ncWMS, MyOcean, etc).

 

Providing leadership in cyberinfrastructure

 

The Common Data Model (CDM) / netCDF-Java library is one of the few general-purpose implementations of the CF (Climate and Forcast) metadata standards. Current active efforts in CF that we are involved with include the Discrete Sampling proposal, as well as draft Satellite and Radar Conventions.

 

The TDS has pioneered the integration of Open Geospatial Consortium (OGC) protocols into the earth science communities. Strong international collaborations have resulted in WCS and WMS services as part of the TDS.

 

Activities Since the Last Status Report

THREDDS Data Server (TDS)

  • TDS currently at 4.2.5. Some major features since 4.2 first released include:
    • Distribution now includes the ncISO metadata services (thanks to Dave Neufeld and Ted Habermann, NOAA/NGDC)
    • Add OPeNDAP handling of ESG access control (other access control improvements)
    • Subsetting services for Point Obs data extended to Station data (other improvements as well)
  • TDS 4.2 became our stable release on 22 November 2010. The main new features were:
    • Refactor FMRC aggregation to simplify configuration and improve performance.
    • Improved integration with ncWMS code including adding the Godiva2 browser based WMS client.
    • Addition of experimental subsetting services for Point Obs data (Discrete Sampling data)

CDM / netCDF-Java library

  • CDM 4.2.23 is the current release. The main features since 4.2.0 include:
    • The Discrete Observations Convention is now complete, should be added to CF-1.6.
    • Nexrad2: dual pole radar
    • GradsBinaryGridServiceProvider (Don Murray)
    • Netcdf-Java can now be built with maven. It will be added to the standard maven repository.
    • Opendap parser now uses the same grammer as the C library.
  • CDM 4.2 includes:
    • FMRC aggregations have been refactored to dynamically adjust to changes in model output, eg from the NCEP IDD feed. The FMRC "Best" dataset can be parameterized by forecast time. The TDS admin can choose which FMRC datasets to expose in the catalogs.
    • A new abstraction was developed for managing large collections of data, which can be used by NcML and other packages.
    • Caching of Grid dataset info.
    • Caching of File System info.
    • GRIB processing now correctly identifies time interval variables such as accumulations, averages, min/max over intervals, etc. This feature was propagated up the CDM stack, eg into FMRC aggregations.
    • OPeNDAP parsing is now done from the same grammar as with the OPeNDAP/netCDF C library.
    • ncstream works for the full netcdf-4 data model. performance tests are very encouraging.
  • Other CDM related activities
    • present at 4th Extremely Large Databases Conference (XLDB)
    • present at USGS Community for Data Integration Workshop.

International Standards Development

  • Track and participate in OGC MetOcean Working Group discussion.
  • Track and participate in OGC WCS development.
  • Track and participate in OGC CF-netCDF development.

Collaboration Activities

  • Discussions with USGS groups on possible CDM/TDS collaborations.
  • Discussions with OPeNDAP on possible Hyrax/TDS collaboration.

Planned Activities

CDM / TDS 4.3 Development

  • GRIB-2 refactor for efficiency and correctness.
  • Refactor THREDDS catalog parse/build software targeting performance and ease of use.
  • Improved metadata extraction from datasets for use in catalogs and other metadata services.
  • Add new services for Discrete Sampling Features (station, profiler, aircraft track data e.g.).
  • Improve performance of collections and aggregations made up of large numbers of datasets.
  • Experimental asynchronous / streaming / remote access services (cdmRemote)