THREDDS

Status Report: May 2011 - October 2011

John Caron, Ethan Davis, Dennis Heimbigner, and Robb Kambic

Strategic Focus Areas

The THREDDS group's work supports the following Unidata funding proposal focus areas:

Advancing data services

The work of the THREDDS group is comprised of two main areas: the THREDDS Data Server (TDS) and the Common Data Model (CDM) / netCDF-Java library. The TDS provides catalog and data access services for scientific data using OPeNDAP, OGC WCS and WMS, HTTP, and other remote data access protocols. The CDM provides data access through the netCDF-Java API to a variety of data formats (e.g., netCDF, HDF, GRIB). Layered above the basic data access, the CDM uses the metadata contained in datasets to provide a higher-level interface to geoscience specific features of datasets, in particular, providing geolocation and data subsetting in coordinate space. The CDM also provides the foundations for all the services made available through the TDS.

The data available from the IDD is a driving force on both the TDS and netCDF-Java development. The ability to read all the IDD data through the netCDF-Java library allows the TDS to serve that data and provide services on/for that data.

Developing and deploying useful tools

Unidata's Integrated Data Viewer (IDV) depends on the netCDF-java library for access to local data, and on the THREDDS Data Server (TDS) for remote access to IDD data. At the same time, the CDM depends on the IDV to validate and test CDM software. Many other tools build on the CDM / netCDF-Java library (eg ERDDAP, Panolpy, VERDI, etc) and on the TDS (LAS, ncWMS, MyOcean, etc).

Providing leadership in cyberinfrastructure

The Common Data Model (CDM) / netCDF-Java library is one of the few general-purpose implementations of the CF (Climate and Forcast) metadata standards. Current active efforts in CF that we are involved with include the Discrete Sampling proposal, as well as draft Satellite and Radar Conventions.

The TDS has pioneered the integration of Open Geospatial Consortium (OGC) protocols into the earth science communities. Strong international collaborations have resulted in WCS and WMS services as part of the TDS.

 

Activities Since the Last Status Report

Collaboration Activities

  • Awarded NOAA funding for "OPeNDAP-­Unidata Linked Servers (OPULS)". $2.9M over 3 years, split with the OPeNDAP group (Dave Fulker, PI). First year work includes creating specification for new version of the DAP along with conformance testing. proposal
  • Awarded NOAA/NOMADS grant for "Project to Improve Software Infrastructure in Support of the NOAA National Climate Model Portal (NCMP)". $125K for one year. This includes scaling TDS to handle 25K file collections, plus writing new IOSPs for NOMADS datasets. SOW

THREDDS Data Server (TDS)

  • TDS 4.2.8 is the current stable release. There have been two bug fix releases since the last status report.
  • TDS 4.2.9 has been released, with various bug fixes and an upgrade of ncISO, and new features to support use of TDS by the Earth System Grid (ESG).
  • Working on TDS 4.3, release planned in 3-6 months (see below for details on planned features).

CDM / netCDF-Java library

  • CDM 4.2.26 is the current release. There have been two bug fix releases since the last status report.
  • Suppport for CF-Radial Convention for radar, lidar, etc.
  • Working on netCDF-Java/CDM 4.3, release planned in 3-6 months (see below for details on planned features).

International Standards Development

  • Track and participate in OGC MetOcean Working Group discussion.
  • Track and participate in OGC WCS development.
  • Track and participate in OGC CF-netCDF development.

Planned Activities

CDM / TDS 4.3 Development (underway)

  • OPeNDAP: support client side certificates. Support Groups with DAP 2 protocol (TDS).
  • Refactor of GRIB-2 and GRIB-1 IOSPs including table handling.
  • New code to handle GRIB collections can scale up to large collections of files, first application is NOMAD's CFSR dataset
  • Improved calendar handling, e.g., 360-day years for climate data.
  • Synchronize CDM with CF 1.6 Discrete Sampling Features (station, profiler, aircraft track data e.g.). Add new services to TDS for these data types.
  • Refactor TDS catalog caching for Earth System Grid (ESG)
  • Improved integration with ncISO
  • Upgrade to use latest ncWMS code
  • Improved TDS support for Tomcat 7 (Servlet 3.0/JSP 2.2)
  • Code repository for 4.3 now under git on gitHub. Evaluating git for collaborative development.

CDM / TDS 4.4 Development

  • Refactor THREDDS catalog parse/build software targeting performance and ease of use.
  • Refactor TDS configuration targeting ease of use and dynamic re-initialization.
  • Improved metadata extraction from datasets for use in catalogs and other metadata services.
  • Improve existing and add new services for CF 1.6 Discrete Sampling Features (station, profiler, aircraft track data e.g.).
  • Improve NetCDF Subset Service (NCSS)
  • Experimental asynchronous / streaming / remote access services (cdmRemote)