THREDDS

Status Report: Oct 2012 - April 2013

John Caron, Ethan Davis, Dennis Heimbigner, Marcos Hermida, Sean Arms, and Lansing Madry

Strategic Focus Areas

The THREDDS group's activities support Unidata's strategic goals in the following ways:

Enable widespread, efficient access to geoscience data

The work of the THREDDS group is comprised of two main areas: the THREDDS Data Server (TDS) and the Common Data Model (CDM) / netCDF-Java library. The TDS provides catalog and data access services for scientific data using OPeNDAP, OGC WCS and WMS, HTTP, and other remote data access protocols. The CDM provides data access through the netCDF-Java API to a variety of data formats (e.g., netCDF, HDF, GRIB). Layered above the basic data access, the CDM uses the metadata contained in datasets to provide a higher-level interface to geoscience specific features of datasets, in particular, providing geolocation and data subsetting in coordinate space. The CDM also provides the foundations for all the services made available through the TDS.

The data available from the IDD is a driving force on both the TDS and netCDF-Java development. The ability to read all the IDD data through the netCDF-Java library allows the TDS to serve that data and provide services on/for that data.

Develop and provide open-source tools for effective use of geoscience data

Unidata's Integrated Data Viewer (IDV) depends on the netCDF-java library for access to local data, and on the THREDDS Data Server (TDS) for remote access to IDD data. At the same time, the CDM depends on the IDV to validate and test CDM software. Many other tools build on the CDM / netCDF-Java library (eg ERDDAP, Panolpy, VERDI, etc) and on the TDS (LAS, ncWMS, MyOcean, etc).

Provide cyberinfrastructure leadership in data discovery, access, and use

The Common Data Model (CDM) / netCDF-Java library is one of the few general-purpose implementations of the CF (Climate and Forcast) metadata standards. Current active efforts in CF that we are involved with include draft Satellite and Radar Conventions.

The TDS has pioneered the integration of Open Geospatial Consortium (OGC) protocols into the earth science communities. Strong international collaborations have resulted in WCS and WMS services as part of the TDS.

The CDM and TDS are widely used implementations of the OPeNDAP DAP2 data access protocol. Unidata is working with the OPeNDAP group to design, develop, and implement a new version of the DAP specification, DAP4.

Activities Since the Last Status Report

Release Status of CDM and TDS

  • CDM and TDS 4.3.16 are the current stable releases.
  • Unidata TDS: Due to major changes in GRIB handling between 4.2 and 4.3, the official Unidata TDS has been migrated to the latest release (4.3.16). Incorrect GRIB data handling in the previous version, 4.2, affects approximately 25% of all GRIB parameters from NCEP models (and possibly much more for non-NCEP models), resulting in misleading or outright incorrect data variable names. To provide users of the Unidata TDS time to transition, we've created a timeline in which services from 4.2 are still accessible in the short term:
    • Friday, March 22nd, 2013 - the official data access URL for the Unidata TDS will be thredds.ucar.edu. This address will resolve to a TDS 4.3 server (the latest version).
    • Saturday, June 1st, 2013 - the previous data access URL, motherlode.ucar.edu, will be changed to use the TDS 4.3 server. This marks the end of TDS 4.2 access on Unidata systems.
    • Thursday, August 1st, 2013 - the data access URL motherlode.ucar.edu will no longer function. All users of the Unidata TDS should transition to using thredds.ucar.edu by this time.
    Note that the current release of the IDV (4.0u1) forces all bundles to load using data from the 4.3 server, thredds.ucar.edu, and the main IDV catalog points to the new server.

Collaboration Activities

  • Work continues on OPULS (OPeNDAP-Unidata Linked Servers) project to finalize DAP4 specification, implement DAP4 clients and servers, develop DAP4 conformance testing capabilities. The OPULS project recieved second year funding from NOAA -- approximately one third of originally budgeted amount.
  • Working with NCAR/MMM WRF group:
    • to improve CF compliance of WRF netCDF output
    • on tools to allow WRF to read input from netCDF files (unnetcdf)
    • on TDS file download subset service to return WRF required fields on requested space/time subset.
    • on improved support of staggered grids and time dependent coordinates
  • Working with NOAA/ESRL/GSD on checking the GRIB tables used with HRRR data.
  • Work underway on project for "Project to Improve Software Infrastructure in Support of the NOAA National Climate Model Portal (NCMP)", funded by NOAA/NOMADS grant.
  • ncSOS plug-in developed by ASA (Applied Science Associates) and USGS CIDA provides OGC SOS service for data conforming to the CF Discrete Geometries data types. Looking at distributing with the TDS version 5.0.

International Standards Development

  • Track and participate in OGC MetOcean Working Group discussion.
  • Track and participate in OGC WCS development.
  • Track and participate in OGC CF-netCDF development.

Planned Activities

CDM / TDS 4.3.17 Development

  • Bug fixes
  • Improve CdmRemoteFeature service for Point/Station Collections
  • GRIB FeatureCollection fixes for NCDC
  • Improve netCDF-4 writing - structures

CDM / TDS 4.3.18 Development

  • Bug fixes only

CDM / TDS 4.4 Development

  • Port Earth System Grid (ESG) requested enhancements from TDS 4.2 (see TDS-213)
  • Refactor TDS catalog caching for Earth System Grid (ESG)
  • Ugrade CDM/TDS use of HttpClient from 3.x to 4.x (3.x is end-of-life)
  • Refactor/harmonize NCSS, CdmRemote, and RadarServer services
  • Add WRF initialization service to NCSS
  • Add WaterML as output format from NCSS point service
  • Experiment with Asynchronous responses

CDM / TDS 4.5 Development

  • DAP4
  • Grid Feature Collection (assimilate FMRC?)
  • Add Constant Forecast Offset/Hour back to GRIB
  • CdmRemoteFeature for all CF-DSG feature types

CDM / TDS 5.0 Development

  • Catalog 2
  • TDS configuration refactor (ease of use and support dynamic re-init)
  • Refactor GridDatatype to Coverage
    • Swath/Image
    • Unstructured Grid
    • Time-dependent coordinate system
    • Cross-seam lat/lon data requests
  • Improved metadata harvesting support
  • Search/discovery service?
  • Require Java 7 and Tomcat 7 (start using java.nio.file)