THREDDS

Status Report: Oct 2013 - May 2014

John Caron, Ethan Davis, Dennis Heimbigner, Sean Arms,
Lansing Madry, Christian Ward-Garrison, and Ryan May

Strategic Focus Areas

The THREDDS group's work supports the following Unidata funding proposal focus areas:

  1. Enable widespread, efficient access to geoscience data

    The work of the THREDDS group is comprised of two main areas: the THREDDS Data Server (TDS) and the Common Data Model (CDM) / netCDF-Java library. The TDS provides catalog and data access services for scientific data using OPeNDAP, OGC WCS and WMS, HTTP, and other remote data access protocols. The CDM provides data access through the netCDF-Java API to a variety of data formats (e.g., netCDF, HDF, GRIB). Layered above the basic data access, the CDM uses the metadata contained in datasets to provide a higher-level interface to geoscience specific features of datasets, in particular, providing geolocation and data subsetting in coordinate space. The CDM also provides the foundations for all the services made available through the TDS.

    The data available from the IDD is a driving force on both the TDS and netCDF-Java development. The ability to read all the IDD data through the netCDF-Java library allows the TDS to serve that data and provide services on/for that data.


  2. Develop and provide open-source tools for effective use of geoscience data

    Unidata's Integrated Data Viewer (IDV) depends on the netCDF-java library for access to local data, and on the THREDDS Data Server (TDS) for remote access to IDD data. At the same time, the CDM depends on the IDV to validate and test CDM software. Many other tools build on the CDM / netCDF-Java library (eg ERDDAP, Panolpy, VERDI, etc) and on the TDS (LAS, ncWMS, MyOcean, etc).


  3. Provide cyberinfrastructure leadership in data discovery, access, and use

    The Common Data Model (CDM) / netCDF-Java library is one of the few general-purpose implementations of the CF (Climate and Forcast) metadata standards. Current active efforts in CF that we are involved with include draft Satellite and Radar Conventions.

    The TDS has pioneered the integration of Open Geospatial Consortium (OGC) protocols into the earth science communities. Strong international collaborations have resulted in WCS and WMS services as part of the TDS.

    The CDM and TDS are widely used implementations of the OPeNDAP DAP2 data access protocol. Unidata is working with the OPeNDAP group to design, develop, and implement a new version of the DAP specification, DAP4.


Activities Since the Last Status Report

Release Status of CDM and TDS

  • CDM and TDS 4.3.21 are the current stable releases.
  • CDM and TDS 4.5 is available as a Beta release. (See below for a list of major changes.)
    • After extensive testing, TDS 4.5 was deployed to Unidata's demonstration TDS server (thredds.ucar.edu). This uncovered a number of bugs not found by earlier testing which led to rolling back the server to TDS 4.3. As a result of this situation, Unidata has decided to implement a number of changes to our server upgrade process. (More details are available here.)

Collaboration Activities

  • Recent non-core awards that involve THREDDS development:
    • EarthCube Building Blocks award: "Integrating Discrete and Continuous Data" with Univ of Texas, Austin and others. Period of performances: Oct 2013 - Sept 2015.
    • EarthCube Building Blocks award: "Specifying and Implementing ODSIP, A Data-Service Invocation Protocol" with OPeNDAP, Inc.
    • EarthCube Building Blocks award: "Deploying Web Services Across Multiple Science Domains" with IRIS, UNAVCO, and others. Period of performances: Oct 2013 - Sept 2015.
    • NASA ROSES ACCESS award: "High Performance Multidisciplinary Open Standard Data Services to Serve Terrestrial Environmental Modeling" with USGS CIDA. Period of performances: 1 Jan 2014 - 31 Dec 2015.
  • Work continues on OPULS (OPeNDAP-Unidata Linked Servers) project:
    • Demonstrated to the NOAA DMIT the interoperability between the OPeNDAP developed server, the Unidata developed server, and the Unidata developed client library.
    • Currently integrating DAP4 server into TDS 4.5
    • The third and final year of OPULS funding ends 30 Sept 2014.
  • Work continues on project to scale-up the TDS to handle large collections of GRIB files continues, "Project to Improve Model Data Access under Unidata's THREDDS Data Server (TDS) in support of NOAA's National Climate Model Portal (NCMP)". Funded by NOAA/NCDC.
  • Working with NCAR/CISL DSS group to make some of their large GRIB model collections available using the new TDS 4.5 Feature Collections.
  • Renewed effort with NOAA/ESRL/GSD to check the GRIB tables used with HRRR and FIM data.
  • ncSOS plug-in developed by ASA (Applied Science Associates) and USGS CIDA provides OGC SOS service for data conforming to the CF Discrete Geometries data types. Looking at distributing with the TDS version 4.6.

International Standards Development

  • Track and participate in OGC MetOcean Working Group discussion.
  • Track and participate in OGC WCS development.
  • Track and participate in OGC CF-netCDF development.

THREDDS Team

  • Christian Ward-Garrison started at Unidata in early December 2013. He is working on THREDDS project, including work on the EarthCube Building Blocks - Integrating Discrete and Continuous Data project.
  • Ryan May started at Unidata in early January 2014. He is working on the THREDDS project and looking into several Python projects.

Planned Activities

CDM/TDS 4.5 Development (underway)

We are skipping the release of 4.4 to focus on getting some important 4.5 changes out

  • Further improvements to the TDS for handling of GRIB collections, including a return of the two time dimension view for forecast models.
  • Improve the TDS handling of collections of point, station, and sounding data. Move (and harmonize) the resulting services under the NCSS interface.
  • Include in the TDS an alpha version of the DAP4 service.
  • Ugrade CDM/TDS use of HttpClient from 3.x to 4.x (3.x is end-of-life)
  • Refactor TDS catalog caching for Earth System Grid (ESG).
  • Require Java 7 and Tomcat 7 (start using java.nio.file)

CDM/TDS 4.5.x Development

  • Add WaterML as output format from NCSS point service
  • Improve the TDS RadarServer services and harmonize it with the NCSS interface.
  • Simplify process for adding new TDS web services. Document simplified process so third-party development groups can more easily add new web services to the TDS.
  • Work with ncWMS group and others to add features to the version of ncWMS used by TDS.
  • Add WRF initialization service to NCSS
  • Port Earth System Grid (ESG) requested enhancements from TDS 4.2 (see TDS-213)

CDM/TDS 4.6 Development

  • Grid Feature Collection (assimilate FMRC?)
  • Experiment in TDS with various server-side processing capabilities.
  • Experiment in TDS with Asynchronous responses
  • Work on providing FMRC Constant Forecast Offset/Hour funtionality for large datasets. (Possibly as NCSS-like service rather than as index based access.)
  • Provide NCSS access for all CF-DSG feature types.

CDM / TDS 5.0 Development

  • Catalog 2
  • TDS configuration refactor (ease of use and support dynamic re-init)
  • Refactor GridDatatype to Coverage
    • Swath/Image
    • Unstructured Grid
    • Time-dependent coordinate system
    • Cross-seam lat/lon data requests
  • Improved metadata harvesting support
  • Search/discovery service?