THREDDS

Status Report: September 2014 - March 2015

John Caron, Ethan Davis, Dennis Heimbigner, Sean Arms, Christian Ward-Garrison, and Ryan May

Strategic Focus Areas

The THREDDS group's work supports the following Unidata funding proposal focus areas:

  1. Enable widespread, efficient access to geoscience data
    The work of the THREDDS group is comprised of two main areas: the THREDDS Data Server (TDS) and the Common Data Model (CDM) / netCDF-Java library. The TDS provides catalog and data access services for scientific data using OPeNDAP, OGC WCS and WMS, HTTP, and other remote data access protocols. The CDM provides data access through the netCDF-Java API to a variety of data formats (e.g., netCDF, HDF, GRIB). Layered above the basic data access, the CDM uses the metadata contained in datasets to provide a higher-level interface to geoscience specific features of datasets, in particular, providing geolocation and data subsetting in coordinate space. The CDM also provides the foundations for all the services made available through the TDS.

    The data available from the IDD is a driving force on both the TDS and netCDF-Java development. The ability to read all the IDD data through the netCDF-Java library allows the TDS to serve that data and provide services on/for that data.
  2. Develop and provide open-source tools for effective use of geoscience data
    Unidata's Integrated Data Viewer (IDV) depends on the netCDF-java library for access to local data, and on the THREDDS Data Server (TDS) for remote access to IDD data. At the same time, the CDM depends on the IDV to validate and test CDM software. Many other tools build on the CDM / netCDF-Java library (eg ERDDAP, Panoply, VERDI, etc) and on the TDS (ESGF, LAS, ncWMS, MyOcean, etc).
  3. Provide cyberinfrastructure leadership in data discovery, access, and use
    The Common Data Model (CDM) / netCDF-Java library is one of the few general-purpose implementations of the CF (Climate and Forecast) metadata standards. Current active efforts in CF that we are involved with include use of the extended netCDF-4 data model (CF 2.0) and for point data (Discrete Sampling Geometry CF-DSG).

    The TDS has pioneered the integration of Open Geospatial Consortium (OGC) protocols into the earth science communities. Strong international collaborations have resulted in WCS and WMS services as part of the TDS.

    The CDM and TDS are widely used implementations of the OPeNDAP DAP2 data access protocol. Unidata has worked with the OPeNDAP group to design, develop, and implement a new version of the DAP specification, DAP4, which is now available in the TDS server and the netCDF-Java client software stack.

Release Status of CDM and TDS

  • CDM 4.5.4 and TDS 4.3.23 are the current stable releases.
  • TDS 4.5 is still beta, since not all features are completely stable, eg FMRC
  • TDS 4.5 is running on Unidata's TDS server (thredds.ucar.edu)
  • TDS 4.6 is available as a Beta release
  • TDS 4.6 is running on Unidata's TDS development server (thredds-dev.unidata.ucar.edu)

Activities Since the Last Status Report

Collaboration Activities

  • Active non-core awards that involve THREDDS development:
  • EarthCube Building Blocks award: "Integrating Discrete and Continuous Data" with Univ of Texas, Austin and others. Period of performances: Oct 2013 - Sept 2015.
  • EarthCube Building Blocks award: "Specifying and Implementing ODSIP, A Data-Service Invocation Protocol" with OPeNDAP, Inc.
  • EarthCube Building Blocks award: "Deploying Web Services Across Multiple Science Domains" with IRIS, UNAVCO, and others. Period of performances: Oct 2013 - Sept 2015.
  • NASA ROSES ACCESS award: "High Performance Multidisciplinary Open Standard Data Services to Serve Terrestrial Environmental Modeling" with USGS CIDA. Period of performances: 1 Jan 2014 - 31 Dec 2015.
  • Two NASA ROSES AIST proposals that involve THREDDS development were not funded:
  • one with SSEC on GPM data
  • the second with PO.DAAC/JPL on polar orbiting satellite data.

CDM / TDS 4.5 Development details

  • OPeNDAP 4.0 protocol TDS server (beta)
  • Improvements to netCDF-4 writing, including
  • writing CF 1.6 DSG (Discrete Sampling Geometry) files.
  • default and user-specified chunking strategies
  • Improvements to the TDS for handling of GRIB files, including
  • two time dimension view for forecast models.
  • scaling to large collections
  • many bug fixes
  • Improvements to the TDS for handling collections of point, station, and sounding data
  • Point Feature Collections use the NCSS interface.
  • Automatic metadata extraction into the THREDDS catalog
  • NcML logicalReduce allows you to remove length 1 dimensions
  • Add WaterML as output format from NCSS point service
  • Ugrade CDM/TDS use of HttpClient from 3.x to 4.x (3.x is end-of-life)
  • Require Java 7 and Tomcat 7 (start using java.nio.file)
  • Updated ncWms and ncIso to latest version

Ongoing Activities

International Standards Development

  • Track and participate in OGC MetOcean Working Group discussion.
  • Track and participate in OGC WCS development.
  • Track and participate in OGC CF-netCDF development.
  • Track and participate in CF Standards development

Cloud Activities

  • Instance of the TDS, a clone of http://thredds.ucar.edu, is up and running in the Microsoft Azure Cloud

  • Investigating use of Docker for deployment of new TDS instances

Collaboration Activities

  • Continuing work with NCAR/CISL DSS group to make their large GRIB model collections available using the new TDS 4.6 GRIB Feature Collections.
  • Working with US IOOS to distribute the ncSOS plug-in with TDS (4.6 or 5.0).
  • Working with Earth System Grid Framework (ESGF) to upgrade TDS functionality for serving CMIPx data.

Planned Activities

CDM / TDS 4.6.X Development

  • Grid Feature Collection version 3
  • scaling to very large collections (again!)
  • make forecast times available on Best datasets
  • best / best complete / analysis as separate datasets
  • Grid Feature Collection version 3
    • scaling to very large collections (again!)
    • make forecast times available on Best datasets
    • best / best complete / analysis as separate datasets
    • File caching improvements
    • Rewrite of client side catalog processing
    • Add WRF initialization service to NCSS
    • Improve the TDS Radar Server services and harmonize it with the NCSS interface.
    • Provide NCSS access for all CF-DSG feature types
    • ncSOS plug-in

    CDM / TDS 5.0 Development

    • Rewrite of server side catalog processing
    • Improvements for ESGF use of TDS
    • Catalog caching improvements
    • Auto tracking of catalog changes
    • Simplify process for adding new TDS web services. Document simplified process so third-party development groups can more easily add new web services to the TDS.
    • Improvements on GRID / SWATH  / IMAGE data types in the CDM

    Areas for Committee Feedback

    • We have requested community input on GRIB collections “Best” Datasets here. Any committee input would be appreciated.        
    • We are expecting to move to using Java 8 soon, as Java 7 will be at End-of-Life in April 2015. Due to security concerns, we have to push our community to adopt Java 8, really for their own good, especially on the server. Thoughts?
    • We would like to restructure of thredds catalog layout. This would not affect IDV bundles, but would would change the way you would explore thredds.ucar.edu. Please see the proposed new layout here, and give us feedback!