[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Streamable NetCDF volumes



Hi Chad:

Streaming netcdf is in our 5-year proposal to NSF as a bullet, under the
strategic goal "Provide cyberinfrastructure leadership in data
discovery, access, and use", where it says

  Planned activities include:

    - Creating a streaming protocol to allow use of netCDF in web
      services, reducing the need to move large datasets to local
      machines.

    - Enhancing the C-based netCDF libraries with client support for new
      remote access protocols such as DAP4, when it is available.

The use in netCDF subsetting by coordinates is also mentioned as
something we will support in the netCDF C-based libraries:

    - Creating C-language netCDF APIs and web services that allow access
      to local and remote data by geospatial coordinates.



We expect to make progress on it in the next 5 years, but exactly
when we will do what is unknown.

Other answers are embedded:


On 8/20/2013 12:28 PM, Chad Trabant wrote:

Hello John,

I work for the IRIS DMC where we manage a large archive of
seismological time series data.  We were recently brainstorming about
using NetCDF as a data delivery format.  We would want to send this
data to users via RESTful web services.   The issue that comes up is
the ability to create NetCDF in a streaming fashion, sending the data
to the user as it is extracted from our archive.  As you are probably
aware, by avoiding the creation of the entire volume prior to sending
to the user we 1) save resources and 2) avoid web service timeouts.

A  bit of searching brought me to the following page and your contact
information:
http://www.unidata.ucar.edu/software/netcdf-java/stream/NcStream.html

 If you do not mind a couple of quick questions:

1) What is the status of support for the ncstream format?  In
particular, will this become a variant supported by standard NetCDF
libraries?

yes


2) Is ncstream only a format for the wire or is this something that
you can save to files for later use?

You can save to a file.


3) Are there alternate efforts to define a streamable NetCDF volume?

not that i know of.


We are not set on creating such an output format, at this point I'm
just trying to get a handle on what developments, if any, are headed
in this direction.

Any information would be appreciated.

regards, Chad Trabant IRIS DMC


Regards,

John and Russ