Re: THREDDS and iRODS - TUES JULY 20 1PM MDT

Hi all:

Some background in anticipation of our call today.

The THREDDS Data Server (TDS) uses the Common Data Model Library (CDM, aka netcdf-Java library) to access local and remote data. The TDS is the web services layer, using servlets. The CDM is a general data access library used in many other applications also. Both are pure java.

The CDM, by default, wants a java.io.RandomAccessFile. We wrap that in a ucar.unidata.io.RandomAccessFile, and also can use subclasses ucar.unidata.io.InMemoryRandomAccessFile and ucar.unidata.io.http.HTTPRandomAccessFile. So one possibility is to provide an iRODS specific subclass of ucar.unidata.io.RandomAccessFile.

Another interesting possibility is the new java.nio.file.spi package in Java 7, which allows one to plug in file system implementations. I suspect it would be a good fit with iRODS. There's a lot of new functionality in Java 7 that we are interested in using. See http://java.sun.com/developer/technicalArticles/javase/nio/ if you havent already been following this. Im guessing we will see a Java 7 release in 6-9 months.

The CDM cant serve arbitrary files, its a subsetting service that needs to understand the details of the file. The list of file formats we currently understand is at:

http://www.unidata.ucar.edu/software/netcdf-java/formats/FileTypes.html

Random access assumes that its efficient to move around in the file and access small chunks of data. Performance depends on file layout and read access patters, both can be hard to predict in some cases.

We have a plug-in architecture for adding new formats. So one aspect of the desirability question is, what holdings are in iRods, and how much work is it to make them readable by the CDM? Also, are the web services appropriate for these files? Who are your clients? TDS is mostly oriented to earth science data.

Mike Conway wrote:

Hi John,

I think one of the basic goals would be an exploration of how we could layer THREDDS on top of the iRODS file system such that the files of whatever format could be served from the data grid.  This is at an exploratory stage, and the basic questions would be on feasibility and desirability.

iRODS does have facilities for file transfer, but I think in the simplest terms it's more about a distributed data grid, or data cloud that can manage distributed data based on policies built into iRODS (replication, metadata extraction, security, federation, etc).

There is a Java library (Jargon) that can integrate Java-based applications with the data grid, and Jargon, among other things, provides an iRODS-specific implementation of the java.io.* libraries.  Could these libraries be plugged in to THREDDS?  That was a primary line of investigation.

Note that THREDDS would see the files as java.io.File, or related streams, but underneath would use the iRODS protocol to expose data stored anywhere on the grid.   Given that that is desirable and feasible, it would allow the data served by THREDDS to gain the distributed and policy-driven management of the iRODS grid.

Regards,

Mike Conway

On Jul 8, 2010, at 8:21 AM, John Caron wrote:

Brian Etherton wrote:

Hi team,

How abouts Tuesday... July 20th?

Brian E.

Hi Brian:

I assume we converged on TUES JULY 20 1PM MDT ?

In preparation, can someone in your group summarize your current thinking about how THREDDS/TDS could be used with iRODS? My limited understanding is that iRODS is oriented towards file transfer, while the TDS is geared towards implementing subsetting protocols like OPeNDAP and WCS/WMS, which requires the ability to parse and understand the semantics in the file. Among other thing, this means that the set of files that can be served by TDS is limited. Any thoughts on that? What goals do you see as possible?

Specific technical background material on iRODS would be welcome also.

John

Mike Conway

DICE Center

Jargon, Java, Interface Developer

address@hidden <mailto:address@hidden>

------------------------------------------------

Google voice/video: address@hidden <mailto:address@hidden>

Skype: michael.c.conway