John Cartwright wrote:
Hello All,
has anyone used TDS for asynchronous data access, e.g. data in a tape
library that might take minutes to an hour to retrieve?
Hi John,
After my RAMADDA talk last week I spoke with someone from NGDC about
just that and that has gotten me thinking about this issue.
I have actually implemented something like this in RAMADDA for remote
files in remote FTP sites:
http://motherlode.ucar.edu/repository/entry/show/RAMADDA/RAMADDA+Examples/FTP+Sites?entryid=4d396a2a-7b09-4e3b-b3ad-165af41026b7
For the FTP service the user specifies a size threshold (e.g., 10MB).
When a service in RAMADDA (e.g., opendap) requires a file the FTP
service looks in its local cache. If not there and if its below the
size threshold then the file is copied over from the FTP server and
stored in the cache.
One of the problems with going to a tape store is the latency in
fetching the file. But, that is only suffered the first time or when
the file is flushed out of the cache.
I've also lately been thinking about integrating with cloud storage
(e.g., amazon s3). This would work just like the FTP service. You have
a local cache and the smarts to fetch it from the cloud storage.
Another approach to accessing cloud storage would be to do it at the
file system level not at the application level, e.g. FUSE:
http://code.google.com/p/s3fs/
-Jeff
