> On Jul 14, 2015, at 12:37 PM, Signell, Richard <rsignell@xxxxxxxx> wrote: > > Folks, > I discovered some info here on OPeNDAP and S3 from the hyrax boyz, in > case of interest: > http://docs.opendap.org/index.php/OPULS:_NOAA_S3_Data_Access And here’s a short paper we wrote on it:
Attachment:
NOAA_cloud_9.6.13.pdf
Description: Adobe PDF document
James PS. after the work on S3 we also expanded the code to use Glacier - a whole different kettle of fish. > > On Tue, Jul 14, 2015 at 3:35 PM, Robert Casey <rob@xxxxxxxxxxxxxxxxxxx> wrote: >> >> Hi Jeff- >> >> Of note, Amazon Glacier is meant for infrequently needed data, so a call-up >> for data from that source will require something on the order of a 5 hour >> wait to retrieve to S3. I think they are developing a near-line storage >> solution that is a bit more expensive to compete with Google's new near-line >> storage, which provides retrieval times on the order of seconds. >> >> -Rob >> >> On Jul 14, 2015, at 10:10 AM, Jeff McWhirter <jeff.mcwhirter@xxxxxxxxx> >> wrote: >> >> On this note - >> What I really want is a file system that can transparently manage data >> between primary (SSD), secondary (S3) and tertiary (Amazon Glacier) stores. >> Actively used data would migrate into primary storage. Old archived data >> moves off into cheaper tertiary storage. I've thought of implementing this >> at the application level in RAMADDA but a file system based approach would >> be much smarter. >> >> How do the archive folks on this list manage these kinds of storage >> environments? >> >> -Jeff >> >> >> >> >> On Tue, Jul 14, 2015 at 10:44 AM, John Caron <caron@xxxxxxxx> wrote: >>> >>> Hi David: >>> >>> At the bottom of the TDM, we rely on RandomAccessFile. Do you know if S3 >>> supports that abstraction (essentially posix file semantics, eg seek(), >>> read()) ? My guess is that S3 only allows complete file transfers (?) >>> >>> Would be worth investigating if anyone has implemented a java >>> FileSystemProvider for S3. >>> >>> Will have a closer look when i get time. >>> >>> John >>> >>> On Mon, Jul 13, 2015 at 7:59 PM, David Nahodil <David.Nahodil@xxxxxxxxxxx> >>> wrote: >>>> >>>> Hi all, >>>> >>>> >>>> We are looking at moving our THREDDS Data Server to Amazon EC2 instances >>>> with the data hosted on S3. I'm just wondering if anyone has tried using >>>> TDS >>>> with data hosted on S3? >>>> >>>> >>>> I had a quick back-and-forth with Sean at Unidata (see below) about this. >>>> >>>> >>>> Regards, >>>> >>>> >>>> David >>>> >>>> >>>>>> Unfortunately, I do not know of anyone who has done this, although we >>>>>> have had at lease one other person ask. From what I understand, there is >>>>>> a >>>>>> way to mount an S3 storage as a virtual file system, in which case I >>>>>> would >>>>>> *think* that the TDS would work as it normally does (depending on the >>>>>> kind >>>>>> of data you have). >>>> >>>> >>>>> We have considered mounting the S3 storage as a filesystem and running >>>>> it like that. However, our feeling was that the tools were not really >>>>> production ready and that we're really misrepresenting S3 by pretending it >>>>> is a file system. So this is why we're investigating if anyone has used >>>>> TDS >>>>> with the S3 API directly. >>>> >>>> >>>>>> What kind of data do you have? Will your TDS also be in the cloud? Do >>>>>> you plan on serving the data inside of amazon to other EC2 instances, or >>>>>> do >>>>>> you plan on crossing the cloud/commodity web boundary with the data, in >>>>>> which case that could get very expensive quite quickly? >>>> >>>> >>>>> We have about 2 terabytes of marine and climate data that we are >>>>> currently serving from our existing infrastructure. The plan is to move >>>>> the >>>>> infrastructure to Amazon Web Services so TDS would be hosted on EC2 >>>>> machines >>>>> and the data on S3. We're hoping this setup should work okay, but we might >>>>> still have a hurdle or two to come. :) >>>> >>>> >>>>> We have someone here who once wrote a plugin/adapter for TDS to work >>>>> with an obscure filesystem that our data used to be stored on. So we have >>>>> a >>>>> little experience in what might be involved in what might be involved for >>>>> doing the same with S3. We just wanted to make sure that if anyone had >>>>> done >>>>> some work already that we made use of that. >>>> >>>> >>>>>> We very, very recently (as in a day ago) got some Amazon resources to >>>>>> play around on, but we won't have a chance to kick those tires until >>>>>> after >>>>>> our training workshops at the end of the month. >>>> >>>> >>>> >>>> University of Tasmania Electronic Communications Policy (December, 2014). >>>> This email is confidential, and is for the intended recipient only. >>>> Access, disclosure, copying, distribution, or reliance on any of it by >>>> anyone outside the intended recipient organisation is prohibited and may be >>>> a criminal offence. Please delete if obtained in error and email >>>> confirmation to the sender. The views expressed in this email are not >>>> necessarily the views of the University of Tasmania, unless clearly >>>> intended >>>> otherwise. >>>> >>>> >>>> _______________________________________________ >>>> thredds mailing list >>>> thredds@xxxxxxxxxxxxxxxx >>>> For list information or to unsubscribe, visit: >>>> http://www.unidata.ucar.edu/mailing_lists/ >>> >>> >>> >>> _______________________________________________ >>> thredds mailing list >>> thredds@xxxxxxxxxxxxxxxx >>> For list information or to unsubscribe, visit: >>> http://www.unidata.ucar.edu/mailing_lists/ >> >> >> _______________________________________________ >> thredds mailing list >> thredds@xxxxxxxxxxxxxxxx >> For list information or to unsubscribe, visit: >> http://www.unidata.ucar.edu/mailing_lists/ >> >> >> >> _______________________________________________ >> thredds mailing list >> thredds@xxxxxxxxxxxxxxxx >> For list information or to unsubscribe, visit: >> http://www.unidata.ucar.edu/mailing_lists/ > > > > -- > Dr. Richard P. Signell (508) 457-2229 > USGS, 384 Woods Hole Rd. > Woods Hole, MA 02543-1598 > > _______________________________________________ > thredds mailing list > thredds@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ -- James Gallagher jgallagher at opendap.org 406.723.8663
Attachment:
signature.asc
Description: Message signed with OpenPGP using GPGMail
thredds archives: