Greetings Joe,
I recently split the netCDF-Java and TDS codebased into their own
repositories, and the repository holding the appropriate TDS code is
located at:
https://github.com/Unidata/tds
If you build the current master branch, you'll have everything you
need at this point. The most recent snapshot should work as well:
https://artifacts.unidata.ucar.edu/repository/unidata-snapshots/edu/ucar/tds/5.0.0-SNAPSHOT/tds-5.0.0-20200308.175757-566.war
(just be sure to rename it to thredds.war before deploying it).
The sample catalog I added to our integration tests for the TDS can be
found here:
https://github.com/Unidata/tds/blob/master/tds/src/test/content/thredds/tds-s3.xml
Cheers,
Sean
On Mon, Mar 9, 2020 at 8:39 AM H. Joe Lee <hyoklee@xxxxxxxxxxxx> wrote:
>
> Thanks, Ethan!
>
> It's so cool to see toolsUI can access NASA HDF-EOS5 on S3.
> I hope both IDV and Panoply can use the new netCDF-Java soon, too.
>
> By the way, will the master branch of THREDDS use the latest netCDF-java?
> If not, where should I modify in the THREDDS source code to build
> THREDDS with netCDF-Java snapshot?
>
> I'm very excited to try the new THREDDS catalog with S3 datasetRoot path!
>
> Sincerely,
>
> --
> Datafy everything in HDF for faster AI.
>
>
>
>
> On Wed, Mar 4, 2020 at 10:52 AM Ethan Davis <edavis@xxxxxxxx> wrote:
>>
>> Hi Joe,
>>
>> [Sorry for the delayed response.]
>>
>> The S3 work moved to the Unidata/netCDF-java repo in PR #173 ("S3 Support").
>> This PR got merged into master a week or so ago and is available in the
>> netCDF-Java 5.3.0-SNAPSHOT release (and will be in the upcoming 5.3.0
>> release). The latest TDS code built with netCDF-Java 5.3.0-SNAPSHOT can be
>> configured to serve an individual netCDF file stored as an S3 object using a
>> datasetRoot configuration, e.g.
>>
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>>
>> <catalog name="Test TDS S3"
>>
>> xmlns="https://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
>>
>> xmlns:xlink="https://www.w3.org/1999/xlink";
>>
>> xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance";
>>
>>
>> xsi:schemaLocation="https://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0
>>
>> https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.6.xsd";>
>>
>>
>> <datasetRoot path="s3-test" location="s3://noaa-goes16" />
>>
>>
>> <dataset name="Test GOES-16 S3" ID="testS3Grid"
>>
>>
>> urlPath="s3-test/ABI-L1b-RadC/2019/363/21/OR_ABI-L1b-RadC-M6C16_G16_s20193632101189_e20193632103574_c20193632104070.nc"
>>
>> dataType="Grid"/>
>>
>>
>> </catalog>
>>
>>
>> In this case, the datasetRoot location is the bucket name, and the urlPath
>> is the datasetRoot path combined with the key. We rely on the AWS Java SDK
>> (v2) to handle credentials, setting of region, etc. For now, you can set the
>> region by creating a credentials file ~/.aws/credentials that looked like:
>>
>>
>> [default]
>>
>> region=us-east-1
>>
>>
>> Which is how netCDF-java knows which region to use for bucket access. We may
>> look at other mechanisms to make that a bit more integrated into TDS
>> configuration but for now that should work.
>>
>>
>> Once the netCDF 5.3.0 release comes out, TDS snapshot builds will be built
>> with this capability. For now, you would need to build the TDS and
>> explicitly tell it to build with netCDF-Java 5.3.0-SNAPSHOT.
>>
>> Cheers,
>>
>> Ethan
>>
>> On Tue, Feb 4, 2020 at 2:30 PM H. Joe Lee <hyoklee@xxxxxxxxxxxx> wrote:
>>>
>>> Hi,
>>>
>>> Is it possible to serve netCDF data on AWS S3 using THREDDS?
>>> I think it seems possible based on the S3 feature branch [1].
>>>
>>> If so, can someone share an example THREDDS catalog configuration?
>>>
>>> Regards,
>>>
>>> [1] https://github.com/Unidata/thredds/tree/feature/s3+hdfs
>>>
>>>
>>> _______________________________________________
>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>> recorded in the Unidata inquiry tracking system and made publicly
>>> available through the web. Users who post to any of the lists we
>>> maintain are reminded to remove any personal information that they
>>> do not want to be made public.
>>>
>>>
>>> thredds mailing list
>>> thredds@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> https://www.unidata.ucar.edu/mailing_lists/
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
