Thanks, Ethan!
It's so cool to see toolsUI can access NASA HDF-EOS5 on S3.
I hope both IDV and Panoply can use the new netCDF-Java soon, too.
By the way, will the master branch of THREDDS use the latest netCDF-java?
If not, where should I modify in the THREDDS source code to build
THREDDS with netCDF-Java snapshot?
I'm very excited to try the new THREDDS catalog with S3 datasetRoot path!
Sincerely,
--
Datafy everything in HDF for faster AI.
On Wed, Mar 4, 2020 at 10:52 AM Ethan Davis <edavis@xxxxxxxx> wrote:
> Hi Joe,
>
> [Sorry for the delayed response.]
>
> The S3 work moved to the Unidata/netCDF-java repo in PR #173
> <https://github.com/Unidata/netcdf-java/pull/173> ("S3 Support"). This PR
> got merged into master a week or so ago and is available in the netCDF-Java
> 5.3.0-SNAPSHOT release (and will be in the upcoming 5.3.0 release). The
> latest TDS code built with netCDF-Java 5.3.0-SNAPSHOT can be configured to
> serve an individual netCDF file stored as an S3 object using a datasetRoot
> configuration, e.g.
>
> <?xml version="1.0" encoding="UTF-8"?>
>
> <catalog name="Test TDS S3"
>
> xmlns="https://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
>
> xmlns:xlink="https://www.w3.org/1999/xlink";
>
> xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance";
>
> xsi:schemaLocation="
> https://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0
>
> https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.6.xsd";>
>
> <datasetRoot path="s3-test" location="s3://noaa-goes16" />
>
> <dataset name="Test GOES-16 S3" ID="testS3Grid"
>
>
>
> urlPath="s3-test/ABI-L1b-RadC/2019/363/21/OR_ABI-L1b-RadC-M6C16_G16_s20193632101189_e20193632103574_c20193632104070.nc"
>
> dataType="Grid"/>
>
> </catalog>
>
> In this case, the datasetRoot location is the bucket name, and the urlPath
> is the datasetRoot path combined with the key. We rely on the AWS Java SDK
> (v2) to handle credentials, setting of region, etc. For now, you can set
> the region by creating a credentials file ~/.aws/credentials that looked
> like:
>
> [default]
>
> region=us-east-1
>
> Which is how netCDF-java knows which region to use for bucket access. We
> may look at other mechanisms to make that a bit more integrated into TDS
> configuration but for now that should work.
>
> Once the netCDF 5.3.0 release comes out, TDS snapshot builds will be built
> with this capability. For now, you would need to build the TDS and
> explicitly tell it to build with netCDF-Java 5.3.0-SNAPSHOT.
>
> Cheers,
>
> Ethan
>
> On Tue, Feb 4, 2020 at 2:30 PM H. Joe Lee <hyoklee@xxxxxxxxxxxx> wrote:
>
>> Hi,
>>
>> Is it possible to serve netCDF data on AWS S3 using THREDDS?
>> I think it seems possible based on the S3 feature branch [1].
>>
>> If so, can someone share an example THREDDS catalog configuration?
>>
>> Regards,
>>
>> [1] https://github.com/Unidata/thredds/tree/feature/s3+hdfs
>>
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> https://www.unidata.ucar.edu/mailing_lists/
>>
>
