Hi Bas,
Sorry again for the delay. I've been out of the office a fair amount
this summer.
Bas Retsios wrote:
So far so good, I got something working. It was a matter of getting
the catalog.xml file right.
At this moment, I get the job done, but I would like you to check if
this is the optimal way.
Glad to hear things are working.
The java code of CrawlableDatasetDods did not change up to now (it is
the same as the one I sent you on June 23rd).
What I changed was the datasetScan and the service sections of my
catalog.xml file. This is what I needed in order to access DODS server
http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/TOMS/EP/
catalog.xml:
============================
...
<service name="remoteopendap" serviceType="OpenDAP"
base="http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/TOMS/"/>
...
...
<datasetScan name="DodsTest" path="EP"
location="http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/TOMS/EP";
ID="nasaTest" addDatasetSize="true" addLatest="true">
<crawlableDatasetImpl
className="thredds.crawlabledataset.CrawlableDatasetDods" />
<metadata inherited="true">
<serviceName>remoteopendap</serviceName>
</metadata>
</datasetScan>
...
============================
Some comments:
1. I see almost the same URL twice in this catalog.xml file: once at
<service> and once at <datasetScan>
2. base + path = location, otherwise it does not work
I was wondering if it is possible to avoid having 2 times the same
URL. Unfortunately something is checking if "base" is a parent of
"location".
I don't believe this is possible at the moment. There are a number of
checks to insure consistency between various locations but the "base"
and "location" aren't being directly compared. In the file based
version, the "base" and "location" are very different. The issue is that
you are dealing with datasets that you aren't serving with the TDS so
the "base" and the "location" have to overlap to a large degree.
One of the reasons for all these issues is that our main design goal was
to allow the server to access local and remote datasets so that the
server could then catalog the data services it was providing for those
datasets. Your case is cataloging the data services provided by a remote
dataset. I certainly was planning on this working and thankfully it
seems to be working for you. However, it wasn't really one of our use
cases so we didn't test it thoroughly or get to think through all the
little design issue that use case would have brought up. I'll have to
think about whether there are some nice ways to make your use case a
little more elegant.
Furthermore, "path" is the same for my local thredds server and the
remote server. In my case, path="EP", thus an example of valid local
and remote URLs is:
http://thredds.itc.nl:8080/thredds/catalog/EP/catalog.html?dataset=nasaTest/TOMS-EP_L3-TOMSEPL3_1996m0802_v8.HDF
http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/TOMS/EP/TOMS-EP_L3-TOMSEPL3_1996m0802_v8.HDF.html
My problem with this is that the string "EP" originates from the
remote URL. I was forced to put it as "path" into catalog.xml. As a
result, "EP" became part of my local server URLs, while I liked to
call it "testDods".
I think you can change the "EP" in your URLs to "testDods" by adding
"EP/" to "base" and changing "path" to "testDods". One issue here would
be that, if you currently have multiple datasetsScans referencing the
same service, this will require multiple service elements each with a
slightly different "base".
Well, what I have mentioned up to now does work, but I need to edit my
catalog.xml file and restart the server everytime I need to include a
new DODS server.
If you have remote configuration setup on your server (see
http://motherlode.ucar.edu:8080/thredds/docs/RemoteManagement.html), you
can reload the catalog.xml file without restarting Tomcat or the TDS.
You just go to the debug page on your server
(http://hostname:port/thredds/debug) and select the "Reinitialize" link.
Ideally I would like to have a location on my server that I can use to
convert any DODS url to a catalog. A location could be e.g.:
http://thredds.itc.nl:8080/thredds/catalog/dodsToCatalog/
Then after /dodsToCatalog/ I could append any remote URL, for which my
server can make a catalog. Is this possible?
That's a great idea but I'm afraid we don't have anything like that at
the moment.
Hope this helps. Let me know if you have any other issues or thoughts on
how this all works.
Thanks,
Ethan
Cheers,
Bas.
--
Ethan R. Davis Telephone: (303) 497-8155
Software Engineer Fax: (303) 497-8690
UCAR Unidata Program Center E-mail: edavis@xxxxxxxx
P.O. Box 3000
Boulder, CO 80307-3000 http://www.unidata.ucar.edu/
---------------------------------------------------------------------------
