[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: FW: Catalog generator crawling remote OPenDAP/DODS servers

Hi Bas,
I'm afraid it isn't as easy as getting a working config file. The catGen servlet used to support crawling an OPeNDAP/DODS server but no one was using it and the code (src/thredds/cataloggen/config/DodsDirDatasetSource.java and related files) did not evolve with the rest of the catGen servlet and so it no longer works.
Basically, what DodsDirDataSource did was take a DODS_dir URL (e.g., http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/TOMS/EP/) and scrape the returned HTML for further URLs to crawl. The scraping was done by thredds.cataloggen.config.DodsURLExtractor. Not terribly elegant but it got the job done. One problem is that dods_dir responses are not very standardized so it might need munging for different OPeNDAP servers.
This document might be useful when you start on implementing a CrawlableDataset to deal with dods_dir pages: http://www.unidata.ucar.edu/projects/THREDDS/tech/cataloggen/devel/ 

I hope this is helpful. Sorry the existing code isn't really working.

Let me know if I can be of further assistance.

Ethan

Bas Retsios wrote:

Hello Ethan,
Valentijn Venus asked me to assist him in getting the remote DODS catalog generator running. He forwarded me your recent email communication (see below).
We have spent alot of time trying to accomplish this with the cataloggen servlet, in Thredds 3.10. As far as I understood it is a matter of adding a correct XML file as a new task with http://<servername>:8080/thredds/cataloggen/admin/ , and have "Period(minutes)" > 0.
Unfortunately I do not have a proper working example of an XML file that crawls a remote DODS server. The existing example (under config/examples/cetGenConf.exampleDods.xml) does nothing, perhaps because the links in that file are outdated. "Does nothing" means that after the "Initial Delay", the "Resulting Catalog" is an XML file of 9 lines, which (to my opinion) does not seem to contain any useful information.
Could you please send me a working version of config/examples/cetGenConf.exampleDods.xml that I could try? (I'll also be happy with another example, as long it is about remode DODS data). Or if you suspect there is another problem in our approach, give another hint?
Our intention was to follow your suggestion and transform the DodsDirDataSource into a CrawlableDataset, but first I would like to understand how the original "cataloggen servlet" code works. 

Best regards,

Bas

--
Ir. V. (Bas) Retsios
Software Developer
IT Department, Sector Remote Sensing & GIS
International Institute for Geo-information Science and Earth Observation (ITC)
P.O. Box 6,  7500 AA Enschede, The Netherlands
Phone +31 (0)53 4874 573, telefax +31 (0)53 4874 335
E-mail address@hidden, Internet http://www.itc.nl


Valentijn Venus wrote:

-----Original Message-----
From: Ethan Davis [mailto:address@hidden] Sent: Wednesday, June 07, 2006 12:29 AM 
To: Valentijn Venus
Cc: address@hidden
Subject: Re: Catalog generator crawling remote OPenDAP/DODS servers

Hi Valentijn,

Actually, the CatalogGen servlet is included as part of the TDS. If you
are writing configuration files that include <datasetScan> elements, you
are using the newer TDS configuration. The URLs for this stuff are of
this form: "http://<server>:<port>/thredds/...". If you are writing
configuration files that include <catalogGenConfig> elements, you are
using the old catGen framework. The CatalogGen servlet is at
"http://<server>:<port>/thredds/cataloggen/...".

The 3.8 code (and now the 3.10 code) are available at

ftp://ftp.unidata.ucar.edu/pub/thredds/tmp/

The code for both the old and new style catalog generation are included.

The old implementation for crawling OPeNDAP/DODS servers is in
src/thredds/cataloggen/config/DodsDirDatasetSource.java. In the new
framework, an implementation of
src/thredds/crawlabledataset/CrawlableDataset.java would have to be
written to handle crawling an OPeNDAP/DODS server. The old stuff might
be easy to get up and running quickly but since the plan is to move
everything to the CrawlableDataset framework, I would encourage you to
go that route.

Let me know what you decide.

Ethan

Valentijn Venus wrote:

Thanks for you response. I guess with thredds.war 3.8 i'm using the

newer TDS configuration which also generates catalogs?

Can you send me a pointer to that old code please? Thanks again...

Valentijn

-----Original Message-----
From: Ethan Davis [mailto:address@hidden]
Sent: Mon 6/5/2006 19:58
To: Valentijn Venus
Cc: address@hidden
Subject: Re: Catalog generator crawling remote OPenDAP/DODS servers
Hi Valentijn, 

Valentijn Venus wrote:


Hi Ethan,
First a fundamental questions related to THREDDS: the idea for the catalog generator is to crawl whatever source of data you have (local 


or
remote) and generate a catalog, correct?


That is the idea but unfortunately crawling a local file system is all



that is currently implemented.


Now I'm looking for some
directions on how to explore all sub-directories on a remote DODS server. The following servers I have in mind: 

MODIS AQUA/TERRA:
http://g0dup05u.ecs.nasa.gov/OPENDAP_DP/long_term/MOAA/MYD04_L2.004/2
002
.07.16/
TOMS Ozone:
http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OP
S/T
OMS/EP/
Recently they moved these servers, and since then only the DODS Dataset Access Form for TOMS seems to work. When accessing one of these HDF files (i.e. 
http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OP
S/T
OMS/EP/TOMS-EP_L3-TOMSEPL3_1996m0725_v8.HDF.html) and once the form is setup, I can see the resulting "get"/"post" parameters trailing 
the "?"
in a DODS url (...?Ozone[0:1:179][0:1:287]) How does this work once the catalog is generated, and a client (IDV, Matlab, etc.) accesses it in real-time? I guess THREDDS supports geographic/paramter subsetting in a similair manner, but how does it work technically?
Generally, the THREDDS catalog provides the base DODS URL and the client needs to understand the OPeNDAP/DODS protocol (the ".dds", ".das", ".dods" extensions and all the stuff after the "?"). THREDDS allows for metadata to indicate the geographic coverage of a dataset but it doesn't support subsetting a particular dataset. The actual subsetting all dependent on the protocol being used to access the dataset, in this case OPeNDAP. 


Back to the catalog generation. At one point I did have OPeNDAP server



crawling capabilities in the catalog generator but no one was using it



and I don't think it is still working after a few major modifications.
Are you using the stand-alone CatalogGen application, the catalog generation servlet interface, or the newer TDS configuration which also generates catalogs? The TDS config is a completely different framework than that used by the app and servlet. The code (non-TDS framework) for crawling OPeNDAP servers is still around if you want to
look into getting it running again. However, I'm planning on moving the CatGen app and servlet over to the new TDS framework at some point 


but that may be awhile.

Hope this helps.

Ethan


Cheers, Valentijn




whom they are addressed.


--
Ethan R. Davis                                Telephone: (303) 497-8155
Software Engineer                             Fax:       (303) 497-8690
UCAR Unidata Program Center                   E-mail:    address@hidden
P.O. Box 3000
Boulder, CO  80307-3000                       http://www.unidata.ucar.edu/
---------------------------------------------------------------------------