[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Catalog generator crawling remote OPenDAP/DODS servers



Hi Valentijn,

Valentijn Venus wrote:

Hi Ethan,

First a fundamental questions related to THREDDS: the idea for the
catalog generator is to crawl whatever source of data you have (local or
remote) and generate a catalog, correct?

That is the idea but unfortunately crawling a local file system is all that is currently implemented.

Now I'm looking for some
directions on how to explore all sub-directories on a remote DODS
server. The following servers I have in mind:

MODIS AQUA/TERRA:
http://g0dup05u.ecs.nasa.gov/OPENDAP_DP/long_term/MOAA/MYD04_L2.004/2002
.07.16/
TOMS Ozone:
http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/T
OMS/EP/
Recently they moved these servers, and since then only the DODS Dataset
Access Form for TOMS seems to work. When accessing one of these HDF
files (i.e.
http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/T
OMS/EP/TOMS-EP_L3-TOMSEPL3_1996m0725_v8.HDF.html) and once the form is
setup, I can see the resulting "get"/"post" parameters trailing the "?"
in a DODS url (...?Ozone[0:1:179][0:1:287]) How does this work once the
catalog is generated, and a client (IDV, Matlab, etc.) accesses it in
real-time? I guess THREDDS supports geographic/paramter subsetting in a
similair manner, but how does it work technically?
Generally, the THREDDS catalog provides the base DODS URL and the client needs to understand the OPeNDAP/DODS protocol (the ".dds", ".das", ".dods" extensions and all the stuff after the "?"). THREDDS allows for metadata to indicate the geographic coverage of a dataset but it doesn't support subsetting a particular dataset. The actual subsetting all dependent on the protocol being used to access the dataset, in this case OPeNDAP.


Back to the catalog generation. At one point I did have OPeNDAP server crawling capabilities in the catalog generator but no one was using it and I don't think it is still working after a few major modifications. Are you using the stand-alone CatalogGen application, the catalog generation servlet interface, or the newer TDS configuration which also generates catalogs? The TDS config is a completely different framework than that used by the app and servlet. The code (non-TDS framework) for crawling OPeNDAP servers is still around if you want to look into getting it running again. However, I'm planning on moving the CatGen app and servlet over to the new TDS framework at some point but that may be awhile.

Hope this helps.

Ethan

Cheers, Valentijn

--
Ethan R. Davis                                Telephone: (303) 497-8155
Software Engineer                             Fax:       (303) 497-8690
UCAR Unidata Program Center                   E-mail:    address@hidden
P.O. Box 3000
Boulder, CO  80307-3000                       http://www.unidata.ucar.edu/
---------------------------------------------------------------------------