Hi Ted,
On Oct 24, 2009, at 10:02 AM, Ted Habermann wrote:
Peter et al.,
I think John C. was correct that this could be done using the
standard web logs. I suspect that a solution build on the standard
logs, rather that extra logging built in to a server, would be most
useful. I could not tell if that was what you were suggesting...
Sorry for the lack of clarity in my message. I am indeed suggesting
using the standard logs. We tried the specialized logs a while ago and
that was a flop. We messed around parsing our web logs a while ago and
there is a lot of work to extract the different OPeNDAP responses -
das, dds,... - so the idea here is to send all the "standard" web logs
to one place and have them generate stats for these logs in which they
handle the OPeNDAP specific info the same way. The analyzer would
still have to handle the specifics of the web site - which URLs
correspond to which data sets,... but about 1/2 (or maybe
substantially more) of the work would be the same, thus reducing a lot
of work for the data providers AND the stats generated would be more
uniform across data provider sites. Finally, all the logs would exist
in one data base so other interesting things could be done with them
such as looking at regions covered by the data across a whole lot of
servers.
Peter
Ted
Peter Cornillon wrote:
On Oct 24, 2009, at 8:18 AM, John Caron wrote:
Peter Cornillon wrote:
Hi Rich,
I was at a NASA meeting this week in which we were discussing
metrics for NASA related projects. As a result of the discussion,
we decided to experiment with a metrics group at Goddard. What I
want to do is to develop a general evaluation of http logs from
OPeNDAP enabled servers. The idea would be to generate a
consistent set of stats from these servers. The job of sorting
out stats can be divided into two parts, one part that is OPeNDAP
specific - how many das, dds,... accesses, how many data
accesses, what the volume of the data access was,... The second
part is provider specific, what constitutes a dataset at that
site. We will be experimenting with a Hyrax and a GDS server, but
I'm assuming that much of the OPeNDAP specific info would be the
same for a TDS. In addition to these one would probably want to
know how many WCS accesses or other accesses were made for a TDS.
Assuming that we make good progress on the servers that we will
be messing around with, it might make sense to extend the project
in the not too distant future to TDS. Comments?
Peter
Hi Peter:
That would be very welcome.
The TDS uses standard apache log format, but we also advise to
append the amount of (wall clock) time it took to process the
request. Its up to the individual user to do this, however. Im not
sure what Hyrax/GDS uses as a log format.
The TDS uses a very regular URL naming schema that makes it
straightforward to figure out the service used. extracting the
dataset name is slightly more complicated. I can send some code
snippets if anyone is interested.
I am sure that this will be of interest when/if we get to that
point. My guess is that it will take a month or so to work out an
approach based on the Hyrax and GDS logs. Once we have done that,
assuming that Goddard is interested in pursuing this on a larger
scale, we can look at TDS logs. If Goddard is not interested in
pursuing this for the community at large, it may be worth seeking
funding to do so. I think that this would provide a very
interesting data set with which to explore use of the DAP. Thanks
for your interest John.
Peter
--
Peter Cornillon
215 South Ferry Road
Telephone: (401) 874-6283
Graduate School of Oceanography Fax:
(401) 874-6283
University of Rhode Island
Internet: pcornillon@xxxxxxxxxxx <mailto:pcornillon@xxxxxxxxxxx>
Narragansett, RI 02882 USA
------------------------------------------------------------------------
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
==== Ted Habermann ===========================
Enterprise Data Systems Group Leader
NOAA, National Geophysical Data Center
V: 303.497.6472 F: 303.497.6513
"If you want to go quickly, go alone.
If you want to go far, go together"
Old Proverb
==== Ted.Habermann@xxxxxxxx ==================
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
Peter Cornillon
215 South Ferry Road Telephone:
(401) 874-6283
Graduate School of Oceanography Fax:
(401) 874-6283
University of Rhode Island
Internet: pcornillon@xxxxxxxxxxx
Narragansett, RI 02882 USA
