[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
Re: Updating Aggregations in TDS
- Subject: Re: Updating Aggregations in TDS
- Date: Thu, 30 Oct 2008 09:37:08 -0600
Hi Roy:
Sorry for taking so long:
Roy Mendelssohn wrote:
Hi All:
We have a question as to exactly how the update of aggregation of files
occurs in concert with the recheckEvery setting. Some background will
help. We update a lot of aggregated datasets quite often, but not at
exactly regular times (say roughly hourly but all over the map). When
a new files has been added, here is a place where a flag is set that
says this is the case. Some of our programs read this, and therefore
expect the new file to be included in the time aggregation. However,
when programs go to the TDS we find that the actual update only occurs
after we make the new request.
Am I correct that this is the way that the aggregation (with caching)
works in concert with recheckEvery:
1. The first time the dataset is requested, the dimensional information
is read and is stored in cache.
2. Nothing happens again, regardless of the value in recheckEvery and
what is in the directory, until a new request is made on that dataset.
3. If a new request is made, and the recheckEvery time has not been
exceeded, then the cached values are used.
4. If a new request is made, and the recheckEvery time has been
exceeded, then the directory is rescanned.
yes
Is this a correct understanding of the process? If so, where our
problem occurs is that some of our directories have lots of files, so
the rescan takes awhile and it can take 2-3 requests to get the new
values showing up.
So my question is there a way to have the directory rescanned when we
have added files, other than sending a request? If we reinitialize the
TDS from the Tomcat manager through a command, would that do it (which
strikes me as overkill since the catalogs themselves are not changing).
basically you want to do the recheck in the background, and then accept
whatever is there when a request comes in? Or you want to detect whenever the
directory changes?
Thanks,
-Roy
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097
e-mail: address@hidden (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"