Rich,
No. I tried this with caching disabled "<scour>-1 hours</scour>" when
I first noticed the problem. But this was not helping. So, I set the
AggregationCache in threddsConfig to the following this past Friday,
yet there is still an hourly scan penalty for many of our datasets
(see attachments), which is overloading our systems.
<AggregationCache>
<scour>24 hours</scour>
<maxAge>90 days</maxAge>
</AggregationCache>
I no longer see the cache triggers in logs/cache.log showing that
files were scrubbed, yet something is still forcing these "static
catalogs" in cache/agg to recheckEvery x=__? value.
Note: the catalogs that I manually specify a "recheckEvery" value
within the aggregation+scan are honoring this and only triggering a
recheck after 2h/6h/12h/etc have passed. All the other datasets that I
do not specify a recheckEvery are by default updating hourly (from the
looks of my nagios checks). Normal response times are ms~sec. When I
see a spike in the response time ~120sec this indicates to me that the
dataset was updating it's cache/agg file - that and the file timestamp
in cache/agg has updated.
/mike
On Mon, Jun 10, 2013 at 1:33 PM, Signell, Richard <rsignell@xxxxxxxx> wrote:
> Do you have settings in threddsConfig that flush the cache every hour? That
> might also cause this behavior.
>
>
> On Friday, June 7, 2013, Michael McDonald wrote:
>>
>> What if there are multiple nested aggregations with a scan locations
>> in a dataset? Where does the "recheckEvery" go, in the outer most
>> aggregation, or in the inner most aggregation?
>>
>> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
>> <aggregation type="union">
>> <netcdf>
>> <aggregation dimName="MT" type="joinExisting" recheckEvery="6 hour">
>> <scan location="/hycom/ftp/datasets/GLBa0.08/expt_90.9/data/"
>> suffix=".????_???_00_2d.nc" subdirs="true" />
>> </aggregation>
>> </netcdf>
>> <netcdf>
>> <aggregation dimName="MT" type="joinExisting" recheckEvery="6 hour">
>> <scan location="/hycom/ftp/datasets/GLBa0.08/expt_90.9/data/"
>> suffix=".????_???_00_3zs.nc" subdirs="true" />
>> </aggregation>
>> </netcdf>
>> </aggregation>
>> </netcdf>
>>
>> I ask this question because it looks as though some of our catalogs
>> are triggering hourly rechecks, even though the "recheckEvery" value
>> is not present in the dataset...
>>
>> Any ideas?
>>
>> _______________________________________________
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
>
>
> --
> Dr. Richard P. Signell (508) 457-2229
> USGS, 384 Woods Hole Rd.
> Woods Hole, MA 02543-1598
Attachment:
Screen Shot 2013-06-10 at 1.58.22 PM.png
Description: PNG image
Attachment:
Screen Shot 2013-06-10 at 1.58.07 PM.png
Description: PNG image
thredds archives: