Jon Blower wrote:
Thanks John - just to clarify, at the moment files are only removed if
the number of elements is exceeded, and their age in the cache is
disregarded?
correct
Out of interest, what was the main motivation for creating the cache
system, i.e. what is the expensive operation that you are optimizing?
1. The IOSP builds the entire NetcdfFile object each time its opened,
rather than building only on demand. Netcdf-3 files are quite fast, but
other IOSPs can be much slower. Recently weve been considering a "fast
open" option, but it may be hard to implement.
2. The OS file handle must be opened each time.
Caching apparently reduces average latency by about 30% (perhaps more).
Not sure about throughput or server resource usage.
I have been mulling over all these pieces for a while, thinking about
what a better system would do. There are certain cases, like
aggregation, where some kind of caching is probably necessary for
performance. Arguably maybe not for single files.
Anyway, you might try running without caching. If you are able to get
any timings for the two cases (with and without), I would be very
interested.
Cheers, Jon
On Mon, Sep 28, 2009 at 5:57 PM, John Caron <caron@xxxxxxxxxxxxxxxx> wrote:
Jon Blower wrote:
Dear all (esp. John),
I think I have been misunderstanding the intended behaviour of the
cache of NetCDF files (the one that's initialized with
NetcdfDataset.initNetcdfFileCache()).
What is the behaviour of the "periodic cleanup"? Does it remove
elements from memory only when the number of elements in memory
exceeds a certain number? Or does it also remove elements based on
their age?
every n minutes it removes files using LRU.
The behaviour I would like is the latter - i.e. I would like to be
able to guarantee that the elements in the cache are no older than,
say, five minutes. But this isn't the behaviour I'm seeing: elements
seem to persist essentially for ever. So perhaps the former behaviour
is the correct one? In which case, how would you recommend I
implement the latter behaviour?
ill probably move to ehcache (version 4.2?), which i think allows this.
ill have to look at the code for a workaround...
(Rich and Clare - I think this might be causing the caching problems
you are seeing with ncWMS.)
Thanks,
Jon
