Re: [thredds] How to control caching of Aggregated datasets?

  • To: "McCann, Mike" <mccann@xxxxxxxxx>
  • Subject: Re: [thredds] How to control caching of Aggregated datasets?
  • From: Heiko Klein <Heiko.Klein@xxxxxx>
  • Date: Fri, 11 Mar 2011 09:31:40 +0100
Thanks Mike,

that explains why I couldn't get it working. We could get your 'dir with links and scan' solution working for us, too.

Since the scripts creating the aggregation are written by some colleges, I need to convince them. It would help me if I could get a copy of your discussion with John?

And if it's going to be a long term issue, it might be nice to see a warning in http://www.unidata.ucar.edu/projects/THREDDS/tech/tds4.2/tutorial/NcML.htm and http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html

Best regards,

Heiko

On 2011-03-10 18:05, McCann, Mike wrote:
Hello Heiko,

I was experiencing the same issue and after some discussion with John Caron
I switched my catalog to use a scan aggregation:

   <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
     <aggregation dimName="TIME" type="joinExisting" recheckEvery="2 hour">
       <scan location="<dir>" suffix="_R_TS.nc" subdirs="false" />
     </aggregation>
   </netcdf>

I needed to create a<dir>  with symbolic links to all the files being
aggregated.  The last file in the aggregation is recreated every two hours
and so far this technique has been working fine.

Hope this helps,
Mike


On 3/10/11 8:44 AM, "Heiko Klein"<Heiko.Klein@xxxxxx>  wrote:

Hi,

I have a thredds aggregated dataset, with the setup in catalog.xml:
     <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
      <aggregation dimName="time" type="joinExisting" recheckEvery="10
seconds">
      <netcdf location="/file1.nc" coordValue="1" />
      <netcdf location="/file2.nc" coordValue="2" />
      </aggregation>
     </netcdf>


After changing the aggregation, i.e. by adding a new file, e.g.
file3.nc, or by changing the coordValue, I usually run reinitialize the
catalogues with /thredds/admin/debug?catalogs/reinit

The changes appear in the opendap stream first after a while (some
hours?). The only way I currently found to make the correct results
really appear in opendap is to do the following shortly after each others:
a) clear file object caches
b) delete the corresponding aggregation cache file from the disk.
(And I'm not sure if this allways works?)

Is there a better way to control aggregation changes in the catalog.xml
file, i.e. to flush any aggregation-caches when the aggregation changes?


Best regards,

Heiko

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit:
http://www.unidata.ucar.edu/mailing_lists/


--
Dr. Heiko Klein                              Tel. + 47 22 96 32 58
Development Section / IT Department          Fax. + 47 22 69 63 55
Norwegian Meteorological Institute           http://www.met.no
P.O. Box 43 Blindern  0313 Oslo NORWAY



  • 2011 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: