Re: [thredds] Problems with large aggregations on JPL THREDDS?

  • To: "Signell, Richard" <rsignell@xxxxxxxx>
  • Subject: Re: [thredds] Problems with large aggregations on JPL THREDDS?
  • From: Roy Mendelssohn - NOAA Federal <roy.mendelssohn@xxxxxxxx>
  • Date: Sun, 17 Feb 2013 09:15:44 -0800
That is my understanding also  (I was just sort of hoping), so can we call it a 
"feature request"  (though the transition will mean a lot of recoding for me!).

-Roy
On Feb 17, 2013, at 8:32 AM, "Signell, Richard" <rsignell@xxxxxxxx> wrote:

> Roy,
> 
> From this page:
> http://www.unidata.ucar.edu/projects/THREDDS/tech/tds4.3/reference/collections/FeatureCollections.html
> 
> It looks like the only "featureType" values currently allowed in a
> FeatureCollection are
> TDS 4.2: FMRC, Point, and Station
> TDS 4.3: FMRC, Point,  Station and GRIB
> 
> We've been doing FeatureCollection FMRC featureTypes with netcdf files
> and it works fine.
> 
> But I don't think you can use FeatureCollection for joinNew or
> joinExisting aggregations of netcdf files yet.
> 
> -Rich
> 
> On Sun, Feb 17, 2013 at 10:26 AM, Roy Mendelssohn - NOAA Federal
> <roy.mendelssohn@xxxxxxxx> wrote:
>> Hi Rich:
>> On Feb 17, 2013, at 4:10 AM, "Signell, Richard" <rsignell@xxxxxxxx> wrote:
>> 
>>> "If you have joinExisting Aggregations, coordinate information will be
>>> written to a cache directory....Every scour amount of time, any item
>>> that hasnt been changed since maxAge time will be deleted. Set scour
>>> to -1 to not scour if you have aggregations that never change."
>>> 
>>> The OSCAR dataset is not changing, so JPL could modify their
>>> threddsConfig.xml to be:
>>> 
>>> ...
>>> <AggregationCache>
>>> ...
>>> <scour>-1 hours</scour>
>>> ...
>>> </AggregationCache>
>>> 
>>> I'm cc'ing the THREDDS list as it seems a common issue.
>> 
>> But then the scour value is set for all datatsets.  I do not know of a way 
>> to set dataset specific rates.  We run into this problem a lot, though not 
>> as extreme as the example given.  Recently I have had to start setting up 
>> featureCollections for some GRIB data, and that has a lot of desired 
>> properties - you can set collection specific times, and you can have an 
>> external process force the update.   But, I have not had any success using 
>> featureCollections for just regular gridded netcdf datasets to replace the 
>> older dataset aggregation.  I have seen others on this list imply that they 
>> haven't quite been able to get it working either.
>> 
>> If someone is using featureCollection to aggregate gridded netcdf files  
>> (not GRIB files) can they post the URL and include a copy of the relevant 
>> portions of the catalog.xml?
>> 
>> Thanks,
>> 
>> -Roy M.
>> 
>> 
>> **********************
>> "The contents of this message do not reflect any position of the U.S. 
>> Government or NOAA."
>> **********************
>> Roy Mendelssohn
>> Supervisory Operations Research Analyst
>> NOAA/NMFS
>> Environmental Research Division
>> Southwest Fisheries Science Center
>> 1352 Lighthouse Avenue
>> Pacific Grove, CA 93950-2097
>> 
>> e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
>> voice: (831)-648-9029
>> fax: (831)-648-8440
>> www: http://www.pfeg.noaa.gov/
>> 
>> "Old age and treachery will overcome youth and skill."
>> "From those who have been given much, much will be expected"
>> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>> 
> 
> 
> 
> -- 
> Dr. Richard P. Signell   (508) 457-2229
> USGS, 384 Woods Hole Rd.
> Woods Hole, MA 02543-1598

**********************
"The contents of this message do not reflect any position of the U.S. 
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" 
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.



  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: