I have offered to provide sample sets that we have also, and sent John the ftp
link that Hoop sent me. Hopefully they can see what is going on and figure out
the necessary corrections.
-Roy
On May 6, 2012, at 8:04 AM, Don Murray wrote:
> John-
>
> On 5/5/12 4:56 PM, John Caron wrote:
>> We have someone working on trying to reproduce this problem. If we can
>> reproduce it, we can fix it. Any help doing that would be appreciated.
>
> Since I'm the one who updates the dataset, here's what happens. The SST
> datasets consist of yearly files which we are trying to aggregate into one
> dataset. The latest file (or files depending if we are crossing a year
> boundary) is updated on a daily basis and the last 14 days of data are
> overwritten and a new timestep (day) is added. We do more than just add a
> timestep because the last 14 days of data are preliminary and subject to
> updates. The file(s) being updated are then synced out to the filesystem that
> TDS looks at.
>
> Hoop has said he would provide any catalog/configuration information you
> require. If it would speed up the process, perhaps someone from Unidata
> could drive across town and visit NOAA.
>
> Don
>
>> On 5/4/2012 5:41 PM, Roy Mendelssohn wrote:
>>> Hi All:
>>>
>>> I have been corresponding with Hoop off list, because the datasets he
>>> is having problems serving are important to NOAA, and getting them
>>> served properly is important. His problem is:
>>>
>>>>> On 05/02/2012 04:28 PM, Hoop wrote:
>>>>>> All,
>>>>>>
>>>>>> This is my latest in a now monthly series of requests for help with
>>>>>> doing aggregations with our TDS. The problem I first reported back
>>>>>> on 23 February, wherein aggregations don't notice time steps added
>>>>>> to the final file in the time series, is unresolved.. Since I last
>>>>>> wrote (4 April), we upgraded to 4.2.10. There was no effect that we
>>>>>> could discern. Whether we use NcML or FeatureCollection, new time
>>>>>> steps in the final file go unnoticed until Tomcat is restarted.
>>>>>> Fabulously, if a new file is added without restarting Tomcat, the
>>>>>> initial time steps in the new final file are added to the aggregation,
>>>>>> leaving a gap where the time steps added to the previous "final" file
>>>>>> since the last Tomcat restart should be. This leads to complaints of
>>>>>> the aggregation not being CF-compliant, since it appears to have
>>>>>> uneven spacing in time.
>>>>>>
>>>>>> Interestingly, doing the aggregation in RAMADDA works as we would
>>>>>> expect, since it is frequently rebuilding the aggregation. So, while
>>>>>> it is perhaps less efficient than TDS, at least it is reliable.
>>>>>>
>>>>>> -Hoop
>>>>>>
>>>
>>> I just checked with my group, and we have several aggregated datasets
>>> that work this way. We have tons of datasets where each file is a new
>>> time period, and they get updated just fine. But the ones where we are
>>> aggregating over files with multiple time periods, and one of those
>>> files is being updated with new data, the aggregation only gets
>>> updated by restarting tomcat. This has not been a major issue because
>>> our datasets are updated monthly, so we do it once and all is good.
>>> But if we were updating regularly, this would be a major issue, as it
>>> appears to be for Hoop's group. Mainly I am writing for three reasons:
>>>
>>> 1. This problem is not isolated to Hoop's group or how they are setup.
>>>
>>> 2. If anyone has this kind of aggregation updating successfully, can
>>> they point to the URL, post the catalog.xml and threddsConfig.xml, and
>>> perhaps sample files so we can test it and see what is different.
>>>
>>> 3. If successful examples can't be found that we can build on, is
>>> there anyway this can elevated in importance at Unidata? I know there
>>> is a lot being worked on, and priorities need to be set, but if a
>>> solution can't be found given the present code base, it strikes me as
>>> a serious bug.
>>>
>>> Thanks
>>>
>>> -Roy
>>>
>>>
>>>
>>> **********************
>>> "The contents of this message do not reflect any position of the U.S.
>>> Government or NOAA."
>>> **********************
>>> Roy Mendelssohn
>>> Supervisory Operations Research Analyst
>>> NOAA/NMFS
>>> Environmental Research Division
>>> Southwest Fisheries Science Center
>>> 1352 Lighthouse Avenue
>>> Pacific Grove, CA 93950-2097
>>>
>>> e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
>>> voice: (831)-648-9029
>>> fax: (831)-648-8440
>>> www: http://www.pfeg.noaa.gov/
>>>
>>> "Old age and treachery will overcome youth and skill."
>>> "From those who have been given much, much will be expected"
>>> "the arc of the moral universe is long, but it bends toward justice"
>>> -MLK Jr.
>>>
>>> _______________________________________________
>>> thredds mailing list
>>> thredds@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>
>> _______________________________________________
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
> --
> Don Murray
> NOAA/ESRL/PSD and CIRES
> 303-497-3596
> http://www.esrl.noaa.gov/psd/people/don.murray/
>
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097
e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
