Re: [thredds] Problem between OPeNDAP and TDS when netCDF file is modified

Hi Roland-

On 5/7/12 8:42 AM, Roland Schweitzer wrote:
An obvious test case would be a copy of a file before the update happens
and a copy of the same file just after the update happens.  With those
two files John's group could simulate the update easily.  :-)

I can save those off for today. However, for these aggregations, even though we are in May, the last dates for 2012 that show up in the aggregations are from January and February - even with tomcat restarts. There is no rhyme nor reason that we can find on what dates show up for each aggregation.

I'll wait to hear from Unidata on what information they need or if they want to come visit. Thanks to the community for pushing for a resolution to this issue.

Don

On Sun, May 6, 2012 at 10:04 AM, Don Murray <don.murray@xxxxxxxx
<mailto:don.murray@xxxxxxxx>> wrote:

    John-


    On 5/5/12 4:56 PM, John Caron wrote:

        We have someone working on trying to reproduce this problem. If
        we can
        reproduce it, we can fix it. Any help doing that would be
        appreciated.


    Since I'm the one who updates the dataset, here's what happens.  The
    SST datasets consist of yearly files which we are trying to
    aggregate into one dataset.  The latest file (or files depending if
    we are crossing a year boundary) is updated on a daily basis and the
    last 14 days of data are overwritten and a new timestep (day) is
    added. We do more than just add a timestep because the last 14 days
    of data are preliminary and subject to updates. The file(s) being
    updated are then synced out to the filesystem that TDS looks at.

    Hoop has said he would provide any catalog/configuration information
    you require.  If it would speed up the process, perhaps someone from
    Unidata could drive across town and visit NOAA.

    Don


        On 5/4/2012 5:41 PM, Roy Mendelssohn wrote:

            Hi All:

            I have been corresponding with Hoop off list, because the
            datasets he
            is having problems serving are important to NOAA, and
            getting them
            served properly is important. His problem is:

                    On 05/02/2012 04:28 PM, Hoop wrote:

                        All,

                        This is my latest in a now monthly series of
                        requests for help with
                        doing aggregations with our TDS. The problem I
                        first reported back
                        on 23 February, wherein aggregations don't
                        notice time steps added
                        to the final file in the time series, is
                        unresolved.. Since I last
                        wrote (4 April), we upgraded to 4.2.10. There
                        was no effect that we
                        could discern. Whether we use NcML or
                        FeatureCollection, new time
                        steps in the final file go unnoticed until
                        Tomcat is restarted.
                        Fabulously, if a new file is added without
                        restarting Tomcat, the
                        initial time steps in the new final file are
                        added to the aggregation,
                        leaving a gap where the time steps added to the
                        previous "final" file
                        since the last Tomcat restart should be. This
                        leads to complaints of
                        the aggregation not being CF-compliant, since it
                        appears to have
                        uneven spacing in time.

                        Interestingly, doing the aggregation in RAMADDA
                        works as we would
                        expect, since it is frequently rebuilding the
                        aggregation. So, while
                        it is perhaps less efficient than TDS, at least
                        it is reliable.

                        -Hoop


            I just checked with my group, and we have several aggregated
            datasets
            that work this way. We have tons of datasets where each file
            is a new
            time period, and they get updated just fine. But the ones
            where we are
            aggregating over files with multiple time periods, and one
            of those
            files is being updated with new data, the aggregation only gets
            updated by restarting tomcat. This has not been a major
            issue because
            our datasets are updated monthly, so we do it once and all
            is good.
            But if we were updating regularly, this would be a major
            issue, as it
            appears to be for Hoop's group. Mainly I am writing for
            three reasons:

            1. This problem is not isolated to Hoop's group or how they
            are setup.

            2. If anyone has this kind of aggregation updating
            successfully, can
            they point to the URL, post the catalog.xml and
            threddsConfig.xml, and
            perhaps sample files so we can test it and see what is
            different.

            3. If successful examples can't be found that we can build
            on, is
            there anyway this can elevated in importance at Unidata? I
            know there
            is a lot being worked on, and priorities need to be set, but
            if a
            solution can't be found given the present code base, it
            strikes me as
            a serious bug.

            Thanks

            -Roy



            **********************
            "The contents of this message do not reflect any position of
            the U.S.
            Government or NOAA."
            **********************
            Roy Mendelssohn
            Supervisory Operations Research Analyst
            NOAA/NMFS
            Environmental Research Division
            Southwest Fisheries Science Center
            1352 Lighthouse Avenue
            Pacific Grove, CA 93950-2097

            e-mail: Roy.Mendelssohn@xxxxxxxx
            <mailto:Roy.Mendelssohn@xxxxxxxx> (Note new e-mail address)
            voice: (831)-648-9029 <tel:%28831%29-648-9029>
            fax: (831)-648-8440 <tel:%28831%29-648-8440>
            www: http://www.pfeg.noaa.gov/

            "Old age and treachery will overcome youth and skill."
            "From those who have been given much, much will be expected"
            "the arc of the moral universe is long, but it bends toward
            justice"
            -MLK Jr.

            _________________________________________________
            thredds mailing list
            thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
            For list information or to unsubscribe, visit:
            http://www.unidata.ucar.edu/__mailing_lists/
            <http://www.unidata.ucar.edu/mailing_lists/>


        _________________________________________________
        thredds mailing list
        thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
        For list information or to unsubscribe, visit:
        http://www.unidata.ucar.edu/__mailing_lists/
        <http://www.unidata.ucar.edu/mailing_lists/>


    --
    Don Murray
    NOAA/ESRL/PSD and CIRES
    303-497-3596 <tel:303-497-3596>
    http://www.esrl.noaa.gov/psd/__people/don.murray/
    <http://www.esrl.noaa.gov/psd/people/don.murray/>


    _________________________________________________
    thredds mailing list
    thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
    For list information or to unsubscribe,  visit:
    http://www.unidata.ucar.edu/__mailing_lists/
    <http://www.unidata.ucar.edu/mailing_lists/>



--
Don Murray
NOAA/ESRL/PSD and CIRES
303-497-3596
http://www.esrl.noaa.gov/psd/people/don.murray/