Re: [netcdf-java] order of aggregation

  • To: Sean Arms <sarms@xxxxxxxx>
  • Subject: Re: [netcdf-java] order of aggregation
  • From: Niels Charlier <niels@xxxxxxxxx>
  • Date: Fri, 7 Oct 2016 09:26:17 +0200
Hi Sean,

So essentially, you're saying that it is the responsibility of the writer of the .ncml writer to put the files in the right order, they can't just specify the files in any order and expect netcdf-java to order them? Okay...

In that case, the issue is only with geotools which is assuming an increasing order (not decreasing).

Regards
Niels

On 06-10-16 21:18, Sean Arms wrote:
Hi Niles,

So, quick question - do you expect the valid time in the file hycom_glb_regp01_old_t000.nc <http://hycom_glb_regp01_old_t000.nc> to be before or after the valid time in hycom_glb_regp01_old_t003.nc <http://hycom_glb_regp01_old_t003.nc>?

Here is what I could tell from the actual data in the files:

hycom_glb_regp01_old_t000.nc <http://hycom_glb_regp01_old_t000.nc>: time -> 2014-07-08T00:00:00Z (127248.0 hours since 2000-01-01) hycom_glb_regp01_old_t003.nc <http://hycom_glb_regp01_old_t003.nc>: time -> 2014-07-07T03:00:00Z (127227.0 hours since 2000-01-01)

hycom_glb_regp01_t000.nc <http://hycom_glb_regp01_t000.nc>: time -> 2014-07-08T00:00:00Z(127248.0 hours since 2000-01-01) hycom_glb_regp01_t003.nc <http://hycom_glb_regp01_t003.nc>: time -> 2014-07-07T03:00:00Z (127227.0 hours since 2000-01-01)

Therefor, an aggregation listing the order such as:

hycom_glb_regp01_old_t000.nc <http://hycom_glb_regp01_old_t000.nc>
hycom_glb_regp01_old_t003.nc <http://hycom_glb_regp01_old_t003.nc>

will result in times decreasing in values (2014-07-08T00:00:00Z, 2014-07-07T03:00:00Z) because that's what is in the actual data files. However, if the expectation is that the valid time in _t003 is after the valid time in _t000, then things might only look correct if you reverse the aggregation order.

Sean



On Thu, Oct 6, 2016 at 12:33 PM, Sean Arms <sarms@xxxxxxxx <mailto:sarms@xxxxxxxx>> wrote:

    Hi Niles,

    If you specify individual files for aggregation, they will be in
    the order you specify (which is exactly what you have found). If
    you instead to a scan to aggregate the file, the files will be
    aggregated in lexicographical order (for example, the order shown
    by the command `ls`). This may or may not be in the appropriate
    order. This is one of many tricky things about using NcML
    aggregations. This is also one of the motivating reasons behind
    the concept of featureCollections in the TDS. In a
    featureCollection (Forecast Model Run Collections (FMRC), for
    example), the coordinate systems of all of the files in a
    collection are examined, and the TDS tries to "do the right thing"
    with them with minimal configuration.

    Cheers,

    Sean

    On Wed, Oct 5, 2016 at 3:32 AM, Niels Charlier <niels@xxxxxxxxx
    <mailto:niels@xxxxxxxxx>> wrote:

        Following this question, what is going to happen in this
        regard if you do a <scan> to list the aggregation files? Will
        they be put in an appropriate order?

        Thanks for your answer.

        Regards

        Niels


        On 09/30/2016 12:16 PM, Niels Charlier wrote:

            Hi Everyone,

            I have changed my mind on this other aggregation issue I
            discussed in one of my emails (see below), I said it was
            unrelated to netcdf-java but it doesn't seem to be the
            case. Well, I really want to hear your opinion. The issue
            is related to the order of files listed for aggregations.

            I have found that the issue lies in part with geotools,
            because it is making certain assumptions about time
            variables being increasing rather than decreasing.
            Convention does demand that a variable is either one of
            the two (but not in random order).

            The question is whether the list of files in an .ncml
            files should also be in a certain order. It seems that
            netcdf-java (which does the actual aggregating of the
            variables) follows the order from the ncml code. That
            means that the values inside the aggregated variable may
            end up in random order.

            For example, assume you have two files.
            One has time values 1,2,3. The the other one has 4,5,6.

            If we now place them in opposite order in our .ncml file,
            netcdf-java appears to create an aggregated variable with
            values 4,5,6,1,2,3

            In that case, geotools cannot even assume that the
            extremes are on both ends of the list, which is IMO faulty
            and can cause many issues with other assumptions in the code.

            Have I found something here, or do you disagree? Thank you
            for responding.

            Should we just demand from users to place the list of file
            sin the same order as the order of values inside the .nc
            files (increasing/decreasing)? is that (un)reasonable?

            Kind Regards
            Niels

            On 08-09-16 14:51, Niels Charlier wrote:

                3. changing the order of the list of files in the
                inner aggregation tags.
                                        <netcdf
                location="hycom_glb_regp01_old_t003.nc
                <http://hycom_glb_regp01_old_t003.nc>"/> <netcdf
                location="hycom_glb_regp01_old_t000.nc
                <http://hycom_glb_regp01_old_t000.nc>"/>
                            instead of  <netcdf
                location="hycom_glb_regp01_old_t000.nc
                <http://hycom_glb_regp01_old_t000.nc>"/> <netcdf
                location="hycom_glb_regp01_old_t003.nc
                <http://hycom_glb_regp01_old_t003.nc>"/>
                   Otherwise the one with the earliest time
                coordinates is last in the list instead of first...
                and geotools fails on this. Is this normal?

                   This issue is definitely unrelated to netcdf-java,
                but I am not sure whether this is a bug in geotools or
                in the .ncml file.


            _______________________________________________
            NOTE: All exchanges posted to Unidata maintained email
            lists are
            recorded in the Unidata inquiry tracking system and made
            publicly
            available through the web.  Users who post to any of the
            lists we
            maintain are reminded to remove any personal information
            that they
            do not want to be made public.


            netcdf-java mailing list
            netcdf-java@xxxxxxxxxxxxxxxx
            <mailto:netcdf-java@xxxxxxxxxxxxxxxx>
            For list information or to unsubscribe, visit:
            http://www.unidata.ucar.edu/mailing_lists/
            <http://www.unidata.ucar.edu/mailing_lists/>


        _______________________________________________
        NOTE: All exchanges posted to Unidata maintained email lists are
        recorded in the Unidata inquiry tracking system and made publicly
        available through the web.  Users who post to any of the lists we
        maintain are reminded to remove any personal information that they
        do not want to be made public.


        netcdf-java mailing list
        netcdf-java@xxxxxxxxxxxxxxxx <mailto:netcdf-java@xxxxxxxxxxxxxxxx>
        For list information or to unsubscribe, visit:
        http://www.unidata.ucar.edu/mailing_lists/
        <http://www.unidata.ucar.edu/mailing_lists/>




  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: