Hi Niles,
So, quick question - do you expect the valid time in the file
hycom_glb_regp01_old_t000.nc to be before or after the valid time in
hycom_glb_regp01_old_t003.nc?
Here is what I could tell from the actual data in the files:
hycom_glb_regp01_old_t000.nc: time -> 2014-07-08T00:00:00Z (127248.0 hours
since 2000-01-01)
hycom_glb_regp01_old_t003.nc: time -> 2014-07-07T03:00:00Z (127227.0 hours
since 2000-01-01)
hycom_glb_regp01_t000.nc: time -> 2014-07-08T00:00:00Z(127248.0 hours
since 2000-01-01)
hycom_glb_regp01_t003.nc: time -> 2014-07-07T03:00:00Z (127227.0
hours since 2000-01-01)
Therefor, an aggregation listing the order such as:
hycom_glb_regp01_old_t000.nc
hycom_glb_regp01_old_t003.nc
will result in times decreasing in values (2014-07-08T00:00:00Z,
2014-07-07T03:00:00Z) because that's what is in the actual data files.
However, if the expectation is that the valid time in _t003 is after the
valid time in _t000, then things might only look correct if you reverse the
aggregation order.
Sean
On Thu, Oct 6, 2016 at 12:33 PM, Sean Arms <sarms@xxxxxxxx> wrote:
> Hi Niles,
>
> If you specify individual files for aggregation, they will be in the order
> you specify (which is exactly what you have found). If you instead to a
> scan to aggregate the file, the files will be aggregated in lexicographical
> order (for example, the order shown by the command `ls`). This may or may
> not be in the appropriate order. This is one of many tricky things about
> using NcML aggregations. This is also one of the motivating reasons behind
> the concept of featureCollections in the TDS. In a featureCollection
> (Forecast Model Run Collections (FMRC), for example), the coordinate
> systems of all of the files in a collection are examined, and the TDS tries
> to "do the right thing" with them with minimal configuration.
>
> Cheers,
>
> Sean
>
> On Wed, Oct 5, 2016 at 3:32 AM, Niels Charlier <niels@xxxxxxxxx> wrote:
>
>> Following this question, what is going to happen in this regard if you do
>> a <scan> to list the aggregation files? Will they be put in an appropriate
>> order?
>>
>> Thanks for your answer.
>>
>> Regards
>>
>> Niels
>>
>>
>> On 09/30/2016 12:16 PM, Niels Charlier wrote:
>>
>>> Hi Everyone,
>>>
>>> I have changed my mind on this other aggregation issue I discussed in
>>> one of my emails (see below), I said it was unrelated to netcdf-java but it
>>> doesn't seem to be the case. Well, I really want to hear your opinion. The
>>> issue is related to the order of files listed for aggregations.
>>>
>>> I have found that the issue lies in part with geotools, because it is
>>> making certain assumptions about time variables being increasing rather
>>> than decreasing. Convention does demand that a variable is either one of
>>> the two (but not in random order).
>>>
>>> The question is whether the list of files in an .ncml files should also
>>> be in a certain order. It seems that netcdf-java (which does the actual
>>> aggregating of the variables) follows the order from the ncml code. That
>>> means that the values inside the aggregated variable may end up in random
>>> order.
>>>
>>> For example, assume you have two files.
>>> One has time values 1,2,3. The the other one has 4,5,6.
>>>
>>> If we now place them in opposite order in our .ncml file, netcdf-java
>>> appears to create an aggregated variable with values 4,5,6,1,2,3
>>>
>>> In that case, geotools cannot even assume that the extremes are on both
>>> ends of the list, which is IMO faulty and can cause many issues with other
>>> assumptions in the code.
>>>
>>> Have I found something here, or do you disagree? Thank you for
>>> responding.
>>>
>>> Should we just demand from users to place the list of file sin the same
>>> order as the order of values inside the .nc files (increasing/decreasing)?
>>> is that (un)reasonable?
>>>
>>> Kind Regards
>>> Niels
>>>
>>> On 08-09-16 14:51, Niels Charlier wrote:
>>>
>>>> 3. changing the order of the list of files in the inner aggregation
>>>> tags.
>>>> <netcdf location="hycom_glb_regp01_old_t003.nc"/>
>>>> <netcdf location="hycom_glb_regp01_old_t000.nc"/>
>>>> instead of <netcdf location="hycom_glb_regp01_old_t000.nc"/>
>>>> <netcdf location="hycom_glb_regp01_old_t003.nc"/>
>>>> Otherwise the one with the earliest time coordinates is last in the
>>>> list instead of first... and geotools fails on this. Is this normal?
>>>>
>>>> This issue is definitely unrelated to netcdf-java, but I am not sure
>>>> whether this is a bug in geotools or in the .ncml file.
>>>>
>>>
>>> _______________________________________________
>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>> recorded in the Unidata inquiry tracking system and made publicly
>>> available through the web. Users who post to any of the lists we
>>> maintain are reminded to remove any personal information that they
>>> do not want to be made public.
>>>
>>>
>>> netcdf-java mailing list
>>> netcdf-java@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>>
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> netcdf-java mailing list
>> netcdf-java@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>
>
