Re: [netcdf-java] runtime aggregation

  • To: Niels Charlier <niels@xxxxxxxxx>
  • Subject: Re: [netcdf-java] runtime aggregation
  • From: Sean Arms <sarms@xxxxxxxx>
  • Date: Thu, 6 Oct 2016 13:57:05 -0600
Hi Niles,

In general, netCDF-Java can handle 2D time coordinates (runtime, time).

CF conventions, which you are using due to the conventions attribute you
added, explains the case of 2D lat/lon variables, but as far as I can tell,
not a two dimensional time coordinate variable. Following CF, a 2D lat/lon
variable is recognized as a coordinate variable if another variable lists
it in its "coordinate" attribute. See
http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#idp5559280
for an example. Since a 2D time coordinate was not explicitly mentioned,
netCDF-Java appears to have chosen to use the same approach, that is,
somewhere in the dataset, the 2D time coordinate must be listed in someones
"coordinate" attribute to be picked up as a coordinate variable. For your
case, it is not listed explicitly in a coordinate attribute, so it's not
being detected as a coordinate variable.

If you want to aggregate using a 2D time (that is, keep time as an
aggregation variable), then you need to add the appropriate coordinate
attribute to each aggregation variable that will use the 2D time. So, for
example,

water_u, salinity, water_v, and water_temp will need the attribute
"coordinates: runtime time depth lat lon", and surf_el will need the
attribute "coordinates: runtime time lat lon"

Circling back to Ben's question which made me feel a bit uneasy, he
asked "Sean,
do you expect aggregations to behave the same as single files with the same
Conventions? It looks like Niels has identified a difference." to which I
responded yes, I expect them to behave the same. However, what was
happening here is that time was being upped to a 2D coordinate in the
aggregation, which then added a new condition for it to be recognized as a
coordinate variable - that is, was it explicitly listed in someones
coordinate attribute? In this case, it wasn't explicitly listed in a
coordinate attribute, so it was interpreted as a non-coordinate variable in
the aggregated dataset. Now, one could argue that maybe netCDF-Java should
automagically recognize that time was a 1D coordinate before it was
aggregated to 2D, and thus should automatically be treated as a 2D
coordinate variable without being listed in another variables coordinate
attribute, but I think it's better to force things to be explicit and allow
the flexibility for this not to happen. Also, I am not sure the aggregation
code tracks things that closely, so automatically doing this would likely
slow down aggregations even further. Yuck.

Cheers,

Sean


On Wed, Oct 5, 2016 at 3:35 AM, Niels Charlier <niels@xxxxxxxxx> wrote:

> Sean, Ben,
>
> Isn't issue #2 related to the lack of support for two-dimension coordinate
> variables that the client talked about with regards to the runtime
> aggregations?
>
> Sorry if I am stating the obvious here, I have only been learning while
> working on this.
>
> I did not fully comprehended the relationship between the two issues at
> first. But reading back my description of what goes wrong in the code, it
> all starts with the time variable not being recognised as an axis because
> of the runtime dimension being added to it as a second dimension.
>
> Still, the aggregation works fine if I leave out the time variable. I am
> still not fully understanding how its presence as "variableAgg" affects the
> end result. I don't see any examples online where axis variables are
> included as variableAgg. This isn't exactly clear to me. Please enlighten
> me on that point.
>
> Kind Regards
>
> Niels
>
> On 09/08/2016 02:51 PM, Niels Charlier wrote:
>
> 2. removal of the "time" variable in the aggregation (see commented line
> in attached .ncml file)
>
>     This issue is related to the previous one, it is again about
> *makeCoordinateSystemsImplicit* versus *makeCoordinateSystemsMaximal*.
>
>     * If the "time" variable is included as aggregation variable, it also
> gets the "runtime" dimension added to its dimensions.
>     * As a consequence, netcdf-java does not recognise it as an AXIS.
>     * As a consequence, *makeCoordinateSystemsImplicit *fails to include
> it in the CRS's for the actual variables (water_u, salinity, etc...).
>     * As a consequence, *makeCoordinateSystemsMaximal *creates a CRS for
> these variables. However, it puts the dimensions in the order of the 
> *dimension
> variable* list
>        (rather than the variable's own dimensions), which is completely
> arbitrary as far as I can see.
>     * "runtime" ends up as the last axis instead of the first. This is
> inconsistent with the order of dimensions in the variable. Geotools fails
> on this inconsistency.
>
>     I am not sure whether the fault here lies with the .ncml file,
> geotools or netcdf-java.
>
>
>
  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: