Re: [netcdf-java] runtime aggregation

Ben, Sean,
I am also including the geotools list in the discussion, because it regards both libraries.
I got the aggregation file to work. Apart from some obvious bug fixes and additions to support the "runtime" dimension in geotools, I had to make a few changes to the xml file. 
What is still unclear to me is whether these changes
- are actually required to work (i.e. the aggregation file is faulty). in that case we need to work on better error reporting in geotools (possibly also in netcdf-java) - should not actually be required, in which case we need to fix these as bugs in either geotools or netcdf-java
I have included the .ncml file in the attachment to demonstrate. There are three such changes I made: 

1. the Conventions global attribute,as discussed before.
2. removal of the "time" variable in the aggregation (see commented line in attached .ncml file)
This issue is related to the previous one, it is again about /makeCoordinateSystemsImplicit/ versus /makeCoordinateSystemsMaximal/.
* If the "time" variable is included as aggregation variable, it also gets the "runtime" dimension added to its dimensions. 
    * As a consequence, netcdf-java does not recognise it as an AXIS.
* As a consequence, /makeCoordinateSystemsImplicit /fails to include it in the CRS's for the actual variables (water_u, salinity, etc...). * As a consequence, /makeCoordinateSystemsMaximal /creates a CRS for these variables. However, it puts the dimensions in the order of the /dimension variable/ list (rather than the variable's own dimensions), which is completely arbitrary as far as I can see. * "runtime" ends up as the last axis instead of the first. This is inconsistent with the order of dimensions in the variable. Geotools fails on this inconsistency.
I am not sure whether the fault here lies with the .ncml file, geotools or netcdf-java. 


3. changing the order of the list of files in the inner aggregation tags.

                       <netcdf location="hycom_glb_regp01_old_t003.nc"/>  <netcdf 
location="hycom_glb_regp01_old_t000.nc"/>
           instead of  <netcdf location="hycom_glb_regp01_old_t000.nc"/>  <netcdf 
location="hycom_glb_regp01_old_t003.nc"/>
Otherwise the one with the earliest time coordinates is last in the list instead of first... and geotools fails on this. Is this normal?
This issue is definitely unrelated to netcdf-java, but I am not sure whether this is a bug in geotools or in the .ncml file. 


Regards
Niels


On 07-09-16 13:23, Niels Charlier wrote:

Ben, Sean,
Changing the Convention has resolved this issue. Thanks both for your tips.
I am now dealing with other issues in the same file... will keep you updated. 

Kind Regards
Niels

On 02-09-16 03:00, Ben Caradoc-Davies wrote:

Niels,
one thing you could try is to change the Conventions global attribute of these files to "CF-1.6" to see if this changes the way they are handled (if these files are close enough to CF).
Sean, do you expect aggregations to behave the same as single files with the same Conventions? It looks like Niels has identified a difference. 

Kind regards,
Ben.

On 30/08/16 03:29, Sean Arms wrote:

Greetings Niels,
The class DefaultConvention is netCDF-Java's way of trying to handle cases in which a convention is not specified. It tries to be pretty forgiving in 
terms of files where people have tried to be compliant with some
convention, but didn't set a convention attribute, or knew enough about
some conventions in general to sort of get things in the ball park...for example, setting positive=up on a vertical coordinate, but not getting the rest of the stuff correct (not true for your case, but our code tries to 
handle the more common case we come across). If a file is actually
following CF, netCDF-Java will use the class CF1Convention. Currently we 
handle (or try to) around 30 specific conventions (not sure all are
actually documented conventions), but when all else fails, we go to the
default convention.
NcML aggregations in general can get pretty gnarly, and are very hard to get correct. Have you looked into using the Forecast Model Run Collection 
(FMRC) featureCollection element to serve these data?
https://www.unidata.ucar.edu/software/thredds/current/tds/tutorial/FmrcFeatureCollectionsTutorial.html 

Cheers,

Sean
On Mon, Aug 29, 2016 at 6:11 AM, Niels Charlier <niels@xxxxxxxxx> wrote: 


Hello,

I am currently working on the netcdf extensions of geotools/geoserver,
although I should say I am fairly new at netcdf, so I am still learning. I am currently trying to get some particular datasets to work with geoserver, and I think I might have hit against some limitations, perhaps even a bug, 
in netcdf-java, so I wanted to ask you for advice.
We have a .ncml have that aggregate datasets based on 'runtime'. This is 
an explanation from the client:
*"'runtime' is not a first class coordinate in the data produced by our 
production groups, so the initial approach to solve this using unidata
mechanisms is to use NCML to aggregate *
*two datasets with different runtimes together using a 'joinNew'
aggregation, which adds a new coordinate axis 'runtime' to the dataset * 
*(this takes place in several of the .ncml files included in the test
data). The runtime axis would then be exposed as a custom dimension in geoserver so that it can be used for subsets. This means that a reader would have to be prepared to handle dimension values that are possibly two dimensional(a set of values for each available 'runtime'), which I don't 
think the current reader can handle? "*
My first question is what it would entail for building support for this in netcdf-java? In the meantime I have been extensively debugging what happens 
when I try to load the aggregation file, and I can give some detailed
information about what goes wrong in the code. I think I might have found 
something else that is going wrong.

* as part of the aggregation, the runtime dimension is being added to
every other variable of each of the .nc files.
* During *CoordSysBuilder.buildCoordinateSystems* some things happen as a 
consequence of that that is different from when I read the .nc files
individually:

      -> In *makeCoordinateSystemsImplicit*, as a consequence of all
variables having an additional dimension, the CRS with lacking runtime
dimension is not considered sufficient and is not being set.
-> However, in *makeCoordinateSystemsMaximal* the CRS's are being 
set, except for one variable called "surf_el", because it is being
considered an AXIS (see line 873,  call to* !vp.isData()*).

* During index building (*NetCDFImageReader.init -> initIndex ->
getCoordinateSystem*), a runtime exception is thrown because surf_el
doesn't have a CRS.
The thing is, surf_el is not an axis. The reason it is considered an axis is because it has a "positive" attribute. This happens in *DefaultConvention.findCoordinateAxes -> getAxisType -> getAxisTypeCoards* (see line 297, *positive != null*). 

To quote my colleague Ben who I asked advice about this:
*"surf_el is a time-varying data variable and not any kind of coordinate. Surface elevation looks to me like an output of the HYCOM ocean model **<https://hycom.org/> 
<https://hycom.org/>**. Any logic that says that a variable is a
coordinate if it has attribute "positive"="up" is broken. There is this 
statement in the CF conventions: *

*"A vertical coordinate will be identifiable by: *
*    units of pressure; or *
*    the presence of the positive attribute with a value of up or down
(case insensitive)." *
*http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html
<http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html>*

*There are such things as time-varying multidimensional coordinate
variables (think of the lat and lon grids for a satellite swathe), but this 
is not one of these. *
*I think that, to be a coordinate variable, a variable must be mentioned in the "coordinates" attribute of another variable in the same file, or have a name that matches its dimension (i.e. a basic single-dimensional 
coordinate variable). " *


So have I indeed identified a bug here? please advice.


Thank you for your help.


Kind Regards,
Niels




_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


netcdf-java mailing list
netcdf-java@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/





_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


netcdf-java mailing list
netcdf-java@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/ 





_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


netcdf-java mailing list
netcdf-java@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/ 



<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>

    <attribute name="Conventions" type="String" value="CF-1.6" />

    <variable name="runtime" type="double">
        <attribute type="string" name="_CoordinateAxisType" value="RunTime"/>
        <attribute type="string" name="units" value="hours since 2014-07-07 
00:00:00 UTC"/>
        <attribute type="string" name="time_origin" value="2014-07-07 
00:00:00"/>
        <values>0 24</values>
    </variable>
    <aggregation type="joinNew" dimName="runtime">
        <netcdf coordValue="0">
            <aggregation type="joinExisting" dimName="time">
                <netcdf location="hycom_glb_regp01_old_t003.nc"/>
                <netcdf location="hycom_glb_regp01_old_t000.nc"/>
            </aggregation>
        </netcdf>
        <netcdf coordValue="24">
            <aggregation type="joinExisting" dimName="time">
                <netcdf location="hycom_glb_regp01_t003.nc"/>
                <netcdf location="hycom_glb_regp01_t000.nc"/>
            </aggregation>
        </netcdf>
        <variableAgg name="water_u"/>
        <variableAgg name="salinity"/>
        <variableAgg name="water_v"/>
        <variableAgg name="water_temp"/>
        <variableAgg name="tau"/>
        <variableAgg name="surf_el"/>
        <!-- <variableAgg name="time"/> -->
    </aggregation>
</netcdf>
  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: