Re: [netcdf-java] compound attributes in NetCDF4 w/ NetCDF-java

Ethan,

Thanks for the pointers.  I'm now creating CDM compliant station observation 
files using the CF specification.

I do have a fix for supporting attributes with primitive arrays (char[]) and 
strings in compound types.  Is there a public SVN server with the NetCDF-java 
source?  It would be nice to be able to work against the latest source and 
provide patches against SVN.

I did try using the CF conventions agains NetCDF-4 w/ complex types and it's 
almost there.  Being able to do this would-be-nice but not required.  I guess 
the advantage would be storing all values for a particular station observation 
contiguously. 

Here's an example of the CDL I tried, with the patch referenced above it loads 
in ToolsUI with _CoordinateAxisType attributes correctly assigned.  It will 
fail when opened as a FeatureType as there are some issues determining 
relationships (i.e. finding observations.time dimension across structure 
boundary, need to query dimension of observations structure, not time field).  

## START
netcdf gsod {
types:
  compound observation_type {
    int time ;
    float tempMean ;
    float tempMin ;
    float tempMax ;
  }; // observation_type
  compound observation_att {
    string time ;
    string tempMean ;
    string tempMin ;
    string tempMax ;
  }; // observation_att
dimensions:
        station = 1 ;
        observation = UNLIMITED ; // (1 currently)
variables:
        int station_id(station) ;
                station_id:standard_name = "station_id";
        float latitude(station) ;
                latitude:units = "degrees_north" ;
        float longitude(station) ;
                longitude:units = "degrees_east" ;
        float elevation(station) ;
                elevation:units = "feet" ;
                elevation:positive = "up" ;
        observation_type observations(station, observation) ;
            observations:coordinates = "observations.time longitude latitude 
elevation" ;
                observation_att observations:units = {"days since 1929-1-1 
0:0:0", "degF", "degF", "degF"} ;
                
// global attributes:
                :Conventions = "CF-1.5" ;
                :CF\:featureType = "stationTimeSeries" ;
} 
## END

Tom Kunicki
Center for Integrated Data Analytics
U.S. Geological Survey
8505 Research Way
Middleton, WI  53562





On Feb 11, 2010, at 5:57 PM, Ethan Davis wrote:

> Hi Tom,
> 
> I'm going to leave the netCDF-4 compound type specific questions for
> John Caron who wrote that part of netCDF-Java (he should be back in the
> office on Tuesday).
> 
> As for a CDM compliant station file, are you following the CF proposed
> specification for station data:
> 
>> https://cf-pcmdi.llnl.gov/trac/wiki/PointObservationConventions
> 
> Ah, just noticed your "cdm_datatype" global attribute, so it looks like
> you are following the older (now deprecated) Unidata Obs Dataset Convention:
> 
>> http://www.unidata.ucar.edu/software/netcdf-java/formats/UnidataObsConvention.html
> 
> The netCDF-Java library should understand station datasets written to
> either convention. However, both conventions are written to the netCDF
> classic data model rather than the enhanced (netCDF-4) data model. So,
> though it may recognize it as a station dataset, I'm not sure (even once
> the bug you are seeing is fixed) whether it will really know how to deal
> with the data as station data.
> 
> Have you considered using the netCDF-4 classic model format? It has many
> of the advantages of netCDF-4 (e.g., compression and chunking) without
> breaking backward compatibility with programs written to the netCDF-3
> API (though they do need to be re-linked to the netCDF-4 library). Also,
> at this point many netCDF-4 aware programs don't have general support
> for parts of the enhanced data model (e.g., user defined types).
> Sticking to the classic model avoids this problem as well. It also has
> the advantage of working with conventions that are written to the
> classic model (all current conventions I'm aware of).
> 
> There is more information on this in the netCDF FAQ section on the
> various formats, data models, and version:
> 
>> http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#formats-and-versions
> 
> There is also more detail on the advantages of the netCDF-4 classic
> model format here:
> 
>> http://www.unidata.ucar.edu/software/netcdf/papers/nc4_conventions.html#Use_of_the_NetCDF_Classic_Mode
> 
> Hope that helps,
> 
> Ethan
> 
> On 2/8/2010 11:33 AM, Tom Kunicki wrote:
>> 
>> Hi,
>> 
>> I'm having a problem parsing a NetCDF4 file in NetCDF-java.
>> 
>> I've using NetCDF 4.1 (C library with Java JNI bindings) in an attempt to 
>> produce a CDM compliant Station file in NetCDF4 format.  I want to put each 
>> stations observation in a compound field.  The file generates and parses 
>> using the ncgen and ncdump but I can't open it in NetCDF-Java 4.1.  I looks 
>> like there's trouble reading char[] or string fields in the compound 
>> attribute type.  The following stack trace is produced on read:
>> 
>> java.lang.NullPointerException
>>        at ucar.ma2.ArrayStructure.extractMemberArray(ArrayStructure.java:407)
>>        at ucar.nc2.iosp.hdf5.H5header.makeAttributes(H5header.java:587)
>>        at ucar.nc2.iosp.hdf5.H5header.makeVariable(H5header.java:1034)
>>        at ucar.nc2.iosp.hdf5.H5header.makeNetcdfGroup(H5header.java:388)
>>        at ucar.nc2.iosp.hdf5.H5header.read(H5header.java:182)
>>        at ucar.nc2.iosp.hdf5.H5iosp.open(H5iosp.java:108)
>>        at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1325)
>>        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:761)
>>        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:395)
>>        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:365)
>>        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:352)
>>        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:340)
>>        at edu.unidata.ucar.jni.NetCDFTest.main(NetCDFTest.java:147)
>> 
>> DIgging through the source it looks like 
>> ucar.nc2.iosp.hdf5.H5header.getNCtype(int hdfType, int size) is returning 
>> null for a char[] compound field type when trying to map the attributes to 
>> the compound fields.
>> 
>> With this be supported in the future?  Is there another way to assign 
>> attributes to compound type fields?
>> 
>> Here's the CDL I am trying to use (the observation dim will eventually be 
>> UNLIMITED):
>> 
>> netcdf gsod {
>> types:
>>  compound observation_type {
>>    int time ;
>>    float tempMean ;
>>    float tempMin ;
>>    float tempMax ;
>>  }; // observation_type
>>  compound observation_atts {
>>    char time(32) ;
>>    char tempMean(32) ;
>>    char tempMin(32) ;
>>    char tempMax(32) ;
>>  }; // observation_atts
>> dimensions:
>>      station = 2 ;
>>      observation = 3 ;
>> variables:
>>      int station_id(station) ;
>>      float station_description(station) ;
>>      float latitude(station) ;
>>              latitude:units = "degrees_north" ;
>>      float longitude(station) ;
>>              longitude:units = "degrees_east" ;
>>      float altitude(station) ;
>>              altitude:units = "feet" ;
>>              altitude:positive = "up" ;
>>      observation_type station_observations(station, observation) ;
>>              observation_atts station_observations:units = {{"days since 
>> 1930-01-01 00 UTC"}, {"degF"}, {"degF"}, {"degF"}} ;
>> 
>> // global attributes:
>>              :cdm_datatype = "Stations" ;
>>              :stationDimension = "station" ;
>>              :observationDimension = "observation" ;
>>              :latitude_coordinate = "latitude" ;
>>              :longitude_coordinate = "longitude" ;
>>              :zaxis_coordinate = "altitude" ;
>>              :time_coordinate = "station_observations.time" ;
>> data:
>> 
>> station_id = _, _ ;
>> 
>> station_description = _, _ ;
>> 
>> latitude = _, _ ;
>> 
>> longitude = _, _ ;
>> 
>> altitude = _, _ ;
>> 
>> station_observations =
>>  {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0},
>>  {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0} ;
>> }
>> 
>> 
>> Thanks,
>> 
>> Tom Kunicki
>> Center for Integrated Data Analytics
>> U.S. Geological Survey
>> 8505 Research Way
>> Middleton, WI  53562
> 
> _______________________________________________
> netcdf-java mailing list
> netcdf-java@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit: 
> http://www.unidata.ucar.edu/mailing_lists/ 



  • 2010 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: