Ethan,
Thanks for the pointers. I'm now creating CDM compliant station observation
files using the CF specification.
I do have a fix for supporting attributes with primitive arrays (char[]) and
strings in compound types. Is there a public SVN server with the NetCDF-java
source? It would be nice to be able to work against the latest source and
provide patches against SVN.
I did try using the CF conventions agains NetCDF-4 w/ complex types and it's
almost there. Being able to do this would-be-nice but not required. I guess
the advantage would be storing all values for a particular station observation
contiguously.
Here's an example of the CDL I tried, with the patch referenced above it loads
in ToolsUI with _CoordinateAxisType attributes correctly assigned. It will
fail when opened as a FeatureType as there are some issues determining
relationships (i.e. finding observations.time dimension across structure
boundary, need to query dimension of observations structure, not time field).
## START
netcdf gsod {
types:
compound observation_type {
int time ;
float tempMean ;
float tempMin ;
float tempMax ;
}; // observation_type
compound observation_att {
string time ;
string tempMean ;
string tempMin ;
string tempMax ;
}; // observation_att
dimensions:
station = 1 ;
observation = UNLIMITED ; // (1 currently)
variables:
int station_id(station) ;
station_id:standard_name = "station_id";
float latitude(station) ;
latitude:units = "degrees_north" ;
float longitude(station) ;
longitude:units = "degrees_east" ;
float elevation(station) ;
elevation:units = "feet" ;
elevation:positive = "up" ;
observation_type observations(station, observation) ;
observations:coordinates = "observations.time longitude latitude
elevation" ;
observation_att observations:units = {"days since 1929-1-1
0:0:0", "degF", "degF", "degF"} ;
// global attributes:
:Conventions = "CF-1.5" ;
:CF\:featureType = "stationTimeSeries" ;
}
## END
Tom Kunicki
Center for Integrated Data Analytics
U.S. Geological Survey
8505 Research Way
Middleton, WI 53562
On Feb 11, 2010, at 5:57 PM, Ethan Davis wrote:
> Hi Tom,
>
> I'm going to leave the netCDF-4 compound type specific questions for
> John Caron who wrote that part of netCDF-Java (he should be back in the
> office on Tuesday).
>
> As for a CDM compliant station file, are you following the CF proposed
> specification for station data:
>
>> https://cf-pcmdi.llnl.gov/trac/wiki/PointObservationConventions
>
> Ah, just noticed your "cdm_datatype" global attribute, so it looks like
> you are following the older (now deprecated) Unidata Obs Dataset Convention:
>
>> http://www.unidata.ucar.edu/software/netcdf-java/formats/UnidataObsConvention.html
>
> The netCDF-Java library should understand station datasets written to
> either convention. However, both conventions are written to the netCDF
> classic data model rather than the enhanced (netCDF-4) data model. So,
> though it may recognize it as a station dataset, I'm not sure (even once
> the bug you are seeing is fixed) whether it will really know how to deal
> with the data as station data.
>
> Have you considered using the netCDF-4 classic model format? It has many
> of the advantages of netCDF-4 (e.g., compression and chunking) without
> breaking backward compatibility with programs written to the netCDF-3
> API (though they do need to be re-linked to the netCDF-4 library). Also,
> at this point many netCDF-4 aware programs don't have general support
> for parts of the enhanced data model (e.g., user defined types).
> Sticking to the classic model avoids this problem as well. It also has
> the advantage of working with conventions that are written to the
> classic model (all current conventions I'm aware of).
>
> There is more information on this in the netCDF FAQ section on the
> various formats, data models, and version:
>
>> http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#formats-and-versions
>
> There is also more detail on the advantages of the netCDF-4 classic
> model format here:
>
>> http://www.unidata.ucar.edu/software/netcdf/papers/nc4_conventions.html#Use_of_the_NetCDF_Classic_Mode
>
> Hope that helps,
>
> Ethan
>
> On 2/8/2010 11:33 AM, Tom Kunicki wrote:
>>
>> Hi,
>>
>> I'm having a problem parsing a NetCDF4 file in NetCDF-java.
>>
>> I've using NetCDF 4.1 (C library with Java JNI bindings) in an attempt to
>> produce a CDM compliant Station file in NetCDF4 format. I want to put each
>> stations observation in a compound field. The file generates and parses
>> using the ncgen and ncdump but I can't open it in NetCDF-Java 4.1. I looks
>> like there's trouble reading char[] or string fields in the compound
>> attribute type. The following stack trace is produced on read:
>>
>> java.lang.NullPointerException
>> at ucar.ma2.ArrayStructure.extractMemberArray(ArrayStructure.java:407)
>> at ucar.nc2.iosp.hdf5.H5header.makeAttributes(H5header.java:587)
>> at ucar.nc2.iosp.hdf5.H5header.makeVariable(H5header.java:1034)
>> at ucar.nc2.iosp.hdf5.H5header.makeNetcdfGroup(H5header.java:388)
>> at ucar.nc2.iosp.hdf5.H5header.read(H5header.java:182)
>> at ucar.nc2.iosp.hdf5.H5iosp.open(H5iosp.java:108)
>> at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1325)
>> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:761)
>> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:395)
>> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:365)
>> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:352)
>> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:340)
>> at edu.unidata.ucar.jni.NetCDFTest.main(NetCDFTest.java:147)
>>
>> DIgging through the source it looks like
>> ucar.nc2.iosp.hdf5.H5header.getNCtype(int hdfType, int size) is returning
>> null for a char[] compound field type when trying to map the attributes to
>> the compound fields.
>>
>> With this be supported in the future? Is there another way to assign
>> attributes to compound type fields?
>>
>> Here's the CDL I am trying to use (the observation dim will eventually be
>> UNLIMITED):
>>
>> netcdf gsod {
>> types:
>> compound observation_type {
>> int time ;
>> float tempMean ;
>> float tempMin ;
>> float tempMax ;
>> }; // observation_type
>> compound observation_atts {
>> char time(32) ;
>> char tempMean(32) ;
>> char tempMin(32) ;
>> char tempMax(32) ;
>> }; // observation_atts
>> dimensions:
>> station = 2 ;
>> observation = 3 ;
>> variables:
>> int station_id(station) ;
>> float station_description(station) ;
>> float latitude(station) ;
>> latitude:units = "degrees_north" ;
>> float longitude(station) ;
>> longitude:units = "degrees_east" ;
>> float altitude(station) ;
>> altitude:units = "feet" ;
>> altitude:positive = "up" ;
>> observation_type station_observations(station, observation) ;
>> observation_atts station_observations:units = {{"days since
>> 1930-01-01 00 UTC"}, {"degF"}, {"degF"}, {"degF"}} ;
>>
>> // global attributes:
>> :cdm_datatype = "Stations" ;
>> :stationDimension = "station" ;
>> :observationDimension = "observation" ;
>> :latitude_coordinate = "latitude" ;
>> :longitude_coordinate = "longitude" ;
>> :zaxis_coordinate = "altitude" ;
>> :time_coordinate = "station_observations.time" ;
>> data:
>>
>> station_id = _, _ ;
>>
>> station_description = _, _ ;
>>
>> latitude = _, _ ;
>>
>> longitude = _, _ ;
>>
>> altitude = _, _ ;
>>
>> station_observations =
>> {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0},
>> {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0} ;
>> }
>>
>>
>> Thanks,
>>
>> Tom Kunicki
>> Center for Integrated Data Analytics
>> U.S. Geological Survey
>> 8505 Research Way
>> Middleton, WI 53562
>
> _______________________________________________
> netcdf-java mailing list
> netcdf-java@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
