[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: scale and offset



Turns out the problem is missing_value, which must be in packed units:

from http://www.unidata.ucar.edu/software/netcdf-java/v4.1/javadoc/index.html

Implementation rules for missing data with scale/offset

  1. Always converted to a float or double type.
  2. _FillValue and missing_value values are always in the units of the external (packed) data.
  3. If valid_range is the same type as scale_factor (actually the wider of scale_factor and add_offset) and this is wider than the external data, then it will be interpreted as being in the units of the internal (unpacked) data. Otherwise it is in the units of the external (packed) data.
  4. The dataType is set to float if all attributes used are float (scale_factor, add_offset valid_min, valid_max, valid_range, missing_data and _FillValue) otherwise the dataType is set to double

technically, valid_min and max should also be in packed units acording to the NUG, but we relaxed that in netcdf-java to accomodate some existing datasets that misread the manual.

also, he is using floats, and is encountering roundoff error, looking in the debugger i see:

valid_min = -140.00045776367188

which will also cause problems.


so:

1:  dont use missing_value or valid min/max unless you need them, eg you actually have missing data.
2. missing_value must be in packed units.
3. make valid_min and valid_max a little wider than you need to accomodate roundoff


On 3/23/2010 4:35 PM, Don Murray wrote:
John-

I'm trying to respond to the ticket that's been flopping back and forth between netcdf-java and IDV  (ticket QAW-317402).  In the file
(testdata/grid/netcdf/G11.20090602.0715.nc), LONGITUDE is defined as:

        int LONGITUDE(lines, samples) ;
                LONGITUDE:units = "" ;
                LONGITUDE:missing_value = -94.f ;
                LONGITUDE:valid_min = -140.0005f ;
                LONGITUDE:valid_max = -95.0994f ;
                LONGITUDE:scale = 1.e-007f ;
                LONGITUDE:offset = -140.0005f ;
                LONGITUDE:scale_factor = 1.e-007f ;
                LONGITUDE:add_offset = -140.0005f ;

The first longitude value is 0, but it comes up through the netCDF-Java stack as a NaN.  I would think that it would be 0*1E-7+-140.0005 = -140.0005 which is the valid min and shouldn't be set to NaN.  The last latitude is also set to NaN, but there, we have 24902*.001+24.999 = 49.901 which is beyond the valid_max of 49.90084.

A bug?  Is the missing value being scaled and offsetted?  That would make it -94*1E-7+-140.0005 = -140.0005094 which might be "close enough" in EnhanceScaleMissingImpl.isMissingValue().  I would think that the missing value would only be scaled and offset if it was narrower than the converted data.  In the toolsUI, I notice that the LONGITUDE looks like:

   float LONGITUDE(lines=505, samples=909);
     :units = "";
     :scale = 1.0E-7f; // float
     :offset = -140.00046f; // float
     :_CoordinateAxisType = "Lon";

a roundoff error for the offset? Maybe that's the problem?

Don
*************************************************************
Don Murray                               UCAR Unidata Program
address@hidden                        P.O. Box 3000
(303) 497-8628                              Boulder, CO 80307
http://www.unidata.ucar.edu/staff/donm
*************************************************************