On Wed, Apr 15, 2015 at 10:12 AM, John Caron <caron@xxxxxxxx> wrote:
> netcdf-java has several levels of API. The lowest level (NetcdfFile)
> doesnt touch the data, so you just get whatever is in there. this is the
> level that users of the C library are used to.
>
> the next highest level of the netcdf-java API (NetcdfDataset) process
> missing data, scale/offset, CF conventions, and other things. At this layer
> the various attribute conventions are used to identify missing data.
>
got it -- and I didn't say it, b ut I was thinking that in a data-model
type lib, you probably want missing values handles well. But I still don't
think I would want values outside the bounds specified as getting turned
into missing values unless I specifically asked for that in some sort of
QA/QC flag.
Which it seemed the OP was expecting.
-Chris
> John
>
> On Wed, Apr 15, 2015 at 10:27 AM, Chris Barker <chris.barker@xxxxxxxx>
> wrote:
>
>> As I read those docs, and as I've always interpreted netcdf, those
>> "rules" are there to inform the user of the data -- I don't expect client
>> software reading the data to replace values in the the arrays for me -- at
>> least not low-level generic data reading libraries.
>>
>> What were you expecting, and why?
>>
>> Higher level libs that represent a data model _may_ do some QA/QC on the
>> data, I suppose, but even then, I'd want to clearly specify what I wanted
>> it to do -- certainly not silently toss out values because the were outside
>> the limits specified in the meta-data.
>>
>> -Chris
>>
>>
>>
>> On Wed, Apr 15, 2015 at 9:18 AM, John Caron <caron@xxxxxxxx> wrote:
>>
>>> The netcdf-java library does indeed ignore that rule. We should revisit
>>> the manual and clarify things i guess.
>>>
>>> On Wed, Apr 15, 2015 at 9:33 AM, Russ Rew <russ@xxxxxxxxxxxxxxxx> wrote:
>>>
>>>> Hi Phil,
>>>>
>>>> I think that's a misinterpretation of the Users Guide attribute
>>>> conventions. Under the description for _FillValue, the Guide states:
>>>>
>>>> The fill value ... is normally outside the valid range and therefore
>>>> treated as missing when read by generic applications. It is legal (but not
>>>> recommended) for the fill value to be within the valid range.
>>>>
>>>>
>>>> The last sentence implies that the valid range is not determined by the
>>>> _Fill_Value.
>>>>
>>>> I think it's intended that the rule about _Fill_Value under the
>>>> description for valid_range only applies in case none of valid_min,
>>>> valid_max, or valid_range are specified, so it wouldn't apply to your
>>>> example:
>>>>
>>>> If neither valid_min, valid_max nor valid_range is defined then generic
>>>> applications should define a valid range as follows. ...
>>>>
>>>>
>>>> However, I suspect that the rule is confusing enough that writers of
>>>> generic clients might just ignore it, even in that case.
>>>>
>>>> --Russ
>>>>
>>>> On Wed, Apr 15, 2015 at 8:55 AM, Bentley, Philip <
>>>> philip.bentley@xxxxxxxxxxxxxxxx> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> I was hoping someone can clarify for me the correct use of the
>>>>> valid_min, valid_max and valid_range attributes by *well-behaved* netCDF
>>>>> clients.
>>>>>
>>>>> Given the netCDF file shown below (in CDL form), and considering the
>>>>> rules for handling the aforementioned attributes, as defined in Appendix A
>>>>> of the NetCDF user guide (see
>>>>> http://www.unidata.ucar.edu/software/netcdf/docs/attribute_conventions.html),
>>>>> I *think* I'd expect conforming netCDF clients to represent the data for
>>>>> each of the variables var1..var3 as an array with values (_, -272, 0, 100,
>>>>> 9999, _), where _ signifies missing data values.
>>>>>
>>>>> ---
>>>>> netcdf mditest {
>>>>>
>>>>> dimensions:
>>>>> dim1 = 6;
>>>>>
>>>>> variables:
>>>>> float var1(dim1);
>>>>> var1:long_name = "var1";
>>>>> var1:units = "1";
>>>>> var1:valid_min = -273.0f;
>>>>> var1:valid_max = 1.0e5f;
>>>>>
>>>>> float var2(dim1);
>>>>> var2:long_name = "var2";
>>>>> var2:units = "1";
>>>>> var2:valid_min = -273.0f;
>>>>> var2:_FillValue = 1.0e5f; // constrains valid_max
>>>>>
>>>>> float var3(dim1);
>>>>> var3:long_name = "var3";
>>>>> var3:units = "1";
>>>>> var3:_FillValue = -273.0f; // constrains valid_min
>>>>> var3:valid_max = 1.0e5f;
>>>>>
>>>>> // global attributes
>>>>> :Conventions = "CF-1.0";
>>>>>
>>>>> data:
>>>>> var1 = -300, -272, 0, 100, 9999, 1e6 ;
>>>>> var2 = -300, -272, 0, 100, 9999, 1e6 ;
>>>>> var3 = -300, -272, 0, 100, 9999, 1e6 ;
>>>>> }
>>>>> ---
>>>>>
>>>>> However, having tried several different netCDF clients - some C, some
>>>>> Java, some Python - none of them appear to adhere consistently to the
>>>>> aforementioned rules for handling the valid min/max/range attributes. The
>>>>> python-based clients, in particular, only seem to honour the _FillValue
>>>>> attribute, reflecting, I believe, the current behaviour of the
>>>>> netcdf4-python module.
>>>>>
>>>>> Am I perhaps misinterpreting the nc attribute-handling conventions?
>>>>>
>>>>> Regards,
>>>>>
>>>>> --
>>>>> Phil Bentley PhD, Climate Science IT Apps Group
>>>>> Met Office Hadley Centre, FitzRoy Road, Exeter, EX1 3PB, United Kingdom
>>>>> Tel: +44 (0)1392 886881
>>>>> Email: philip.bentley@xxxxxxxxxxxxxxxx Web:
>>>>> http://www.metoffice.gov.uk/
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> netcdfgroup mailing list
>>>>> netcdfgroup@xxxxxxxxxxxxxxxx
>>>>> For list information or to unsubscribe, visit:
>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> netcdfgroup mailing list
>>>> netcdfgroup@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe, visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>
>>>
>>> _______________________________________________
>>> netcdfgroup mailing list
>>> netcdfgroup@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>>
>>
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R (206) 526-6959 voice
>> 7600 Sand Point Way NE (206) 526-6329 fax
>> Seattle, WA 98115 (206) 526-6317 main reception
>>
>> Chris.Barker@xxxxxxxx
>>
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>
>
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@xxxxxxxx
