Re: [cf-satellite] Proposal for handling multiple "missing_value" values

• To: Jim Biard <Jim.Biard@xxxxxxxx>
• Subject: Re: [cf-satellite] Proposal for handling multiple "missing_value" values
• From: Russ Rew <russ@xxxxxxxxxxxxxxxx>
• Date: Fri, 15 Jul 2011 13:09:25 -0600

Hi Jim,

> In the ESIP Federation CF-Satellite session yesterday (July 14, 2011), I 
> suggested that it would be good to have a standard scheme for handling 
> multiple marker values in satellite data that indicate missing data.  
> There is confusion in the current documentation (between netCDF and CF) 
> about the proper use of the missing_value attribute, in particular about 
> whether or not it is acceptable for it to be a vector attribute.  I 
> think there is good reason to allow it to be a vector attribute, and 
> that allowing the attribute to be a vector quantity is useful beyond the 
> satellite / remote sensing arena.  The problem that arises with having a 
> vector missing_value attribute is the assignment of meanings to the 
> different values.
> 
> One way to deal with this is to have a missing_value_meanings 
> attribute.  The missing_value_meanings attribute would be formed in the 
> same way as the flag_meanings attribute.  There would be one string of 
> non-whitespace characters per element of the missing_value attribute, 
> separated by spaces.  Each string would be a human-readable phrase 
> (delineating the words of the phrase using underscores or camel-casing) 
> that explained the meaning associated with the corresponding 
> missing_value element.  Here's an example fragment of cdl.
> 
>     float foo(points);
> 
>         foo:missing_value = -999.1, -999.2;
>         foo:missing_value_meanings = "InputsOutOfRange
>         SolutionDidNotConverge";

I agree that it would be very useful to have an attribute for multiple
special values for satellite data as well as other kinds of data.  IIRC,
you gave an example of 10 different kinds of missing values for an NPP
sensor.

I'm a little concerned about breaking backward compatibility by
extending the "missing_value" attribute for this purpose, because there
may be software that depends on a one-to-one mapping between a
missing_value for a variable and some alternate representation for it,
such as the '_' notation ncdump already uses.  The fact that the ncdump
and ncgen utilities are inverses of each other, converting netCDF data
to CDL and back to netCDF without changing the data, requires a single
value for the missing value to avoid loss of information when converting
from '_' back to the corresponding value.

I suggest just using the existing CF flag conventions for this purpose
instead, using the variable attributes "flag_values", and
"flag_meanings" for any number of special values for a variable.

If one of those values is a missing value, the associated flag_meaning
could just be "missing_value", but flag_meanings can also specify an
arbitrary number of other nuanced kinds of missing values or various
other kinds of special values.

  
http://cfconventions.org/documents/cf-conventions/1.5/cf-conventions.html#flags

--Russ
_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
russ@xxxxxxxxxxxxxxxx                     http://www.unidata.ucar.edu

• Follow-Ups:
  • Re: [cf-satellite] Proposal for handling multiple "missing_value" values
    • From: Jim Biard

• References:
  • [cf-satellite] Proposal for handling multiple "missing_value" values
    • From: Jim Biard