NOTE: The cf-satellite mailing list is no longer active. The list archives are made available for historical reasons.
On Oct 31, 2011, at 1:51 PM, Tom Whittaker wrote: > Hi Randy... > > My read on the "Flags" in Section 3.5 of the CF Conventions is that > the intended use is that a variable contains "flag values" which are > described by the various "flag_" attributes (flag_values, > flag_meanings, etc). While the implication is that the variable > contains only "flag" values, I'm not certain that is essential. > > When you say your variables have "10s of millions of data points where > each has a distinct quality flag", I am not sure if you are saying the > variables _contain_ the quality flag value(s) or somehow they are in a > different variable. For example, you might have actual data values > in the range of 0:100, but if the data value at a point is "101" then > it has a special meaning. If that is the case, could you not simply > use the "flag_values" to indicate "101" is a flag value? Unless the I don't want to speak for Randy, but I know it is quite common in Level 2 data from the Earth Observing System to have quality variables where there is a one-to-one match at the pixel level between a given quality variable and 1 or more data variables. We see this case in AIRS Level 2 as well as MODIS Level 2 atmospheres products. (It is not the same as simply saying what the valid_range is, or giving a special value that applies to all pixels in the data variable.) I admit I, too, was thinking along the same lines as Randy in terms of an analogue to the "coordinates" attribute (e.g., var:coordinates = "latitude longitude" ;) So we might have: totH2OStd:quality_flags = "Qual_H2O" ; Of course, the file is still not completely self describing as the semantics of the flags like Qual_H2O are critical to interpretation and often unique (we actually use an ontology to model this in our Data Quality Screening Service), but it is a step in the right direction. A further complication is that often the quality variable has one less dimension than the data variable, so that MLS, for example, has one quality variable value per vertical profile. Similarly, AIRS 3-d variables (e.g., temperature profiles in X-Y-Z) are described by 2-D quality variables, one value for each horizontal location (X-Y). > netCDF library handles variables with these attributes in a special > way, it would be up to the application that reads the data to pick up > the "flag_" attributes and do something with the values that match. > If this is the case, I don't see that your file size would increase > substantially...of course, I could be mis-interpreting what you said, > too. > > tom > > On Mon, Oct 31, 2011 at 11:12 AM, Randy Horne <rhorne@xxxxxxxxxxxxxxxxx> > wrote: >> In the application I am working on, there are specific types of products >> where where the same quality flags apply to multiple variables. These >> products will be contained in a single NetCDF file. >> >> The current CF conventions dictate that quality flags are attached to >> specific variables. The implication is that comforming with CF conventions >> would require the same quality flags to be stored multiple times in our >> NetCDF product files. This is potentially problematic due to the resulting >> increase in product file size. Our variables have on the order of 10s of >> millions of data points where each has a distinct quality flag. >> >> This issue has come up before, and any design to optimize flags has been >> dismissed due to the overriding objective of having variables that are >> self-describing. >> >> Product file compression is a potential answer, but this presents its own >> set of performance issues. >> >> Maybe flags could be handled in a similar manner as coordinates to allow >> sharing among variables ? >> >> >> >> ..............End of Message ...............................--> >> >> >> >> >> >> _______________________________________________ >> cf-satellite mailing list >> cf-satellite@xxxxxxxxxxxxxxxx >> For list information or to unsubscribe, visit: >> http://www.unidata.ucar.edu/mailing_lists/ >> > > > > -- > Tom Whittaker > University of Wisconsin-Madison > Space Science & Engineering Center (SSEC) > Cooperative Institute for Meteorological Satellite Studies (CIMSS) > 1225 W. Dayton Street > Madison, WI 53706 USA > ph: +1 608 262 2759 > > _______________________________________________ > cf-satellite mailing list > cf-satellite@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ Christopher Lynnes Goddard Earth Sciences Data and Information Center, NASA/GSFC 301-614-5185
cf-satellite archives: