Re: [cf-satellite] Sharing quality flags among multiple variables

NOTE: The cf-satellite mailing list is no longer active. The list archives are made available for historical reasons.

Didn't realize that this message is from cf-satellite group. The example that I gave is for in-situ data. But as can be seen, similar issue arises for in-situ data as well. CCing this to CF-METADATA group. 

Upendra

On 11/1/2011 10:47 AM, Upendra Dadi wrote:
The same issue occurs with World Ocean Database which consists of mainly profile data. Each profile typically consists of several variables measured along the depth. The quality flags used for all the variable are same. I have attached a snippet below of how a CF compliant netCDF would look. The over-head due to repeating the quality flags is not much when the data is stored is a multi-profile format, but when each profile is stored in an individual file, the overhead becomes significant, since these flags have to be present in each file. 

Upendra


 float temperature(z) ;
                temperature:standard_name = "sea_water_temperature" ;
                temperature:long_name = "sea water temperature" ;
                temperature:units = "degree_C" ;
                temperature:coordinates = "time lon lat z" ;
                temperature:WODprofile_flag = 0 ;
                temperature:_FillValue = -1.e+10f ;
                temperature:grid_mapping = "crs" ;
temperature:cell_methods = "time:point longitude:point latitude: 
point z:point" ;
                temperature:ancillary_variables = "temperatu
re_WODflag" ;
        int temperature_WODflag(z) ;
temperature_WODflag:standard_name = "sea_water_temperature statu 
s_flag" ;
temperature_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7, 8, 9" 
 ;
temperature_WODflag:flag_meanings = "accepted_value range_outlie r failed_inversion_check failed_gradient_check observed_level_bullseye_flag_and_ zero_gradient_check combined_gradient_and_inversion_checks failed_range_and_inve rsion_checks failed_range_and_gradient_checks failed_range_and_questionable_data 
_checks failed_range_and_combined_gradient_and_inversion_checks" ;
                temperature_WODflag:reference = "" ;


        float salinity(z) ;
                salinity:standard_name = "sea_water_salinity" ;
                salinity:long_name = "sea water salinity" ;
                salinity:units = "" ;
                salinity:coordinates = "time lon lat z" ;
                salinity:WODprofile_flag = 0 ;
                salinity:_FillValue = -1.e+10f ;
                salinity:grid_mapping = "crs" ;
salinity:cell_methods = "time:point longitude:point latitude:poi 
nt z:point" ;
                salinity:ancillary_variables = "salinity_WODfla
g" ;
        int salinity_WODflag(z) ;
salinity_WODflag:standard_name = "sea_water_salinity status_flag 
" ;
salinity_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7, 8, 9" ; salinity_WODflag:flag_meanings = "accepted_value range_outlier f ailed_inversion_check failed_gradient_check observed_level_bullseye_flag_and_zer o_gradient_check combined_gradient_and_inversion_checks failed_range_and_inversi on_checks failed_range_and_gradient_checks failed_range_and_questionable_data_ch 
ecks failed_range_and_combined_gradient_and_inversion_checks" ;
                salinity_WODflag:comment = "" ;


        float oxygen(z) ;
oxygen:standard_name = "volume_fraction_of_oxygen_in_sea_water" 
;
                oxygen:long_name = "Oxygen" ;
                oxygen:units = "ml/l" ;
                oxygen:coordinates = "time lon lat z" ;
                oxygen:WODprofile_flag = 0 ;
                oxygen:_FillValue = -1.e+10f ;
                oxygen:grid_mapping = "crs" ;
oxygen:cell_methods = "time:point longitude:point latitude:point 
 z:point" ;
                oxygen:ancillary_variables = "oxygen_WODflag" ;
        int oxygen_WODflag(z) ;
oxygen_WODflag:standard_name = "volume_fraction_of_oxygen_in_sea 
_water status_flag" ;
oxygen_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7, 8, 9" ; oxygen_WODflag:flag_meanings = "accepted_value range_outlier fai led_inversion_check failed_gradient_check observed_level_bullseye_flag_and_zero_ gradient_check combined_gradient_and_inversion_checks failed_range_and_inversion _checks failed_range_and_gradient_checks failed_range_and_questionable_data_chec 
ks failed_range_and_combined_gradient_and_inversion_checks" ;
                oxygen_WODflag:comment = "" ;


        float phosphate(z) ;
phosphate:standard_name = "mole_concentration_of_phosphate_in_se 
a_water " ;
                phosphate:long_name = "Phosphate" ;
                phosphate:units = "umol/l" ;
                phosphate:coordinates = "time lon lat z" ;
                phosphate:WODprofile_flag = 0 ;
                phosphate:_FillValue = -1.e+10f ;
                phosphate:grid_mapping = "crs" ;
phosphate:cell_methods = "time:point longitude:point latitude:po 
int z:point" ;
                phosphate:ancillary_variables = "phosphate_WOD
flag" ;
        int phosphate_WODflag(z) ;
phosphate_WODflag:standard_name = "mole_concentration_of_phospha 
te_in_sea_water status_flag" ;
phosphate_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7, 8, 9" ; phosphate_WODflag:flag_meanings = "accepted_value range_outlier failed_inversion_check failed_gradient_check observed_level_bullseye_flag_and_ze ro_gradient_check combined_gradient_and_inversion_checks failed_range_and_invers ion_checks failed_range_and_gradient_checks failed_range_and_questionable_data_c 
hecks failed_range_and_combined_gradient_and_inversion_checks" ;
                phosphate_WODflag:comment = "" ;


        float silicate(z) ;
silicate:standard_name = "mole_concentration_of_silicate_in_sea_ 
water" ;
                silicate:long_name = "Silicate" ;
                silicate:units = "umol/l" ;
                silicate:coordinates = "time lon lat z" ;
                silicate:WODprofile_flag = 0 ;
                silicate:_FillValue = -1.e+10f ;
                silicate:grid_mapping = "crs" ;
silicate:cell_methods = "time:point longitude:point latitude:point z:point" ; silicate:ancillary_variables = "silicate_sigfigs silicate_WODflag" ; 
        int silicate_WODflag(z) ;
silicate_WODflag:standard_name = "mole_concentration_of_silicate_in_sea_water status_flag" ; silicate_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7, 8, 9" ; silicate_WODflag:flag_meanings = "accepted_value range_outlier failed_inversion_check failed_gradient_check observed_level_bullseye_flag_and_zero_gradient_check combined_gradient_an d_inversion_checks failed_range_and_inversion_checks failed_range_and_gradient_checks failed_range_and_questionable_data_checks failed_range_and_combined_gradient_and_inversion_checks" ; 
                silicate_WODflag:comment = "" ;


        float nitrate(z) ;
nitrate:standard_name = "mole_concentration_of_nitrate_in_sea_water" ; 
                nitrate:long_name = "Nitrate" ;
                nitrate:units = "umol/l" ;
                nitrate:coordinates = "time lon lat z" ;
                nitrate:WODprofile_flag = 0 ;
                nitrate:_FillValue = -1.e+10f ;
                nitrate:grid_mapping = "crs" ;
nitrate:cell_methods = "time:point longitude:point latitude:point z:point" ; 
                nitrate:ancillary_variables = "nitrate_WODflag" ;
        int nitrate_WODflag(z) ;
nitrate_WODflag:standard_name = "mole_concentration_of_nitrate_in_sea_water status_flag" ; nitrate_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7, 8, 9" ; nitrate_WODflag:flag_meanings = "accepted_value range_outlier failed_inversion_check failed_gradient_check observed_level_bullseye_flag_and_zero_gradient_check combined_gradient_and _inversion_checks failed_range_and_inversion_checks failed_range_and_gradient_checks failed_range_and_questionable_data_checks failed_range_and_combined_gradient_and_inversion_checks" ; 
                nitrate_WODflag:comment = "" ;


        float nitrite(z) ;
                nitrite:long_name = "Nitrite" ;
                nitrite:units = "umol/l" ;
                nitrite:coordinates = "time lon lat z" ;
                nitrite:WODprofile_flag = 0 ;
                nitrite:_FillValue = -1.e+10f ;
                nitrite:grid_mapping = "crs" ;
nitrite:cell_methods = "time:point longitude:point latitude:point z:point" ; 
                nitrite:ancillary_variables = "nitrite_WODflag" ;
        int nitrite_WODflag(z) ;
nitrite_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7, 8, 9" ; nitrite_WODflag:flag_meanings = "accepted_value range_outlier failed_inversion_check failed_gradient_check observed_level_bullseye_flag_and_zero_gradient_check combined_gradient_and _inversion_checks failed_range_and_inversion_checks failed_range_and_gradient_checks failed_range_and_questionable_data_checks failed_range_and_combined_gradient_and_inversion_checks" ; 
                nitrite_WODflag:comment = "" ;

On 10/31/2011 12:12 PM, Randy Horne wrote:
In the application I am working on, there are specific types of products where where the same quality flags apply to multiple variables. These products will be contained in a single NetCDF file.
The current CF conventions dictate that quality flags are attached to specific variables. The implication is that comforming with CF conventions would require the same quality flags to be stored multiple times in our NetCDF product files. This is potentially problematic due to the resulting increase in product file size. Our variables have on the order of 10s of millions of data points where each has a distinct quality flag.
This issue has come up before, and any design to optimize flags has been dismissed due to the overriding objective of having variables that are self-describing.
Product file compression is a potential answer, but this presents its own set of performance issues.
Maybe flags could be handled in a similar manner as coordinates to allow sharing among variables ? 



..............End of Message ...............................-->





_______________________________________________
cf-satellite mailing list
cf-satellite@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/ 

_______________________________________________
cf-satellite mailing list
cf-satellite@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
  • 2011 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the cf-satellite archives: