[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [netcdf-java] Announcing Netcdf-Java / CDM version 4.3.8 BETA and emergency shutdown procedures

Hi John-

On 2/10/12 1:02 PM, John Caron wrote:

- you say that the variable names have changed between 4.2 and 4.3 for
GRIB files. What is the extent of these changes and when do you expect
these will be used in the TDS? The IDV relies on the variable names to
look up parameters in bundles, to associate user preferences for
displays (color table, units, contouring information) and the variable
name is encoded in bundles to they always get the same data. So, if
the variable names are different than before, that will mean that IDV
bundles accessing TDS servers using the 4.3 library will be broken.
That would be a major problem for the IDV community.


the names have to change for lots of reasons. some background is here:

http://www.unidata.ucar.edu/blogs/developer/en/entry/dataset_schemas_are_lost_in

the previous algorithm tried to modify the names based on what other
variables with the same parameter exists in the same file. but this
breaks aggregation and has other, subtle effects that are not
maintainable. plus there are lots of errors in 4.2 tables. plus
intervals were ignored, and statistical types not always handled well,
etc. The previous GRIB handling is flawed enough that I think that users
need to repick which variables they really want. I think the IDV is
going to have to develop a graceful way to allow bundles to evolve.
Adding some feature to the IDV to do a table lookup between old and new names would be one way to handle this. The problem is one of timing. On the day that the motherlode TDS server switches to use the 4.3 library, every IDV user will need to have whatever mechanism is developed to handle the new names. Many bundles are stored on servers or encoded in JNLP files (like the Millersville LEAD project bundles), so it's not just a matter of saving a new bundle locally.
Also, how will you handle the FMRC stuff where older files were indexed with the old names. Does 4.3 still read the old index files? 


however, im not particular happy with the what i have. for one thing,
tables get tweaked, eg by the WMO. Also, we may realize that some
obscure field in the PDS must be added to the name. ugh.



its possible that an opaque id is a better idea, with all semantics in
the long name. Another alternative is what NCL does with their encoding,
see:

http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/NclFormatSupport.shtml#GRIB

im open to other suggestions.
It seems like adding attributes to handle changes is better than constantly changing the variable names. However, one thing the netCDF-Java library has over NCL is that it provides human readable names. For example, here's a comparison of NCL and netCDF-Java 4.2 names that I've been using for my work with GFS Reforecast GRIB files: 

"TMP_SFC"    : {"TMP_P0_L1_GGA0"        :  "Temperature_surface"},
"HGT_HYBR"   : {"HGT_P0_L105_GGA0"      :  "Geopotential_height_hybrid"},
"PVORT_ISEN" : {"PVORT_P0_L107_GGA0"    :  "Potential_vorticity"},
"TMP_PVOR" : {"TMP_P0_L109_GGA0" : "Temperature_potential_vorticity_surface"}, 
"VOGRD_HGT"  : {"VOGRD_P0_L103_GGA0"    :  "V-component_of_current"},
"WEASD_SFC" : {"WEASD_P0_L1_GGA0" : "Water_equivalent_of_accumulated_snow_depth"}, 
"LHTFL_SFC"  : {"LHTFL_P8_L1_GGA0_avg"  :  "Latent_heat_net_flux"},
"PRES_SFC"   : {"PRES_P0_L1_GGA0"       :  "Pressure_surface"},
"PRES_PVOR" : {"PRES_P0_L109_GGA0" : "Pressure_potential_vorticity_surface"}, 
"VGRD_HYBR"  : {"VGRD_P0_L105_GGA0"     :  "V-component_of_wind_hybrid"},
"SOILW_BGRND": {"SOILW_P0_2L106_GGA0" : "Volumetric_Soil_Moisture_Content"}, "TMP_BGRND" : {"TMP_P0_2L106_GGA0" : "Temperature_depth_below_surface_layer"}, "TCOLC_EATM" : {"TCOLC_P0_L200_GGA0" : "Total_Column-Integrated_Condensate"}, 
"UGRD_PRES"  : {"UGRD_P0_L100_GGA0"     :  "U-component_of_wind"},
"WATR_SFC"   : {"WATR_P8_L1_GGA0_acc"   :  "Water_runoff"},
"VGRD_HGT" : {"VGRD_P0_L103_GGA0" : "V-component_of_wind_height_above_ground"}, 
"PWAT_EATM"  : {"PWAT_P0_L200_GGA0"     :  "Precipitable_water"},
"VGRD_PRES"  : {"VGRD_P0_L100_GGA0"     :  "V-component_of_wind"},
"PRES_MSL"   : {"PRES_P0_L101_GGA0"     :  "Pressure"},
"DSWRF_SFC"  : {"DSWRF_P8_L1_GGA0_avg"  :  "Downward_Short-Wave_Rad_Flux"},
"TMP_HGT" : {"TMP_P0_L103_GGA0" : "Temperature_height_above_ground"}, 
"TMP_PRES"   : {"TMP_P0_L100_GGA0"      :  "Temperature"},
"APCP_SFC"   : {"APCP_P8_L1_GGA0_acc"   :  "Total_precipitation"},
"TMIN_HGT"   : {"TMIN_P8_L103_GGA0"     :  "Minimum_temperature"},
"ULWRF_TATM" : {"ULWRF_P8_L8_GGA0_avg"  :  "Upward_Long-Wave_Rad_Flux"},
"ULWRF_SFC" : {"ULWRF_P8_L1_GGA0_avg" : "Upward_Long-Wave_Rad_Flux_surface"}, 
"TCDC_EATM"  : {"TCDC_P0_L200_GGA0"     :  "Total_cloud_cover"},
"VVEL_PRSS"  : {"VVEL_P0_L100_GGA0"     :  "Vertical_velocity_pressure"},
"CAPE_SFC" : {"CAPE_P0_L1_GGA0" : "Convective_available_potential_energy"}, 
"CIN_SFC"    : {"CIN_P0_L1_GGA0"        :  "Convective_inhibition"},
"SHTFL_SFC"  : {"SHTFL_P8_L1_GGA0_avg"  :  "Sensible_heat_net_flux"},
"GFLUX_SFC"  : {"GFLUX_P8_L1_GGA0_avg"  :  "Ground_Heat_Flux"},
"UGRD_HYBR"  : {"UGRD_P0_L105_GGA0"     :  "U-component_of_wind_hybrid"},
"SPFH_HGT" : {"SPFH_P0_L103_GGA0" : "Specific_humidity_height_above_ground"}, "UGRD_PVOR" : {"UGRD_P0_L109_GGA0" : "U-component_of_wind_potential_vorticity_surface"}, 
"UOGRD_HGT"  : {"UOGRD_P0_L103_GGA0"    :  "U-component_of_current"},
"USWRF_SFC"  : {"USWRF_P8_L1_GGA0_avg"  :  "Upward_Short-Wave_Rad_Flux"},
"UGRD_HGT" : {"UGRD_P0_L103_GGA0" : "U-component_of_wind_height_above_ground"}, 
"SPFH_PRES"  : {"SPFH_P0_L100_GGA0"     :  "Specific_humidity"},
"WMIXE_HGT"  : {"WMIXE_P0_L103_GGA0"    :  "Wind_mixing_energy"},
"TMAX_HGT"   : {"TMAX_P8_L103_GGA0"     :  "Maximum_temperature"},
"DLWRF_SFC"  : {"DLWRF_P8_L1_GGA0_avg"  :  "Downward_Long-Wave_Rad_Flux"},
"VGRD_PVOR" : {"VGRD_P0_L109_GGA0" : "V-component_of_wind_potential_vorticity_surface"}, 
"HGT_PRES"   : {"HGT_P0_L100_GGA0"      :  "Geopotential_height"},
"SUNS_SFC"   : {"SUNS_P0_L1_GGA0"       :  "Sunshine"}

}
The first column is a combination of the wgrib2/NCEP standard variable name (from NCEP Table 4.2 - http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_table4-2.shtml) with a level type (that I made up based on GEMPAK level names) (var_lev). The second column is the NCL name, and the last column is the netCDF-Java name (some have been slightly modified).
The NCL names change depending on the forecast hour to indicate the time averaging (e.g. LHTFL_P8_L1_GGA0_avg at forecast hour 0 and LHTFL_P8_L1_GGA0_avg6h at fhour 12) which would be a nightmare for aggregation. Lat/lon grids have different variable names than gaussian grids (LL vs. GG). I would avoid using their whole naming convention.
In the end, we need something simple and unique for the variable name and let the details be handled in the attributes. Here's a sample NCL set of attributes: 

        float LHTFL_P8_L1_GGA0_avg(lat_0, lon_0) ;
                LHTFL_P8_L1_GGA0_avg:initial_time = "01/01/2007 (00:00)" ;
                LHTFL_P8_L1_GGA0_avg:forecast_time_units = "hours" ;
                LHTFL_P8_L1_GGA0_avg:forecast_time = 3 ;
LHTFL_P8_L1_GGA0_avg:statistical_process_duration = "initial time to forecast time" ; LHTFL_P8_L1_GGA0_avg:type_of_statistical_processing = "Average" ; 
                LHTFL_P8_L1_GGA0_avg:level = 0.f ;
LHTFL_P8_L1_GGA0_avg:level_type = "Ground or water surface" ;
LHTFL_P8_L1_GGA0_avg:parameter_template_discipline_category_number = 8, 0, 0, 10 ; LHTFL_P8_L1_GGA0_avg:parameter_discipline_and_category = "Meteorological products, Temperature" ; LHTFL_P8_L1_GGA0_avg:grid_type = "Gaussian latitude/longitude" ; 
                LHTFL_P8_L1_GGA0_avg:_FillValue = 1.e+20f ;
                LHTFL_P8_L1_GGA0_avg:units = "W m-2" ;
                LHTFL_P8_L1_GGA0_avg:long_name = "Latent heat net flux" ;
LHTFL_P8_L1_GGA0_avg:production_status = "Operational products" ; LHTFL_P8_L1_GGA0_avg:center = "US National Weather Service - NCEP (WMC)" ; 

which encapsulate a lot of the ancillary PDS parameters.
One thing that would help is to generate a list of 4.2 variable names with the corresponding 4.3 names for all the GRIB datasets on motherlode. That could be used by the IDV for the lookup table. 



- On the netCDF-Java documentation page, there is a link to the 4.3
ncIdv.jar. This does not work with the IDV, so I would suggest
removing that until it does to avoid unnecessary support requests.


ok, thanks. we plan to integrate 4.3 and IDV when Yuan gets back from
his trip.
Again, you will need to have something stable in place and tested well before motherlode switches over. Otherwise, all of Tom Y's precipitable water bundles will break, as will Jim Steenburgh's bundles that he uses in class and NOAA's vizwall bundles. ;-) 

Don
--
Don Murray
NOAA/ESRL/PSD and CIRES
303-497-3596
http://www.esrl.noaa.gov/psd/people/don.murray/