Hal, who do you serve: machines or humans?
Tyn
Op Feb 28, 2012 om 3:52 heeft "Don Murray" <don.murray@xxxxxxxx> het volgende
geschreven:
> For the sake of the IDV community, I'm reposting my response to the
> netcdf-java and thredds lists:
>
> <repost>
> Hi John and Ethan-
>
> As I have discussed with you at length privately, I am not in favor of
> this change. This will break every IDV bundle that points to GRIB data
> in a local file or on a TDS server. This will also affect users of the
> TDS on the NCDC NOMADS servers who access data either through scripts or
> the IDV. It's not a simple matter of users just picking new names and
> resaving the bundles when the bundles are stored on remote servers or
> used in a classroom setting.
>
> Below, for the benefit of the list, are my arguments for using the human
> readable variable names in the previous netCDF-Java 4.3 beta release:
>
> <quote>
> I believe keeping the human readable variable names (as in the previous
> 4.3 release - with slight modifications) is much preferable and backward
> compatible. I understand your reasons for wanting to change, but while
> that makes the programmer's life easier, it makes the user's (and other
> programmers') life harder.
>
> For example, from a user perspective, with your changes, I'm going to
> have to modify 50 or more bundles that are on my local machines
> (including the NOAA viz wall) or stored on RAMADDA servers which will
> take several days. I'm also going to have to modify the customizations
> to my IDV parameter tables that I've made over the past 7 years.
>
> From a programmer's perspective, here are the impacts of your changes
> to the IDV:
>
> - bundles which use the variable name for lookup
> - data aliases used for derived quantities
> - parameter aliases used for automatically assigning color tables,
> contour intervals and units
> - User guide and workshop documentation and examples will need to be
> updated
>
> For the past 7 or so years, IDV users have been able to access realtime
> GRIB datasets and have had stability in using and interchanging those
> datasets. For example, I have a bundle:
>
> http://motherlode.ucar.edu/repository/entry/get/GFS%2080%20km.xidv?entryid=9f77ca66-2264-4f8b-a460-e02fb42606ea
>
> which has displays of 500 hPa geopotential heights, sea level pressure
> and precipition from the GFS 80km data. These are simple, commonly used
> parameters. The IDV has a DataAlias table that equates the variable
> name Geopotential_height with a canonical name of HGT which is used to
> present derived quantities to the user of thickness and geostrophic
> wind. It also uses this name to assign a color table, unit and contour
> levels for any display created for the variable Geopotential height.
> Same idea goes for Pressure_reduced_to_MSL and Total_precipitation. It
> doesn't matter whether I go to the GFS 80 km (grib1) or the GFS .5
> degree global (grib2), or even a NAM 80km dataset. I can apply the
> bundle and use the same information to get the same type of display.
>
> Under the scheme in the previous version of 4.3beta, Geopotential_height
> will change to Geopotential_height_Pressure, Pressure_reduced_to_MSL
> will change to Pressure_reduced_to_MSL_Msl and Total_precipitation will
> change to one of:
>
> Total_precipitation_Surface_12_Hour_Accumulation
> Total_precipitation_Surface_1_Hour_Accumulation
> Total_precipitation_Surface_3_Hour_Accumulation
> Total_precipitation_Surface_6_Hour_Accumulation
> Total_precipitation_Surface_Mixed_intervals_Accumulation
>
> From the IDV perspective, the DataAlias and ParameterDefaults use
> patterns and case insensitive, so this should not be a problem because
> the old names would match into the new names. For the bundles, this
> will be problem, but one that can be dealt with on the IDV or
> netCDF-Java side with a paramater lookup as discussed at the recent IDV
> Developers teleconference and which is outlined from the IDV perspective
> here:
>
> https://mcidasv.ssec.wisc.edu/issues/11
>
> With the new naming:
>
> VAR_%d-%d-%d[_error][_L%d][_layer][_I%s_S%d][_D%d][_Prob_%s]
>
> The three variables would have different names depending on whether they
> came from a grib1 or grib2 dataset. This would require the Unidata IDV
> programmers to redo all the alias and parameter default tables and
> require a more complicated lookup just to find the 500 hPa geopotential
> height, sea level pressure and total_precipitation field depending on
> the dataset used. I think providing consistency between grib1 and grib2
> datasets at the very least is an important consideration - in the end,
> it's all GRIB. GEMPAK and McIDAS (as well as wgrib2 and NCL) create the
> same names for their variables independent of whether they came from
> Grib1 or 2.
>
> I fully support the notion of adding in the level information to the
> variable name as is the case for Geopotential_height. I know for
> variables like Temperature in the 4.2 scheme can provide different
> results depending on whether your grib files had a mixture of 2D and 3D
> varaibles (Temperature = the one on pressure levels) or just 2D
> variables (Temperature = the one on height above ground level). I
> understand the problems it creates on both the netCDF-Java/TDS side and
> sometimes the IDV side (e.g. creating derived quantities) and think that
> this change can be handled pretty well on the IDV side.
>
> I support adding the accumulation interval for parameters like
> Total_precipitation above because now some variables have a mixture of
> the different types of intervals.
>
> One of your arguments is that over time, names change and it's difficult
> to maintain tables. While that may be true for lesser variables, I
> would suggest that the most commonly used variable names rarely change
> (Temperature, geopotential height, relative humidity, u and v wind
> components, etc). Unidata has always been in the business of
> maintaining tables and that's part of the job it does to support the
> user community. While it's not easy, it is a necessary function of the
> services that Unidata provides. And, changing the names just pushes the
> work off to others at Unidata. Perhaps Unidata could look at having
> common tables used by all it's software for consistency. Or perhaps
> Unidata could work with the NCL group and use their lookup tables?
>
> In the end, I would like to see the netCDF-Java library evolve to suit
> the needs of the data providers, while also maintaining as much backward
> compatibility for the end users and software developers who rely on it.
> I think a lot of the ancillary information can be provided through
> variable attributes as it is in 4.2 (description, table number,
> Discipline/Category/Parmeter, GRIB GDS/PDS information) as NCL does, but
> leave human readable variable names.
> </quote>
>
> Outside the IDV, I have been using the netCDF-Java library in
> conjunction with PyNIO to convert grib2 data to netCDF. I use the
> human-readable netCDF-Java 4.2 variable names on my output files instead
> of the PyNIO names because I believe that the users of my output would
> prefer to see those than something like VAR_0-0-0_L6_I6_Hour_S194.
> </repost>
>
> And I agree with Jeff McWhirter's sentiment that it is not an onerous
> task to maintain the GRIB tables - Robb Kambic did it for years, Chiz
> did it for GEMPAK. If it is a priority of the community, then Unidata
> should undertake that task.
>
> Don
>
> On 2/27/12 5:24 PM, Yuan Ho wrote:
>> Hi All:
>> Please read the following message from thredds email list. As you know,
>> the IDV relies on the netCDF-Java library, and the coming changes on
>> netCDF-Java 4.3 would have huge impacts (all bundles with grib data need
>> to be redone; all preference tables need to be updated; etc.) on the
>> IDV. We need to have your feedback here.
>>
>>
>> Thanks,
>>
>>
>> IDV developer team
>>
>> -------- Original Message --------
>> Subject: [thredds] GRIB variable name changes in 4.3
>> Date: Mon, 27 Feb 2012 11:17:23 -0700
>> From: John Caron <caron@xxxxxxxxxxxxxxxx>
>> To: Java NetCDF <netcdf-java@xxxxxxxxxxxxxxxx>, THREDDS community
>> <thredds@xxxxxxxxxxxxxxxx>
>>
>> To all:
>>
>> The CDM / netCDF-Java library version 4.3 (and also TDS version 4.3) is
>> considering a radical change in the way that GRIB variables are named.
>> Instead of nice human readable names like
>>
>> float Temperature(time=1, lat=361, lon=720);
>>
>> they are now like
>>
>> float VAR_0-0-0_L6_I6_Hour_S194(time=1, lat=361, lon=720);
>>
>> with "human readable names" in the long_name:
>>
>> :long_name = "Temperature (6_Hour Average) @ Maximum wind level";
>>
>> The reasons for this change are that the "nice human readable names"
>> come from external GRIB tables, that is, the names are not in the files
>> themselves. GRIB table parameter names have no requirement to be unique
>> nor simple nor unchanging, i.e. they have no requirement to be suitable
>> as netCDF variable names. Maintainers of GRIB tables often make minor
>> changes to GRIB names, correcting typos or otherwise improving the
>> readability of the name. In some cases, the GRIB names are completely
>> changed. When the CDM starts to use new versions of the tables, the
>> variable names can and do change. Since calls to access data use the
>> name of the variable, many things break if the name changes.
>>
>> Any GRIB to netCDF translation software is in the position of either
>> hand-maintaining the tables to prevent names from changing (and fixing
>> duplicates or unsuitable names), or doing something else. Hand
>> maintaining GRIB tables is not a viable option due to resource
>> constraints. The something else is to give variables unique names based
>> only on the information actually in the file. The NCL package has
>> adopted a similar scheme:
>>
>> http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/NclFormatSupport.shtml#GRIB
>>
>>
>>
>> More background on this problem is here:
>>
>> http://www.unidata.ucar.edu/staff/caron/papers/GRIBarchivals.pdf
>>
>> Another aspect of this problem is that errors were found in version 4.2
>> with GRIB tables, with handling GRIB time intervals and ensemble data,
>> as well as with the algorithm for generating names when multiple
>> variables from the same parameter are in the same file. About 1 in 5
>> variable names (in the NCEP IDD data) need to change to fix these
>> problems. In reviewing how variable names are created, and how GRIB
>> tables are handled, these other problems became clear. Rather than
>> fixing the problem piecemeal, we are trying to make one big change all
>> at once, then do our best to not let this happen again.
>>
>> The main impact this will have is probably on:
>> 1) scripts or IDV bundles that have a GRIB variable name in them;
>> hopefully a one-time change will fix this.
>> 2) interactive applications that are built on top of the CDM. For
>> GRIB, users will need to see the long_name, not the variable name, to
>> know what they want. However, the CDM presents a uniform interface for
>> all files, not just GRIB, so the application can't assume that the
>> long_name is even present. So the application should present both the
>> variable name and the long_name (if it exists) to the user when
>> selecting variables.
>>
>> We think that this change, though painful, is a necessary way forward,
>> but we want to get input from users, and especially application
>> developers. The latest 4.3 snapshot has these changes, please try it out
>> and let us know what you think, and how it will affect you. Post your
>> comments to these email lists so the entire discussion can be public.
>>
>> thanks
>>
>> John, Ethan
>>
>> _______________________________________________
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>> _______________________________________________
>> idvdevelopers mailing list
>> idvdevelopers@xxxxxxxxxxxxxxxx
>> For list information, to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>
> --
> Don Murray
> NOAA/ESRL/PSD and CIRES
> 303-497-3596
> http://www.esrl.noaa.gov/psd/people/don.murray/
>
> _______________________________________________
> idvdevelopers mailing list
> idvdevelopers@xxxxxxxxxxxxxxxx
> For list information, to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
Faculty of Geo-Information Science and Earth Observation (ITC)
University of Twente
Chamber of Commerce: 501305360000
E-mail disclaimer
The information in this e-mail, including any attachments, is intended for the
addressee only. If you are not the intended recipient, you are hereby notified
that any disclosure, copying, distribution or action in relation to the content
of this information is strictly prohibited. If you have received this e-mail by
mistake, please delete the message and any attachment and inform the sender by
return e-mail. ITC accepts no liability for any error or omission in the
message content or for damage of any kind that may arise as a result of e-mail
transmission.
