Re: [netcdf-java] GRIB v2 files

On 6/20/2011 11:07 AM, Comiskey, Glenn wrote:
Hi John,
Thank you kindly for the feedback, much appreciated.
Regarding storing data in NetCDF format, this is not something that locally is wanted to be done as the source files are GRIB v2 and it would require manual intervention to create the NetCDF file. The purpose of the conversion/ncdump header file sent was to show how the GRIB file is wanted to be published, i.e. three dimensions and 16 distinct data variables. Currently, if the GRIB file is published in its native form it is read by NetCDF-Java as a four dimensional/3 data variable file, i.e. dimensions ordered_sequence_of_data, time, lat, lon and only SWELL, SWDIR, SWPER data variables. This differes from earlier versions that read the file as a three dimensional/13 data variable file, i.e. dimensions time, lat, lon and only SWELL, SWDIR, SWPER data variables that were "2 in sequence" - that it say the "1 in sequence" data variables were ignored.

Its a complicated problem. WIthout manual intervention, theres no way for the library to know that those variables should remain 2 dimensional. Heres a blog about it if you are interested:

http://www.unidata.ucar.edu/blogs/developer/en/entry/dataset_schemas_are_lost_in

The thing I find most odd is that given my understanding of the GRIB v2 file specification, the octet that defines "ordered_sequence_of_data" is a locally defined value. In the file sent, it being value 241 (decimal) as defined by NCEP/NOAA (_http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_table4-5.shtml_), and therefore wouldn't have thought NetCDF-Java would have been able to determine what the significance of the octet value meant.

We have NCEP local tables, so we know what this local variable means. Im hoping NCEP (soon!) will publish their local tables so we can stop maintaining our own version. That goes for all WMO centers using local tables.

Thanks for the info. regarding conversion to NetCDF-4 being only a factor of 2. My current 'wgrib2' only allows conversion to NetCDF-3 (classic) hence why the issue of disk capacity. Will source alternate software to be able to convert to NetCDF-4.

Eventually, we hope!

Kind regards,
Glenn

------------------------------------------------------------------------
*From:* John Caron [mailto:caron@xxxxxxxxxxxxxxxx]
*Sent:* 20 June 2011 17:02
*To:* Comiskey, Glenn
*Cc:* netcdf-java@xxxxxxxxxxxxxxxx
*Subject:* Re: [netcdf-java] GRIB v2 files

On 6/20/2011 8:11 AM, Comiskey, Glenn wrote:
John,
While it is possible to present the data in this format having converted the GRIB v2 file to a NetCDF file, as you'll note from the quoted file sizes at the top of header.txt it results in an 11-fold increase in file size. If this was to be used for all GRIB v2 files it would require an enormous increase in storage capacity.
Regards,
Glenn

Hi Glenn:

Ive taken the liberty of cc'ing this to the netcdf-java list, as others may want to hear about this also.

1) The CDM reads the data in its native (GRIB2) format and does the conversion on the fly.

2) If you want to store the data in netCDF, you will get a factor of 10 or more increase in size for netCDF-3 format. The netCDF-4 library (built on HDF-5) allows one to store the data compressed. Our experiments indicate that GRIB2 compression still outperforms this by about a factor of 2. So currently we can reduce your factor of 11 to a factor of 2, if you switch to netCDF-4.

3) AFAIK, mostly this factor of 2 is due to GRIB JPEG-2000 wavelet compression. Eg this is what the data you sent me uses for encoding. We are working on adding this kind of compression to the netcdf-4 C library, and HDF-5 is interested in including this also. Our intention is to make the netCDF-4 format as space efficient as GRIB2. Im not sure if we will run into any roadblocks on this, but we are motivated to remove obstacles for netCDF adoption. I personally think that GRIB-2 should not be used as a long-term archive format, due to problems with tables, and also the kinds of problems that you have reported.

John

  • 2011 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: