On 6/20/2011 8:11 AM, Comiskey, Glenn wrote:
John,
While it is possible to present the data in this format having
converted the GRIB v2 file to a NetCDF file, as you'll note from the
quoted file sizes at the top of header.txt it results in an 11-fold
increase in file size. If this was to be used for all GRIB v2 files it
would require an enormous increase in storage capacity.
Regards,
Glenn
Hi Glenn:
Ive taken the liberty of cc'ing this to the netcdf-java list, as others
may want to hear about this also.
1) The CDM reads the data in its native (GRIB2) format and does the
conversion on the fly.
2) If you want to store the data in netCDF, you will get a factor of 10
or more increase in size for netCDF-3 format. The netCDF-4 library
(built on HDF-5) allows one to store the data compressed. Our
experiments indicate that GRIB2 compression still outperforms this by
about a factor of 2. So currently we can reduce your factor of 11 to a
factor of 2, if you switch to netCDF-4.
3) AFAIK, mostly this factor of 2 is due to GRIB JPEG-2000 wavelet
compression. Eg this is what the data you sent me uses for encoding. We
are working on adding this kind of compression to the netcdf-4 C
library, and HDF-5 is interested in including this also. Our intention
is to make the netCDF-4 format as space efficient as GRIB2. Im not sure
if we will run into any roadblocks on this, but we are motivated to
remove obstacles for netCDF adoption. I personally think that GRIB-2
should not be used as a long-term archive format, due to problems with
tables, and also the kinds of problems that you have reported.
John
