« Compression by Bit-S... | Main | Part 5: Converting... »

Both GRIB1 and GRIB2 have an optional bitmap section which is a bit array for marking missing data. There is a single bit for each data value, so a data array of N points requires N/8 bytes for the bitmap array. A bit value of 0 indicates that the data is missing at that point, so that data value doesn't have to be stored.

Seems like that has to be a good idea, and the more missing values there are, the better. Right?

Turns out that combined with compression, its not such a good idea, and the more missing values there are, the worse it is. The problem is that the bitmap section is not compressed, only the data section is.

Heres an extreme case, NOAA wave watch data off the west coast and including Hawaii. The grid size is 526 x 736, giving 387,136 points, most of which are missing:

Examining a random record in ToolsUI IOSP/GRIB2/GRIB2data, we see that there are only 11,127 non-missing data values. These are compressed nicely into 7702 bytes by GRIB wavelet compression. The problem is that we need 387136/8 = 48392 bytes to store the bitmap, which is not compressed. That makes the entire GRIB message 56270 bytes.

I didn't really think about it until I noticed with my compression tests that other compression schemes are beating GRIB by factors greater than 20 on some records, which is surprising. Using bzip2, for that example record, the entire array including missing values is compressed to a size of 9576 bytes. Deflate (zip) compression compresses it into 25786 bytes. LZMA (7zip) gets it down to 8994, which is 6 times smaller.

Overall, on that entire file, the estimated file sizes for various compression schemes are:

Standard compression algorithms are very, very good at compressing repeated bytes of data. The more missing data, the better they do. On that file, every record was at least a factor or 2 smaller for bzip2 and 7zip compression, than the GRIB record size.

I didn't test this, but its likely the JPEG-2000 wavelet compression would do much better at compressing the data with missing values in it, compared to the current technique of removing the missing values with an uncompressed bitmap.

(PS: I just noticed that bitmaps can be shared between records, by using the 'repeating section' feature of GRIB2. This will mitigate the above conclusion by an unknown amount).

Here's a gratuitous shot of the ToolsUI IOSP/GRIB2/GRIB2data tab on that example file (click on image for more detail):

If you right click on one of the records, and choose "Compute Scale/offset of data" from the context menu, you can see some alternate compression sizes for that record. For example the example record shows:

 nbits = 10
 npoints = 387136
 width = 1022 (0x3fe) 
 scale = 0.0100000 
 resolution = 0.00500000 
 range = 10.230000 

           actual    computed
 dataMin = 1.390000 1.390000
 dataMax = 11.250000 11.620000
 actual range = 9.860000
 scale_factor = 0.00964775
 add_offset = 1.39000

 max_diff = 0.00481459
 avg_diff = 7.18643e-05
 std_diff = 0.000479742

Compression
 number of values = 387136
 uncompressed as floats = 1548544
 uncompressed packed bits = 483920
 grib data length = 7702
 grib msg length = 56270

deflate (float)
 compressedSize = 19374
 ratio floats / size = 79.928978
 ratio packed bits / size = 24.977806
 ratio size / grib = 0.344304

deflate (scaled ints)
 compressedSize = 16339
 ratio floats / size = 94.775932
 ratio packed bits / size = 29.617479
 ratio size / grib = 0.290368

bzip2 (floats)
 compressedSize = 9771
 ratio floats / size = 158.483673
 ratio packed bits / size = 49.526150
 ratio size / grib = 0.173645

bzip2 (scaled ints)
 compressedSize = 9455
 ratio floats / size = 163.780426
 ratio packed bits / size = 51.181385
 ratio size / grib = 0.168029

Lots of other info is available from that context menu. Welcome to the inner workings of GRIB sausage making.