[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: HDF5 NetCDF4 CDM - bit manipulation





John J Bates wrote:
HDF group, please pass to Mike Folk...thanks

John, we are continuing to look at how we might implement the common data model i/o service provider layer on existing satellite data set formats. When one already has tens of thousands of files, it seems that you might want to write the format to the API using the i/o service provider, what we are calling bringing the API to the data (as opposed to converting all the data to the a new format). The old so-called 1b format satellite data sets are, unfortunately, in bit packed formats. I seem to recall that although HDF5 advertises bit manipulation, you indicated that it was not perhaps really there yet. If this is the case, then we could not bring the API to the data and we would only look at bringing the data to the API (i.e., rewrite all the data into a new format...doable but...).

So, does HDF5 support bit manipulation; within the framework of the CDM? Perhaps Mike knows the answer.

Thanks, John


Hi John:

So there's 2 parts to this:

1) The IOSP that you have to write to "bring the API to the data" (I like it, 
can I steal it?) has to be able to read the file format, bit packing and all, and make it 
available for reading through the API. This is sufficient for giving access to the data.

2) If you want to rewrite the data into Netcfd-4/HDF5, then you want the bit-packing abilities of HDF5 in order to save space. However, even if those arent ready, you can still write to N4, but your files will be bigger. So its an optimization that hopefully we will get soon. As I understand the situation, HDF5 has not yet implemented real bit-packing (eg where 100 11 bit values should take about 1100 bits), however they are working on it. It is possible that current compression schemes will work for your case, but this needs testing.
If you want more details, I can ask Ed Hartnett, out Netcdf-4 expert, to 
provide them.