OK I have had success in writing ZFP necdf4 files using the netcdf-c API,
yay!
To reiterate, I want to use the netcdf-C API to compress data using the
accuracy approach, where you supply a floating point value that is the
level of accuracy you need in your 3D variable. Each 3D field has its own
accuracy parameter that the user has to decide, based upon how much
accuracy they really need.
So if you want to do this:
Build and install hdf5, netcdf4 (this may already be done)
Build ZFP and h5z-zfp. Get both from github. Read carefully about how to
configure zfp for H5Z-ZFP (in Config).
Stick libh5zzfp.so (installed when you did make install on the H5Z-ZFP
code) in a directory, set HDF5_PLUGIN_PATH=/path/to/that/directory
There is a utility that comes with H5Z-ZFP called
print_h5repack_farg_zfpmode. It's just for convenience. To test that I
could write ZFP correctly I chose zfpmode 3 and accuracy 2.0 to get the
magical arguments:
% print_h5repack_farg zfpmode=3 acc=2.0
h5repack -f argument...
*-f UD=32013,4,3,0,0,1073741824*
I tried that and for some reason h5repack failed to write my compressed
variables properly. There is a patch for h5repack that comes with the
H5Z-ZFP plugin that I did not apply as I am using an installed version of
h5repack, it's likely that's the problem.
But I used the above information from print_h5repack_farg_zfpmodule to pass
to nc_dev_var_filter:
#define ZFP_ID 32013
const unsigned int zfpargs[4] = {3,0,0,1073741824};
status = nc_def_var_filter(nc->ncid,nc->varnameid[ivar],ZFP_ID,4,zfpargs);
And that worked! I have a nice small file compared to uncompressed that
views nicely with ncview (how I initially interrogate my data, usually).
Instead of using the above utility, I can set the cdvalues using
(effectively) the H5P_set_zfp_accuracy_cdata macro found in H5Zzfp_plugin.h
in the H5Z-ZFP code:
#define set_zfp_accuracy_cdata(A, CD) \
{ double *p = (double *) &CD[2]; \
CD[0]=CD[1]=CD[2]=CD[3]=0; \
CD[0]=3; *p=A;}
In my test code:
acc=2.0
set_zfp_accuracy_cdata(acc,cdata);
for (i=0;i<4;i++)
{
printf("cdata[%i]=%i\n",i,cdata[i]);
}
produces:
fuggle: /home/orf/2021/projects/hdf2nc-zfp ./bitfarts
cdata[0]=3
cdata[1]=0
cdata[2]=0
cdata[3]=1073741824
which gives us the arguments we need for nc_def_var_filter
Leigh
On Tue, Mar 23, 2021 at 1:13 PM Leigh Orf <leigh.orf@xxxxxxxxx> wrote:
> OK perhaps I may have figured it out...
>
> With ZFP you can invoke the accuracy parameter option in two ways, one
> that is intuitive (floating point value) and one that is not. Of course I
> require the non-intuitive one:
>
> *H5Pset_zfp_accuracy_cdata(acc, cd_nelmts, cd_values);*
>
> *cd_values index:*
> * 0 1 2 3 4 5*
> accuracy: 3 unused *accA accB* unused unused
>
> *accA* and *aacB* are "high/low 32-bit words of a double"
>
> Ah hah! So I think I just need to mangle my 64 bit floating point accuracy
> value and chop it into high and low parts (cast as unsigned int or
> something), and pass them as options 2 and 3 into the 6 element cd_values
> array?
>
> I believe then those would be the arguments passed to
>
> nc_def_var_filter(nc_file_id,nc_variable,H5Z_FILTER_ZFP,cd_nelmts,
> *cd_values*);
>
> That is what I will try...
>
> Leigh
>
> On Tue, Mar 23, 2021 at 12:52 PM Leigh Orf <leigh.orf@xxxxxxxxx> wrote:
>
>> Joe,
>>
>> Thanks, when I am completely in HDF5 land, I can do the ZFP stuff!
>>
>> I don't know how to twiddle the ZFP settings in netcdf API land... and I
>> don't know how to make netCDF4 files in HDF5 API land...
>>
>> On Tue, Mar 23, 2021 at 12:47 PM H. Joe Lee <hyoklee@xxxxxxxxxxxx> wrote:
>>
>>> Please check this out:
>>>
>>> https://github.com/HDFGroup/hdf5_plugins/
>>>
>>> - Joe
>>> ---
>>> Deploy XaaS quickly using HDF in Distributed Cloud.
>>>
>>>
>>>
>>>
>>> On Tue, Mar 23, 2021 at 12:03 PM Dennis Heimbigner <dmh@xxxxxxxx> wrote:
>>>
>>>> If you have the HDF5 ZFP plugin available, then you can certainly
>>>> use it in netcdf.
>>>> Some documentation on netcdf filter support is here:
>>>> >https://github.com/Unidata/netcdf/blob/master/NUG/filters.md
>>>>
>>>> =Dennis Heimbigner
>>>> Unidata
>>>>
>>>>
>>>> On 3/23/2021 10:52 AM, Leigh Orf wrote:
>>>> > Hello,
>>>> >
>>>> > I'm a long time user of HDF and netcdf. I create lots of data in my
>>>> > research (severe weather simulation) and ZFP compression in HDF5
>>>> files
>>>> > has been great. I'd like to be able to create netcdf4 files
>>>> > (preferably using the netcdf API) with 3D fields compressed using the
>>>> > lossy ZFP compression algorithm. I know that netcdf4 is HDF5 so it's
>>>> > certainly possible.
>>>> >
>>>> > I'm wondering whether anyone could provide any pointers. I see 3
>>>> > options really.
>>>> >
>>>> > 1. Create ZFP compressed netcdf4 files using the normal netCDF-C API
>>>> > (my code is mostly C)
>>>> > 2. Create ZFP compressed netcdf4 files using the HDF5 API (including
>>>> > all the HDF5 special sauce that makes netcdf4 netcdf4)
>>>> > 3. Use a version of nccopy that understands ZFP
>>>> >
>>>> > Are any of these options possible right now? If not where are we on
>>>> > efforts to include ZFP into netCDF? I did grab the latest code on
>>>> > github and saw references to ZFP but no filter code.
>>>> >
>>>> > Leigh Orf
>>>> > UW-Madison
>>>> >
>>>> > _______________________________________________
>>>> > NOTE: All exchanges posted to Unidata maintained email lists are
>>>> > recorded in the Unidata inquiry tracking system and made publicly
>>>> > available through the web. Users who post to any of the lists we
>>>> > maintain are reminded to remove any personal information that they
>>>> > do not want to be made public.
>>>> >
>>>> >
>>>> > netcdfgroup mailing list
>>>> > netcdfgroup@xxxxxxxxxxxxxxxx
>>>> > For list information or to unsubscribe, visit:
>>>> https://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>> _______________________________________________
>>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>>> recorded in the Unidata inquiry tracking system and made publicly
>>>> available through the web. Users who post to any of the lists we
>>>> maintain are reminded to remove any personal information that they
>>>> do not want to be made public.
>>>>
>>>>
>>>> netcdfgroup mailing list
>>>> netcdfgroup@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe, visit:
>>>> https://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>
