Thanks for the quick suggestions, everyone!
ZFP requires input as to how it is going to compress (there are different
options). In my writeout code (Fortran) use the "h5pset_zfp_accuracy" call
in HDF5 which is applied to chunked data. This call takes a floating point
value that is the required accuracy parameter:
call h5pcreate_f(H5P_DATASET_CREATE_F,chunk_id,ierr);check_err(ierr)
call h5pset_chunk_f(chunk_id,rank,chunkdims,ierr);check_err(ierr)
ierr = h5pset_zfp_accuracy(chunk_id,accuracy);check_err(ierr)
where accuracy=1.0E-4 or something similar.
So it looks like the most promising avenue is to use the nc_def_var_filter
routine:
int nc_def_var_filter(int ncid,
int varid,
unsigned int id,
size_t nparams,
const unsigned int * parms
)
What I don't understand is how to convey the floating point accuracy
parameter to the nc_def_var_filter routine. The only generic arguments
appear to take unsigned int values.
nc_def_var_filter(nc_file_id,nc_variable,H5Z_FILTER_ZFP,cd_nelmts,cd_values);
https://h5z-zfp.readthedocs.io/en/latest/interfaces.html describes how the
accuracy parameter can be applied and other info (there are different ways
to interface with ZFP).
The same issue seems to arise with h5repack (what to pass to the -f flag).
I suspect there may be a way to turn the floating accuracy parameter into a
series of options via cd_values.... perhaps....
Leigh
On Tue, Mar 23, 2021 at 12:03 PM Dennis Heimbigner <dmh@xxxxxxxx> wrote:
> If you have the HDF5 ZFP plugin available, then you can certainly
> use it in netcdf.
> Some documentation on netcdf filter support is here:
> >https://github.com/Unidata/netcdf/blob/master/NUG/filters.md
>
> =Dennis Heimbigner
> Unidata
>
>
> On 3/23/2021 10:52 AM, Leigh Orf wrote:
> > Hello,
> >
> > I'm a long time user of HDF and netcdf. I create lots of data in my
> > research (severe weather simulation) and ZFP compression in HDF5 files
> > has been great. I'd like to be able to create netcdf4 files
> > (preferably using the netcdf API) with 3D fields compressed using the
> > lossy ZFP compression algorithm. I know that netcdf4 is HDF5 so it's
> > certainly possible.
> >
> > I'm wondering whether anyone could provide any pointers. I see 3
> > options really.
> >
> > 1. Create ZFP compressed netcdf4 files using the normal netCDF-C API
> > (my code is mostly C)
> > 2. Create ZFP compressed netcdf4 files using the HDF5 API (including
> > all the HDF5 special sauce that makes netcdf4 netcdf4)
> > 3. Use a version of nccopy that understands ZFP
> >
> > Are any of these options possible right now? If not where are we on
> > efforts to include ZFP into netCDF? I did grab the latest code on
> > github and saw references to ZFP but no filter code.
> >
> > Leigh Orf
> > UW-Madison
> >
> > _______________________________________________
> > NOTE: All exchanges posted to Unidata maintained email lists are
> > recorded in the Unidata inquiry tracking system and made publicly
> > available through the web. Users who post to any of the lists we
> > maintain are reminded to remove any personal information that they
> > do not want to be made public.
> >
> >
> > netcdfgroup mailing list
> > netcdfgroup@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
>
