Re: [netcdfgroup] ZFP compression for NetCDF4

  • To: Leigh Orf <leigh.orf@xxxxxxxxx>
  • Subject: Re: [netcdfgroup] ZFP compression for NetCDF4
  • From: "H. Joe Lee" <hyoklee@xxxxxxxxxxxx>
  • Date: Tue, 23 Mar 2021 12:46:58 -0500
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=hdfgroup.org; dmarc=pass action=none header.from=hdfgroup.org; dkim=pass header.d=hdfgroup.org; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zIQzF79DtIExzIAEb6JTiP7K/e3dHPM2Uo3PXIe1CZ0=; b=i8GtxPF1qiNnVJPLE8WYEMnkj5ZltLZionI4qmDhU+pqArx8NpIcp3XDsRk03nwfs3I8zlCOtNLNJkoq7DtURxYMLJld0QCWAxFTPSMr1kAVcD8gnQ+SXpkT/TAcHebxczpXoUNzk4c/7hHcHm/bnkjf5hHTn4FyQ8c4KxJCiz/pML5I8oNjFNYtZsJ1B2dNoegW/C6aFQ+uU/D+GAE6OX9YiwiNGBsutfNEleqWQOLHgo1Kj8VAJTykHbNNFqjKiATvrXJLy5OVTtd0du6svuNO10qGbK1IjFgo2zmXAl8NHNxq08P2OjEyUk6OLUVFAVNZtVGzI8Wct27TxL8B2w==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=P0nw7BrM9/zaYj+tW1JrS6i9fHvnRXaLQtFg1B8O11+fD/cHvFSDpUYl18Lly+4g81J2WlAdt8JLLVWL07Mum2i5s1iLEOdNVaAPuxmsQRAm/zwhjZ0ijnMYr4qqWtA05hwkeKlrWo3BA9E+ouUEBArhh0iJzGU/CHCtjQl5Biv1Ey+oIXC0UF0nhcWWsMru0ONfgF1Wz3+Fn3dWyPyPP3bVPBM9Q2jyOZvrq63L1hUpKSjccAvgMwq0VTPo5VRMUVmmpLidPFOFzQwJGRk7gQ/A4Tj5aZ6eX/M6nvw7+ry0HeSYYmWtzucRYJYHI8hr+ICfNpgqtsxPs+K9CTZLBg==
  • Authentication-results: unidata.ucar.edu; dkim=none (message not signed) header.d=none;unidata.ucar.edu; dmarc=none action=none header.from=hdfgroup.org;
Please check this out:

https://github.com/HDFGroup/hdf5_plugins/

- Joe
---
Deploy XaaS quickly using HDF in Distributed Cloud.




On Tue, Mar 23, 2021 at 12:03 PM Dennis Heimbigner <dmh@xxxxxxxx> wrote:

> If you have the HDF5 ZFP plugin available, then you can certainly
> use it in netcdf.
> Some documentation on netcdf filter support is here:
>  >https://github.com/Unidata/netcdf/blob/master/NUG/filters.md
>
> =Dennis Heimbigner
>    Unidata
>
>
> On 3/23/2021 10:52 AM, Leigh Orf wrote:
> > Hello,
> >
> > I'm a long time user of HDF and netcdf. I create lots of data in my
> > research (severe weather simulation) and ZFP compression in HDF5 files
> > has been great. I'd like to be able to create netcdf4 files
> > (preferably using the netcdf API) with 3D fields compressed using the
> > lossy ZFP compression algorithm. I know that netcdf4 is HDF5 so it's
> > certainly possible.
> >
> > I'm wondering whether anyone could provide any pointers. I see 3
> > options really.
> >
> > 1. Create ZFP compressed netcdf4 files using the normal netCDF-C API
> > (my code is mostly C)
> > 2. Create ZFP compressed netcdf4 files using the HDF5 API (including
> > all the HDF5 special sauce that makes netcdf4 netcdf4)
> > 3. Use a version of nccopy that understands ZFP
> >
> > Are any of these options possible right now? If not where are we on
> > efforts to include ZFP into netCDF? I did grab the latest code on
> > github and saw references to ZFP but no filter code.
> >
> > Leigh Orf
> > UW-Madison
> >
> > _______________________________________________
> > NOTE: All exchanges posted to Unidata maintained email lists are
> > recorded in the Unidata inquiry tracking system and made publicly
> > available through the web.  Users who post to any of the lists we
> > maintain are reminded to remove any personal information that they
> > do not want to be made public.
> >
> >
> > netcdfgroup mailing list
> > netcdfgroup@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe,  visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
  • 2021 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: