Ed,
Thanks let me know if you need a guinea pig, though the make system of Baselibs
is...odd.
Looks like I am doing something right in a synthetic-y tester. Both level 4 and
-4 seem to work...though -4 is fun in netCDF land:
VAR3D:_Filter = "32015,4294967292" ;
Of course, my tester is all one value across the whole variable, so it
compresses *REAL* well!
Tomorrow I hope to test on real data!
Matt
--
Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
NASA GSFC, Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
Phone: 301-614-6712 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson
From: Ed Hartnett <edwardjameshartnett@xxxxxxxxx>
Date: Tuesday, August 2, 2022 at 4:40 PM
To: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]"
<matthew.thompson@xxxxxxxx>
Cc: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>
Subject: Re: [EXTERNAL] Re: [netcdfgroup] Can I build zstandard support in
netCDF with a static build?
I will take a look at that test in netcdf-fortran...
I guess we all should have collaborated more and agreed to a common configure
option name for the plugin path. :-(
Now that the quantize and zstandard are in netcdf-c/netcdf-fortran, CCR does
not have anything additional to offer at this time. (But BLOSC is coming for
CCR...)
WRT zstandard, here's a chart from a recent AGU presentation:
Quantization and Next-Generation Zlib Compression for Fully
Backward-Compatible, Faster, and More Effective Data Compression in NetCDF
Files,
https://www.researchgate.net/publication/357000984_Quantization_and_Next-Generation_Zlib_Compression_for_Fully_Backward-Compatible_Faster_and_More_Effective_Data_Compression_in_NetCDF_Files<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.researchgate.net%2Fpublication%2F357000984_Quantization_and_Next-Generation_Zlib_Compression_for_Fully_Backward-Compatible_Faster_and_More_Effective_Data_Compression_in_NetCDF_Files&data=05%7C01%7Cmatthew.thompson%40nasa.gov%7Cdca876114e2d468b0f7208da74c72de7%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637950696060139025%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=GdVcdN5ja9zL2aKOkOrBgVRNXhDY6lBbUh9ugtSxuqw%3D&reserved=0>
[cid:image001.png@01D8A691.216FE680]
On Tue, Aug 2, 2022 at 2:20 PM Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND
APPLICATIONS INC] <matthew.thompson@xxxxxxxx<mailto:matthew.thompson@xxxxxxxx>>
wrote:
Ed,
Well, I suppose it was inevitable and I haven't made a massive change in GEOS'
build system in a while...
I went back to my last attempt to build CCR (someday I'll get it working!), and
I think I need to add:
1. HDF5: --with-default-plugindir=$(HDF5_PLUGIN_PATH)
2. netCDF-C: --with-plugin-dir=$(HDF5_PLUGIN_PATH)
3. NCO: --with-hdf5-plugin-path=$(HDF5_PLUGIN_PATH)
I suppose I should ask Charlie Zender about the last one...but it seems right.
Now for some reason I can't get the "make check" step for netCDF-fortran to
recognize HDF5_PLUGIN_PATH, so the zstandard tests fail...but I can run them by
hand. Not sure if it's on my end or I'm not passing things down right to the
check step? :shrug:
Now onto testing it. Be prepared for questions like "What are good values to
set zstandard_level and quantizing and ...". (Well, first I'll go read your
papers and the tests in the code, but then I'll probably annoy you :D )
Matt
--
Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
NASA GSFC, Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
Phone: 301-614-6712 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson
From: Ed Hartnett
<edwardjameshartnett@xxxxxxxxx<mailto:edwardjameshartnett@xxxxxxxxx>>
Date: Tuesday, August 2, 2022 at 1:03 PM
To: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]"
<matthew.thompson@xxxxxxxx<mailto:matthew.thompson@xxxxxxxx>>
Cc: "netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>"
<netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>>
Subject: [EXTERNAL] Re: [netcdfgroup] Can I build zstandard support in netCDF
with a static build?
Howdy Matt!
For zstandard, and other filter features, you must build and use shared
libraries.
Also you need to build netcdf-c with --with-plugins=/some/path and then set
environment variable HDF5_PLUGIN_PATH to /some/path before you build
netcdf-fortran (or use any program with the zstandard filter).
Ed
On Tue, Aug 2, 2022 at 10:57 AM Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND
APPLICATIONS INC] via netcdfgroup
<netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>> wrote:
All,
I recently grabbed netCDF-C v4.9.0 and netCDF-Fortran v4.6.0 and thought, heck,
let's try to get zstd support as well! zstd built just fine and when I built
netCDF-C it seemed pretty happy:
ZSTD Support: yes
but then when I tried to build netCDF-Fortran:
configure: WARNING: ------------------------------------------
configure: WARNING: libnetcdf was built with zstd support, but
HDF5_PLUGIN_PATH is not set.
Either set the HDF5_PLUGIN_PATH environmental variable if zstandard tests
fail,
or use --disable-zstandard-plugin when running configure.
configure: WARNING: ------------------------------------------
Okay. New to me. I then went back to the netCDF-C build and saw:
checking whether dynamically loaded plugins is enabled... yes
configure: WARNING: --disable-shared => --disable-plugins
...
checking whether and where we should install plugins... no
...
Plugin Install Prefix: N.A.
Seeing that warning reminded me I build netCDF-C and netCDF-Fortran as a static
library mainly for historical reasons (was static in the past and, well, if it
works, don't mess with it unless you need to!)
So I suppose the question is: Can I use zstd with a static build of netCDF?
Or is it time to bite the bullet and reconfigure my software to use shared
libraries...
Matt
--
Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
NASA GSFC, Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
Phone: 301-614-6712 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
https://www.unidata.ucar.edu/mailing_lists/<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.unidata.ucar.edu%2Fmailing_lists%2F&data=05%7C01%7Cmatthew.thompson%40nasa.gov%7Cdca876114e2d468b0f7208da74c72de7%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637950696060139025%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=G%2BipPDNTjYMWqTwOOql9TCrNFH0H1OvFwNrgJCYbbQc%3D&reserved=0>
