Hi Quincey,
1. Are there benefits to building HDF without --enable-threadsafe?
As Orion mentioned, the C++ and FORTRAN wrappers aren't currently
compatible with the threadsafe option. I don't think it would be too
hard to address this, but it's probably enough work that we should look
for some funding to do it.
I hope HDF finds the resources to keep the C and Fortran API at
equivalent levels of functionality.
The only other downside really is the increased overhead for each
HDF5 API call (to perform the semaphore locking that allows threadsafety).
Can you post or point me to numbers that quantify this?
If not, can you make it the HDF/netCDF4 default? at least
on multi-core systems?
That's probably difficult for us to detect at configure time. :-/
Yes, easier said than done.
One might try imitating how packages like MPICH2 handle this.
We have learned, I think, to disable our progams' (NCO's)
threading unless it is linked to threadsafe HDF.
Otherwise users will experience unpredictable NCO failure.
2. How should we test, at NCO compile time, whether the
underlying netCDF4/HDF install is threadsafe?
The "H5_HAVE_THREADSAFE" macro will be defined when the "hdf5.h"
header is included and threadsafety is enabled.
Good. This token will help us optimize NCO when built from source.
I wanted our NCO programs to have threading available for general
users but the configuration issues seem to dictate otherwise.
For instance, we distribute NCO .debs and RPMs now with threading
enabled by default because it "just works" with libnetcdf3.a.
With netCDF4, however, pre-built binaries like debs and RPMs
will depend on the distribution's libnetcdf4 .deb and RPM
which likely will not be built on top of HDF --threadsafe.
One outcome is that NCO users who rely on .debs and RPMs over custom
installs will lose NCO threading on (at least) netCDF4 files.
That is actually not
a big deal since threading only speeds up math-heavy jobs (ncwa,
ncap2) because I/O bottlenecks and threading overhead combine to
slow down I/O intensive jobs (ncdiff, ncra,..). It's possible
we may force OMP_NUM_THREADS=1 at run-time when input
or output files are netCDF4 format, and only leave threading
enabled for netCDF3 I/O in pre-built executables.
Charlie
--
Charlie Zender, Department of Earth System Science, UC Irvine
Sab. at CNRS/LGGE-Grenoble until 20080815 :) 011+33+476+824236
Laboratoire de Glaciologie et Géophysique de l'Environnement
54 rue Molière BP 96, 38402 Saint Martin d'Hères Cedex, France
