NOTE: The netcdf-hdf mailing list is no longer active. The list archives are made available for historical reasons.
Hi Charlie, On May 23, 2008, at 8:42 AM, Charlie Zender wrote:
Hi Quincey,As Orion mentioned, the C++ and FORTRAN wrappers aren't currently compatible with the threadsafe option. I don't think it would be too hard to address this, but it's probably enough work that we should look for some funding to do it.1. Are there benefits to building HDF without --enable-threadsafe?I hope HDF finds the resources to keep the C and Fortran API at equivalent levels of functionality.
We do try, yes. For instance the upcoming 1.8.1 release will have many new FORTRAN wrappers for new 1.8 API routines.
The only other downside really is the increased overhead for each HDF5 API call (to perform the semaphore locking that allows threadsafety).Can you post or point me to numbers that quantify this?
I don't have any, sorry.
If not, can you make it the HDF/netCDF4 default? at least on multi-core systems?That's probably difficult for us to detect at configure time. :-/Yes, easier said than done. One might try imitating how packages like MPICH2 handle this.
Hmm, I'm not familiar with how MPICH2 works for this. Do you have a pointer?
The "H5_HAVE_THREADSAFE" macro will be defined when the "hdf5.h" header is included and threadsafety is enabled.We have learned, I think, to disable our progams' (NCO's) threading unless it is linked to threadsafe HDF. Otherwise users will experience unpredictable NCO failure. 2. How should we test, at NCO compile time, whether the underlying netCDF4/HDF install is threadsafe?Good. This token will help us optimize NCO when built from source. I wanted our NCO programs to have threading available for general users but the configuration issues seem to dictate otherwise. For instance, we distribute NCO .debs and RPMs now with threading enabled by default because it "just works" with libnetcdf3.a. With netCDF4, however, pre-built binaries like debs and RPMs will depend on the distribution's libnetcdf4 .deb and RPM which likely will not be built on top of HDF --threadsafe. One outcome is that NCO users who rely on .debs and RPMs over custom installs will lose NCO threading on (at least) netCDF4 files. That is actually not a big deal since threading only speeds up math-heavy jobs (ncwa, ncap2) because I/O bottlenecks and threading overhead combine to slow down I/O intensive jobs (ncdiff, ncra,..). It's possible we may force OMP_NUM_THREADS=1 at run-time when input or output files are netCDF4 format, and only leave threading enabled for netCDF3 I/O in pre-built executables.
Even when a threadsafe HDF5 is used, and NCO is only reading from netCDF-4 files, you could still have problems for multi-threaded access. For instance, there are probably some global data structures in netCDF-4 that won't be protected by a semaphore and could get corrupted. Maybe Ed can think about this and offer an opinion...
Quincey
Charlie -- Charlie Zender, Department of Earth System Science, UC Irvine Sab. at CNRS/LGGE-Grenoble until 20080815 :) 011+33+476+824236 Laboratoire de Glaciologie et Géophysique de l'Environnement 54 rue Molière BP 96, 38402 Saint Martin d'Hères Cedex, France
netcdf-hdf archives: