[netcdfgroup] status of thread safety
Hi All,
Just wanted to voice concern about the status of thread safety in NetCDF
4 HDF5. The locking strategy we've successfully used with NetCDF classic
is not sufficient for NetCDF 4 with HDF5. In addition to our locking
strategy HDF5 needs to be compiled with a thread safe option.
Unfortunately HDF5's thread safe config is officially mutually exclusive
with the HDF5 HL API used by NetCDF. When HDF5 is forced to compile with
thread safety and HDF5 HL API, our threaded code runs without issue. It
also performs well, which is important. My concern is the fact that we
now rely upon a build configuration that is officially unsupported by HDF5.
Given the continual evolution to many core architectures, the horrendous
latency on modern parallel file systems on super computing platforms,
and that we have to deal with datasets structured such that latency is a
major issue, threading is ever more critical. It's really important that
we have a viable path to thread safety that is officially supported by
HDF5 and performant. We don't want to be facing problems down the road
due to use of the unsupported HDF5 config. Using the unsupported config
creates a deployment issue as we'd like to rely on HDF5 installed at HPC
centers or in official Linux distros, neither of whom will likely be
compiling HDF5 in an unsupported configuration. I also believe that for
the best performance locking is better done at the lowest level where it
can be fine grained, hence locking all NetCDF I/O in our application is
undesirable.
I'm hoping that this conversation can be a data point that people are
using threads to speed processing of large datasets on parallel file
systems. It's important for us to have an officially supported thread
safe option for NetCDF 4 HDF5 format.
Burlen
