Dennis,
Great! Glad to know you are working on it! NetCDF C API is the only one
I care about.
Your scenario 1 is the one I'm using to hide latency of Lustre file
system when we need to extract metadata from large dataset. We have some
that contain 10k files. A pool of threads picks away at the list of
NetCDF files, only a single thread accesses any given file. We hold a
lock while we open and close the file, what happens in between open and
close, the reading and parsing of the metadata, occurs freely. Only 1
thread is touching any given file. This is the scenario where I need
thread safety baked into HDF5, or I end up intermittently crashing.
To get more info about the complication of configuring HDF5 thread safe
with the HDF5 HL API try to configure HDF5 with both thread safety and
the HL API. here is the output from hdf5 1.8.17
./configure --enable-threadsafe --enable-hl
checking for thread safe support... configure: error: The thread-safe
library is incompatible with the high-level library. --disable-hl can be
used to prevent building the high-level library (recommended).
Alternatively, --enable-unsupported will allow building the high-level
library, though this configuration is not supported by The HDF Group.
Building in the unsupported mode is what we are currently doing to work
around the issue. I don't know what the plan for Hdf5 HL API and theads
is, but perhaps NetCDF should make use of the "low level" HDF5 API and
avoid the issue altogether?
Your scenario 2 is also of interest. in that scenario I currently
protect all nc calls and this serializes the I/O but still lets our
calculations run in parallel. The calcs take much longer than the I/O so
even though I/O is serialized this still works out well.
Burlen
On 07/18/2016 03:23 PM, dmh@xxxxxxxx wrote:
> Unfortunately HDF5's thread safe config is officially mutually
> exclusive with the HDF5 HL API used by NetCDF.
I was unaware of this. My understanding was that the thread-safe
HDf5 operated by providing a global lock to serialize all
accesses. Assuming I am correct, you seem to be saying that this
lock is not used at the HL API level.
To date WRT netcdf, there have been two notions of thread-safe:
1. Allow multiple threads to operate as long as they are operating
on different files.
2. Allow multiple threads to operate on the same file.
#1 is doable -- just time consuming to implement.
In fact I have a netcdf-c branch that should allow this for
netcdf 3 (classic) files. The approach is to isolate all mutable
global state used by the library and surround operations on that
state (both read and write) with a mutex lock. Since none of the
state accesses are all that long, this should not affect
performance very much. Note that an implicit assumption is that
all c-library calls (esp. malloc) are or can be made thread-safe.
This approach might also work for netcdf-4 files
except that we are limited by what the HDF5 library does.
If there API is globally serialized, then our locking regime
will not help.
There is no obvious reason AFIAK why the HDF5 library could not
be modified to do a similar isolation of global state. Note this
issue crops up for the pnetcdf library also.
#2 is much harder and would require significant refactoring of
any library that attempted it. The reason is that access to EVERY
piece of state (global or not) must be made thread-safe.
Finally, note that this issue is largely independent of parallel IO
using e.g. MPIO.
I look forward to further discussion of this issue; especially
any complication I might be overlooking.
=Dennis Heimbigner
Unidata
On 7/18/2016 12:24 PM, Burlen Loring wrote:
Hi All,
Just wanted to voice concern about the status of thread safety in NetCDF
4 HDF5. The locking strategy we've successfully used with NetCDF classic
is not sufficient for NetCDF 4 with HDF5. In addition to our locking
strategy HDF5 needs to be compiled with a thread safe option.
Unfortunately HDF5's thread safe config is officially mutually exclusive
with the HDF5 HL API used by NetCDF. When HDF5 is forced to compile with
thread safety and HDF5 HL API, our threaded code runs without issue. It
also performs well, which is important. My concern is the fact that we
now rely upon a build configuration that is officially unsupported by
HDF5.
Given the continual evolution to many core architectures, the horrendous
latency on modern parallel file systems on super computing platforms,
and that we have to deal with datasets structured such that latency is a
major issue, threading is ever more critical. It's really important that
we have a viable path to thread safety that is officially supported by
HDF5 and performant. We don't want to be facing problems down the road
due to use of the unsupported HDF5 config. Using the unsupported config
creates a deployment issue as we'd like to rely on HDF5 installed at HPC
centers or in official Linux distros, neither of whom will likely be
compiling HDF5 in an unsupported configuration. I also believe that for
the best performance locking is better done at the lowest level where it
can be fine grained, hence locking all NetCDF I/O in our application is
undesirable.
I'm hoping that this conversation can be a data point that people are
using threads to speed processing of large datasets on parallel file
systems. It's important for us to have an officially supported thread
safe option for NetCDF 4 HDF5 format.
Burlen
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
