Hi Russ,
thanks for your answer and the explanation from the hdfgroup.
For our purposes, it is not the parallel IO, which is important, but the
bottleneck is in many cases zlib. Are there other ways to parallelize
the 'zlib' portion of netcdf/hdf?
Heiko
PS: Concerning unclear error messages: When linking a non-parallel
netcdf-library with a program using nc_create_par, I got a error message
like: "Parallel operation on file opened for non-parallel access." That
message didn't help me to understand that I forgot to set LD_LIBRARY_PATH.
On 2015-01-06 15:46, Russ Rew wrote:
Heiko,
HDF5 supports reading compressed data in parallel, but cannot write
compressed data in parallel. The netCDF library should return a more
specific error than "HDF5 error" in this case. Here's an explanation
<http://www.hdfgroup.org/hdf5-quest.html#p5comp> for why HDF5 doesn't
write compressed data in parallel.
--Russ
On Tue, Jan 6, 2015 at 5:26 AM, Heiko Klein <Heiko.Klein@xxxxxx
<mailto:Heiko.Klein@xxxxxx>> wrote:
Hi,
we are usually using compressed netcdf4 files for our forecast
meteorology files. This changes our bottleneck from IO to CPU, since
compression (usually deflate level 3) takes some time.
To speed this up, I intended to use the mpi-enabled parallel
netcdf4/hdf5. I've been trying with netcdf 4.3.0 and netcdf 4.3.2
and hdf5-1.8.12 and hdf5-1.8.11 (I tried 1.8.13 and 1.8.14, too, but
that didn't compile because of removed MPIPOSIX in hdf5).
In all cases, I managed to get parallel netcdf working, but I never
managed to create a deflated variable with parallel netcdf. As soon
as I set something like
nc_def_var_deflate(ncid, varid, 1, 1, 3)
I get a 'HDF5 error' when I call nc_enddef the next (first) time.
Removing the deflate command, everything works again. Please find
attached your tst_parallel4 test-file with the added
nc_def_var_deflate command.
Best regards,
Heiko
--
Dr. Heiko Klein Tel. + 47 22 96 32 58
<tel:%2B%2047%2022%2096%2032%2058>
Development Section / IT Department Fax. + 47 22 69 63 55
<tel:%2B%2047%2022%2069%2063%2055>
Norwegian Meteorological Institute http://www.met.no
P.O. Box 43 Blindern 0313 Oslo NORWAY
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
Dr. Heiko Klein Tel. + 47 22 96 32 58
Development Section / IT Department Fax. + 47 22 69 63 55
Norwegian Meteorological Institute http://www.met.no
P.O. Box 43 Blindern 0313 Oslo NORWAY
