Heiko,
HDF5 supports reading compressed data in parallel, but cannot write
compressed data in parallel. The netCDF library should return a more
specific error than "HDF5 error" in this case. Here's an explanation
<http://www.hdfgroup.org/hdf5-quest.html#p5comp> for why HDF5 doesn't write
compressed data in parallel.
--Russ
On Tue, Jan 6, 2015 at 5:26 AM, Heiko Klein <Heiko.Klein@xxxxxx> wrote:
> Hi,
>
> we are usually using compressed netcdf4 files for our forecast meteorology
> files. This changes our bottleneck from IO to CPU, since compression
> (usually deflate level 3) takes some time.
>
> To speed this up, I intended to use the mpi-enabled parallel netcdf4/hdf5.
> I've been trying with netcdf 4.3.0 and netcdf 4.3.2 and hdf5-1.8.12 and
> hdf5-1.8.11 (I tried 1.8.13 and 1.8.14, too, but that didn't compile
> because of removed MPIPOSIX in hdf5).
>
> In all cases, I managed to get parallel netcdf working, but I never
> managed to create a deflated variable with parallel netcdf. As soon as I
> set something like
>
> nc_def_var_deflate(ncid, varid, 1, 1, 3)
>
> I get a 'HDF5 error' when I call nc_enddef the next (first) time. Removing
> the deflate command, everything works again. Please find attached your
> tst_parallel4 test-file with the added nc_def_var_deflate command.
>
> Best regards,
>
> Heiko
>
> --
> Dr. Heiko Klein Tel. + 47 22 96 32 58
> Development Section / IT Department Fax. + 47 22 69 63 55
> Norwegian Meteorological Institute http://www.met.no
> P.O. Box 43 Blindern 0313 Oslo NORWAY
>
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
