[netcdfgroup] FW: Netcdf-4.1.1 fault

Hello NetCDF Group,

In previous mail I have reported Netcdf-4.1.1 fault. Further
investigation shows that this is common NetCDF API problem.
I make a simple test program; it writes a requested number of small
identical items (1000 ints) into NetCDF file. The items are stored in
groups by 10. All items uses the same dummy dimension (may be this is
the root of the problem) in the root group. For all testing I use
HDF5-1.8.4patch1.
The plot attached shows file writing time as a function of number of
items. The traces are marked as: 401 - version 4.0.1 without chunking;
401C - with chunking: chunk size = 1000; 411 - version 4.1.1 without
chunking; 411C - with chunking.
H5 is time of HDF5 file writing; the file contains the same information
except dummy dimension: the item size is stored in the item itself.
The program crashes with two reasons: "HDF error" or "glibc detect:
double free or corruption". The crashes happen when the memory
consumption (by "top") exceeds 3 GB.
The memory consumption for 10000 items is approximately:
401  - 880 MB;
401C - 295 MB, note: better than 401;
411  - 848 MB;
411C - 40000 MB!!! - extrapolated, it crashes at 754 items.
Obviously there is a bug in chunking handling in version 4.1.1. Test
with compression gives practically the same result.
And there is a limit in NetCDF file complexity, at least for existing
API implementation.

Regards,
Sergei

-----Original Message-----
From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx
[mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Shibaev,
Sergei
Sent: 27 October 2010 14:32
To: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: [netcdfgroup] Netcdf-4.1.1 fault


Hi,

I have installed netcdf-4.1.1 with recommended hdf5-1.8.4patch1 and got
an interesting effect. My program writes a complex netcdf4 file which
combines data from many sources including hdf5 files, so both netcdf4
and hdf5 APIs are used simultaneously. The program can write only first
780 data items produced as copies from 4 hdf5 files and fails on
nc_put_vara_short() with error -101: "Error at HDF5 layer"; the hdf
error printout is attached.
And this error is not "at HDF5 layer". When I downgrade netcdf back to
version 4.0.1 (only netcdf, hdf5 is the same 1.8.4patch1) the problem
disappears; the program successfully writes 6200 items, copies 26 hdf
files.
It seems that netcdf4.1.1 interferes with hdf5 library not only at file
closing(netcdf file is opened before hdf5 files and closed last).

Regards,
Sergei


Click
https://www.mailcontrol.com/sr/5c2sCHrq!EHTndxI!oX7UtvqrUUui+B51SNEzqrhG
6z8NVVzf7ejwU5SFfSwNIvPJlWwE0Ej!UdTTjvRhZzcVQ==  to report this email as
spam.

Attachment: nc_crash.png
Description: nc_crash.png

  • 2010 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: