Ed,
Getting at the HDF5 error messages is possible, but not easy:
If you build
the
netCDF
C library
with
the
--enable-logging
configure option, rebuild the netCDF Fortran library using that C
library (or maybe it will just work if its a shared library)
, then
put the following
in your code
somewhere before the HDF error occurs
n
f
_set_log_level(3)
y
ou will get
lots
of output
, including various messages on the HDF5 error stack when the error occurs
.
If you change
the "3" to a "1"
, you
get less output, or to a
"
5
", you
get more.
Sometimes this can be helpful in debugging a problem in the HDF5 layer or
a bug in how it's called.
--Russ
On Thu, Jul 24, 2014 at 10:44 AM, David W. Pierce <dpierce@xxxxxxxx> wrote:
> Seems like the null hypothesis is simply that the files are corrupted,
> perhaps because of disk or copying errors. A few bad values in many
> terabytes of data seem consistent with that at first glance. Have you
> tried remaking the source files on a separate disk, if that is
> possible? Or are there other copies elsewhere that can be checksummed
> and compared to your copy?
>
> Or perhaps I'm misunderstanding, and your basic point is that the HDF
> layer should cope with disk errors more gracefully? I don't know how
> the compression works in detail, but I've always imagined that once
> you get a error in a compressed file you can't recover the subsequent
> data very easily. If you *could* recover data subsequent to an error,
> that would imply some level of repeated information, which presumably
> the compression is supposed to remove in the first place.
>
> Regards,
>
> --Dave
>
>
> On Thu, Jul 24, 2014 at 9:26 AM, ezaron <ezaron@xxxxxxx> wrote:
> >
> > Thanks, Rob.
> >
> > The frustrating thing about it is that the error message is completely
> > opaque.
> >
> > I wrap my netcdf calls in a function that reports the error. If the
> return
> > code is not equal to NF_NOERR, then I call nf_strerror. The error occurs
> on
> > a call to nf_get_vara_int, which returns a status of -101. The
> nf_strerror
> > reports this as
> > "NetCDF: HDF error"
> >
> > In case you (or a netcdf developer) can take a look, I have posted an
> > example file here:
> >
> >
> http://maki.cee.pdx.edu/~ezaron/NETCDF4examplefailure/ncom_relo_amseas_2010072600_t015.nc4
> >
> > The error occurs with a call like the following:
> > nf_get_vara_int(ncid,varid,istart4,icount4,tmp)
> > istart4 = (384,371,1,1)
> > icount4 = (2,2,40,1)
> > varid corresponds to the "water_temp" variable
> >
> > It is a very strange error. If I open the file in ncview, for example, I
> can
> > view the other variables without problems. But if I try to view the
> > water_temp variable, ncview crashes and reports "NetCDF: HDF error" also.
> >
> > ncks reports the same error when I try to extract water_temp.
> >
> > But if use "ncdump -v water_temp", the values are extracted from the file
> > without any problems up to a certain point when the program crashes,
> > reporting:
> > NetCDF: HDF error
> > Location: file vardata.c; line 479
> >
> > I see this bug mentioned on the ncks release notes:
> > netCDF #HZY-708311
> >
> http://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2014/msg00045.html
> > and wonder if it is the same problem.
> >
> > -Ed
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> http://netcdf-group.1586084.n2.nabble.com/HDF-error-and-now-what-tp7575504p7575506.html
> > Sent from the NetCDF Group mailing list archive at Nabble.com.
> >
> > _______________________________________________
> > netcdfgroup mailing list
> > netcdfgroup@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
>
>
> --
> David W. Pierce
> Division of Climate, Atmospheric Science, and Physical Oceanography
> Scripps Institution of Oceanography, La Jolla, California, USA
> (858) 534-8276 (voice) / (858) 534-8561 (fax) dpierce@xxxxxxxx
>
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
>
