Thanks, Dave. I'm not an hdf5 power user (yet), but I will check it out.
-Ed
On Jul 24, 2014, at 12:30 PM, Dave Allured - NOAA Affiliate
<dave.allured@xxxxxxxx> wrote:
> Ed,
>
> The HDF group has diagnostic tools for HDF5 files. They should also work for
> netcdf4 files.
>
> http://www.hdfgroup.org/products/hdf5_tools/
>
> I found this sequence to be helpful with your sample file:
>
> h5dump ncom_relo_amseas_2010072600_t015.nc4 >& dump2
> grep -n '[^ ()0-9,.:-]' dump2 > no-data.2
>
> This found a problem in the variable water_temp, and everything else seemed
> okay.
>
> Would these tools help your debugging strategy and reduce your need to
> enhance diagnostics in the Netcdf library?
>
> --Dave
>
> On Thu, Jul 24, 2014 at 12:07 PM, ezaron <ezaron@xxxxxxx> wrote:
> >
> >
> > Thank you for your help, Russ and David.
> >
> > 1) Re: log level
> > This is a great tip. I saw reference to the nf_set_log_level function on a
> > listserv, but I could not find the symbol in the library. Somehow I missed
> > the build option.
> >
> > 2) Re: corrupted file
> > Yes, some bits in the file seem to be corrupted. I can re-download the
> > file, and compress it, and it appears fixed. But it seems that many data
> > values in the file are not corrupted, and it is a pain to have to
> > re-download the whole file (about 24 min per file). The somewhat worrisome
> > part is that nccopy will happily compress the original netcdf without
> > throwing an error. It would be nice if the library provided a little more
> > information for me to read around the bad bits. I suppose that is what I do
> > now, but I wonder if I am throwing out more than necessary.
> >
> > Regards,
> >
> > Ed
> >
> > On Jul 24, 2014, at 10:29 AM, Russ Rew <russ@xxxxxxxxxxxxxxxx> wrote:
> >
> > Ed,
> >
> > Getting at the HDF5 error messages is possible, but not easy:
> >
> > If you build
> > the
> > netCDF
> > C library
> > with
> > the
> > --enable-logging
> > configure option, rebuild the netCDF Fortran library using that C library
> > (or maybe it will just work if its a shared library)
> > , then
> > put the following
> > in your code
> > somewhere before the HDF error occurs
> >
> >
> > n
> > f
> > _set_log_level(3)
> >
> > y
> > ou will get
> > lots
> > of output
> > , including various messages on the HDF5 error stack when the error occurs
> > .
> > If you change
> > the "3" to a "1"
> > , you
> > get less output, or to a
> > "
> > 5
> > ", you
> > get more.
> >
> > Sometimes this can be helpful in debugging a problem in the HDF5 layer or a
> > bug in how it's called.
> >
> > --Russ
> >
> >
> >
> > On Thu, Jul 24, 2014 at 10:44 AM, David W. Pierce <dpierce@xxxxxxxx> wrote:
> >>
> >> Seems like the null hypothesis is simply that the files are corrupted,
> >> perhaps because of disk or copying errors. A few bad values in many
> >> terabytes of data seem consistent with that at first glance. Have you
> >> tried remaking the source files on a separate disk, if that is
> >> possible? Or are there other copies elsewhere that can be checksummed
> >> and compared to your copy?
> >>
> >> Or perhaps I'm misunderstanding, and your basic point is that the HDF
> >> layer should cope with disk errors more gracefully? I don't know how
> >> the compression works in detail, but I've always imagined that once
> >> you get a error in a compressed file you can't recover the subsequent
> >> data very easily. If you *could* recover data subsequent to an error,
> >> that would imply some level of repeated information, which presumably
> >> the compression is supposed to remove in the first place.
> >>
> >> Regards,
> >>
> >> --Dave
> >>
> >>
> >> On Thu, Jul 24, 2014 at 9:26 AM, ezaron <ezaron@xxxxxxx> wrote:
> >> >
> >> > Thanks, Rob.
> >> >
> >> > The frustrating thing about it is that the error message is completely
> >> > opaque.
> >> >
> >> > I wrap my netcdf calls in a function that reports the error. If the
> >> > return
> >> > code is not equal to NF_NOERR, then I call nf_strerror. The error occurs
> >> > on
> >> > a call to nf_get_vara_int, which returns a status of -101. The
> >> > nf_strerror
> >> > reports this as
> >> > "NetCDF: HDF error"
> >> >
> >> > In case you (or a netcdf developer) can take a look, I have posted an
> >> > example file here:
> >> >
> >> > http://maki.cee.pdx.edu/~ezaron/NETCDF4examplefailure/ncom_relo_amseas_2010072600_t015.nc4
> >> >
> >> > The error occurs with a call like the following:
> >> > nf_get_vara_int(ncid,varid,istart4,icount4,tmp)
> >> > istart4 = (384,371,1,1)
> >> > icount4 = (2,2,40,1)
> >> > varid corresponds to the "water_temp" variable
> >> >
> >> > It is a very strange error. If I open the file in ncview, for example, I
> >> > can
> >> > view the other variables without problems. But if I try to view the
> >> > water_temp variable, ncview crashes and reports "NetCDF: HDF error" also.
> >> >
> >> > ncks reports the same error when I try to extract water_temp.
> >> >
> >> > But if use "ncdump -v water_temp", the values are extracted from the file
> >> > without any problems up to a certain point when the program crashes,
> >> > reporting:
> >> > NetCDF: HDF error
> >> > Location: file vardata.c; line 479
> >> >
> >> > I see this bug mentioned on the ncks release notes:
> >> > netCDF #HZY-708311
> >> > http://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2014/msg00045.html
> >> > and wonder if it is the same problem.
> >> >
> >> > -Ed
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >> > http://netcdf-group.1586084.n2.nabble.com/HDF-error-and-now-what-tp7575504p7575506.html
> >> > Sent from the NetCDF Group mailing list archive at Nabble.com.
> >> >
> >> > _______________________________________________
> >> > netcdfgroup mailing list
> >> > netcdfgroup@xxxxxxxxxxxxxxxx
> >> > For list information or to unsubscribe, visit:
> >> > http://www.unidata.ucar.edu/mailing_lists/
> >>
> >>
> >>
> >> --
> >> David W. Pierce
> >> Division of Climate, Atmospheric Science, and Physical Oceanography
> >> Scripps Institution of Oceanography, La Jolla, California, USA
> >> (858) 534-8276 (voice) / (858) 534-8561 (fax) dpierce@xxxxxxxx
> >>
> >> _______________________________________________
> >> netcdfgroup mailing list
> >> netcdfgroup@xxxxxxxxxxxxxxxx
> >> For list information or to unsubscribe, visit:
> >> http://www.unidata.ucar.edu/mailing_lists/
> >>
> >
> >
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > Edward D. Zaron
> > Research Assistant Professor
> > Department of Civil and Environmental Engineering
> > Portland State University
> > Portland, OR 97207-0751
> > Phone: (503)-725-2435
> > FAX: (503)-725-5950
> > ezaron@xxxxxxx
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >
> > _______________________________________________
> > netcdfgroup mailing list
> > netcdfgroup@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> > http://www.unidata.ucar.edu/mailing_lists/
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Edward D. Zaron
Research Assistant Professor
Department of Civil and Environmental Engineering
Portland State University
Portland, OR 97207-0751
Phone: (503)-725-2435
FAX: (503)-725-5950
ezaron@xxxxxxx
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
