Seems like the null hypothesis is simply that the files are corrupted,
perhaps because of disk or copying errors. A few bad values in many
terabytes of data seem consistent with that at first glance. Have you
tried remaking the source files on a separate disk, if that is
possible? Or are there other copies elsewhere that can be checksummed
and compared to your copy?
Or perhaps I'm misunderstanding, and your basic point is that the HDF
layer should cope with disk errors more gracefully? I don't know how
the compression works in detail, but I've always imagined that once
you get a error in a compressed file you can't recover the subsequent
data very easily. If you *could* recover data subsequent to an error,
that would imply some level of repeated information, which presumably
the compression is supposed to remove in the first place.
Regards,
--Dave
On Thu, Jul 24, 2014 at 9:26 AM, ezaron <ezaron@xxxxxxx> wrote:
>
> Thanks, Rob.
>
> The frustrating thing about it is that the error message is completely
> opaque.
>
> I wrap my netcdf calls in a function that reports the error. If the return
> code is not equal to NF_NOERR, then I call nf_strerror. The error occurs on
> a call to nf_get_vara_int, which returns a status of -101. The nf_strerror
> reports this as
> "NetCDF: HDF error"
>
> In case you (or a netcdf developer) can take a look, I have posted an
> example file here:
>
> http://maki.cee.pdx.edu/~ezaron/NETCDF4examplefailure/ncom_relo_amseas_2010072600_t015.nc4
>
> The error occurs with a call like the following:
> nf_get_vara_int(ncid,varid,istart4,icount4,tmp)
> istart4 = (384,371,1,1)
> icount4 = (2,2,40,1)
> varid corresponds to the "water_temp" variable
>
> It is a very strange error. If I open the file in ncview, for example, I can
> view the other variables without problems. But if I try to view the
> water_temp variable, ncview crashes and reports "NetCDF: HDF error" also.
>
> ncks reports the same error when I try to extract water_temp.
>
> But if use "ncdump -v water_temp", the values are extracted from the file
> without any problems up to a certain point when the program crashes,
> reporting:
> NetCDF: HDF error
> Location: file vardata.c; line 479
>
> I see this bug mentioned on the ncks release notes:
> netCDF #HZY-708311
> http://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2014/msg00045.html
> and wonder if it is the same problem.
>
> -Ed
>
>
>
>
>
> --
> View this message in context:
> http://netcdf-group.1586084.n2.nabble.com/HDF-error-and-now-what-tp7575504p7575506.html
> Sent from the NetCDF Group mailing list archive at Nabble.com.
>
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
--
David W. Pierce
Division of Climate, Atmospheric Science, and Physical Oceanography
Scripps Institution of Oceanography, La Jolla, California, USA
(858) 534-8276 (voice) / (858) 534-8561 (fax) dpierce@xxxxxxxx
