Hello all,
I just wanted to jump in and comment that this issue, recently reported to
us by David Knaak at Cray, is now handled in the netCDF-C development
branch on GitHub. This fix will be in the upcoming release candidate and
eventual final release of netCDF-C 4.4.0.
Regarding the question of short reads providing more warning; netcdf
specifically was already checking for short reads when ‘paging in’ data
from a file, but was assuming an error when one would occur (due to a
non-zero errno value). The fix shouldn’t incur any performance penalty. The
new thing I learned about “short reads” is that it is possible for this to
occur *without* being the result of an error, but rather the result of an
interrupt.
In general, since they are technically allowed I think developers are going
to have to accommodate the possibility of short reads in their software,
one way or another. Developers should already be checking the return value
of read(), and when short, the fix is essentially:
1. Check to see if errno is EINTR
2. If so, perform some calculations and resume the read.
This works as a “true” error would have a different non-zero errno value.
But I’m sure it will depend on the individual application.
- The details of the bug report from David and the fix can be found in
the Unidata JIRA system at
https://bugtracking.unidata.ucar.edu/browse/NCF-337.
- The (ultimately simple) fix can be viewed at
https://github.com/Unidata/netcdf-c/commit/382b24c2c9739f69c20f09bf14f9d9ca11b73e1.
- More information about short reads (and the read() function in
general) may be found at
http://pubs.opengroup.org/onlinepubs/009695399/functions/read.html.
Thanks all, have a great day,
-Ward
Ward Fisher
wfisher@xxxxxxxx
UCAR/Unidata - Software Engineer
On Tue, Aug 18, 2015 at 1:05 PM, Ted Mansell <ted.mansell@xxxxxxxx> wrote:
So it turned out to be a Lustre issue:
>
>
> https://jira.hpdd.intel.com/browse/LU-6389?focusedCommentId=110278&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-110278
>
> The bug affects Lustre 2.5.x and the system I was using had just been
> updated from Lustre 2.4.3 to 2.5.37. The sysadmins tested a patch on a
> couple nodes, and that fixed the problem.
>
> Many thanks to Wei-keng Liao for helping to identify Lustre as the culprit!
>
> It brings up the issue, however, of whether netcdf and other applications
> should provide more warning when "short reads" are encountered? In this
> case a 1MB "slice" of data was missing from the requested data. Should
> netcdf be checking for this and report errors? Or does this happen too
> rarely? Does checking it therefore have too much overhead? The philosophy
> for Lustre is that it should never deliver short reads because applications
> normally do not expect (or handle) them, even though the Posix spec
> technically allows it to happen.
>
> Cheers,
>
> -- Ted
>
> __________________________________________________________
> | Edward Mansell <ted.mansell@xxxxxxxx>
> | National Severe Storms Laboratory
> |--------------------------------------------------------------
> | "The contents of this message are mine personally and
> | do not reflect any position of the U.S. Government or NOAA."
> |--------------------------------------------------------------
>
>
> On Aug 2, 2015, at 10:46 AM, Ted Mansell <Ted.Mansell@xxxxxxxx> wrote:
>
> > Update: I am wondering if it is an issue with file striping on Lustre.
> The files are writing through direct pnetcdf calls with striping set to 16:
> >
> > lfs setstripe -c 16 .
> >
> > When I cp a cf-2 file to a higher directory (not striped), perhaps this
> rearranges the storage, because there are no reading issues with the copy,
> even after mv'ing the copy back to the original directory. (This is for
> reading either with a serial application like ncview or parallel read via
> the netcdf4 interface to pnetcdf.)
> >
> > -- Ted
> >
> >> Howdy,
> >>
> >> I seeing some strange behavior in reading large files, and I'm
> wondering if anybody has similar experience. This is on a Lustre file
> system that was upgraded in June (I assume to the latest version, but I
> will check that). I had no previous problems. Anyway, the symptom is that I
> am reading reasonably large float arrays (400 x 400 x 130), and randomly
> some parts of the array will not be read in, leaving the initialized (zero)
> values. The arrays are ordered x,y,z (fortran convention), and the failures
> usually show up as partial or complete x-y planes. A file might have 15 to
> 500 arrays of this size and one time level.
> >>
> >> I am writing the files with a pnetcdf interface, and as far as I can
> tell, the writes are going through fine, although I can't say 100% for
> sure. The files are then read either through the netcdf4 interface for
> pnetcdf or as plain CF-2 files (e.g., with ncview). Ncview shows these
> failed reads as bands of zero values, for example, here is an X-Z
> cross-section:
> >>
> >> <PastedGraphic-1.tiff>
> >>
> >> where the blue section is just zeros that should not be zero. When I
> step to the next XZ plane and back, the values are usually filled in. I
> assume that ncview is re-reading the file. Since Ncview is linked to a
> normal serial library of netcdf4/hdf5 (4.3.2/1.8.9), so I believe that the
> pnetcdf libs probably do not play a role in the read problems. When the
> files are read back in by my cloud model, these read failures also happen .
> Those arrays are initialized to zero before reading, so the read failures
> show up as zero. The last time I ran these codes in April, everything
> worked fine. I'm just picking up this work again this month, and these
> problems appeared. The only thing I can think of that is different is the
> Lustre upgrade. For example, I copied a given file to my home directory,
> and ncview has no problems with it there.
> >>
> >> So, has anybody seen IO problems on recent (latest, I think) version of
> Lustre, particularly with large arrays in 64-bit offset files? It might not
> be just netcdf files, but that is all I'm using.
> >>
> >> Cheers,
> >> - Ted Mansell
> >>
> >> P.S. The files are written using Netcdf 4.3.1.1, hdf5 1.8.9, pnetcdf
> 1.5.0, netcdf-fortran 4.4.0.
> >>
> >> __________________________________________________________
> >> | Edward Mansell <ted.mansell@xxxxxxxx>
> >> | National Severe Storms Laboratory
> >> |--------------------------------------------------------------
> >> | "The contents of this message are mine personally and
> >> | do not reflect any position of the U.S. Government or NOAA."
> >> |--------------------------------------------------------------
> >>
>
>
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
>
