So it turned out to be a Lustre issue:
https://jira.hpdd.intel.com/browse/LU-6389?focusedCommentId=110278&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-110278
The bug affects Lustre 2.5.x and the system I was using had just been updated
from Lustre 2.4.3 to 2.5.37. The sysadmins tested a patch on a couple nodes,
and that fixed the problem.
Many thanks to Wei-keng Liao for helping to identify Lustre as the culprit!
It brings up the issue, however, of whether netcdf and other applications
should provide more warning when "short reads" are encountered? In this case a
1MB "slice" of data was missing from the requested data. Should netcdf be
checking for this and report errors? Or does this happen too rarely? Does
checking it therefore have too much overhead? The philosophy for Lustre is that
it should never deliver short reads because applications normally do not expect
(or handle) them, even though the Posix spec technically allows it to happen.
Cheers,
-- Ted
__________________________________________________________
| Edward Mansell <ted.mansell@xxxxxxxx>
| National Severe Storms Laboratory
|--------------------------------------------------------------
| "The contents of this message are mine personally and
| do not reflect any position of the U.S. Government or NOAA."
|--------------------------------------------------------------
On Aug 2, 2015, at 10:46 AM, Ted Mansell <Ted.Mansell@xxxxxxxx> wrote:
> Update: I am wondering if it is an issue with file striping on Lustre. The
> files are writing through direct pnetcdf calls with striping set to 16:
>
> lfs setstripe -c 16 .
>
> When I cp a cf-2 file to a higher directory (not striped), perhaps this
> rearranges the storage, because there are no reading issues with the copy,
> even after mv'ing the copy back to the original directory. (This is for
> reading either with a serial application like ncview or parallel read via the
> netcdf4 interface to pnetcdf.)
>
> -- Ted
>
>> Howdy,
>>
>> I seeing some strange behavior in reading large files, and I'm wondering if
>> anybody has similar experience. This is on a Lustre file system that was
>> upgraded in June (I assume to the latest version, but I will check that). I
>> had no previous problems. Anyway, the symptom is that I am reading
>> reasonably large float arrays (400 x 400 x 130), and randomly some parts of
>> the array will not be read in, leaving the initialized (zero) values. The
>> arrays are ordered x,y,z (fortran convention), and the failures usually show
>> up as partial or complete x-y planes. A file might have 15 to 500 arrays of
>> this size and one time level.
>>
>> I am writing the files with a pnetcdf interface, and as far as I can tell,
>> the writes are going through fine, although I can't say 100% for sure. The
>> files are then read either through the netcdf4 interface for pnetcdf or as
>> plain CF-2 files (e.g., with ncview). Ncview shows these failed reads as
>> bands of zero values, for example, here is an X-Z cross-section:
>>
>> <PastedGraphic-1.tiff>
>>
>> where the blue section is just zeros that should not be zero. When I step to
>> the next XZ plane and back, the values are usually filled in. I assume that
>> ncview is re-reading the file. Since Ncview is linked to a normal serial
>> library of netcdf4/hdf5 (4.3.2/1.8.9), so I believe that the pnetcdf libs
>> probably do not play a role in the read problems. When the files are read
>> back in by my cloud model, these read failures also happen. Those arrays are
>> initialized to zero before reading, so the read failures show up as zero.
>> The last time I ran these codes in April, everything worked fine. I'm just
>> picking up this work again this month, and these problems appeared. The only
>> thing I can think of that is different is the Lustre upgrade. For example, I
>> copied a given file to my home directory, and ncview has no problems with it
>> there.
>>
>> So, has anybody seen IO problems on recent (latest, I think) version of
>> Lustre, particularly with large arrays in 64-bit offset files? It might not
>> be just netcdf files, but that is all I'm using.
>>
>> Cheers,
>> - Ted Mansell
>>
>> P.S. The files are written using Netcdf 4.3.1.1, hdf5 1.8.9, pnetcdf 1.5.0,
>> netcdf-fortran 4.4.0.
>>
>> __________________________________________________________
>> | Edward Mansell <ted.mansell@xxxxxxxx>
>> | National Severe Storms Laboratory
>> |--------------------------------------------------------------
>> | "The contents of this message are mine personally and
>> | do not reflect any position of the U.S. Government or NOAA."
>> |--------------------------------------------------------------
>>
