Re: [netcdfgroup] read performance slow compared to netCDF on other systems

Dear Liam
Your results are really surprising as the time spend is in the compute part
and not in io at all.  Right? We see that user time is 28+ seconds while
sys time that includes whatsoever syscalls including IO behaves differently.
If I had to speculate I would say that the new ncks does much more checks
than the old version or that it is simply not compiled with -O3.

Regards
Julian

Am 14.11.2016 12:11 nachm. schrieb "Liam Forbes" <loforbes@xxxxxxxxxx>:

> On Sun, Nov 13, 2016 at 5:24 AM, Julian Kunkel <
> juliankunkel@xxxxxxxxxxxxxx> wrote:
>
>> Dear Liam,
>> to investigate the problem and identify the cause, I propose the
>> following:
>> - on your old system, copy the file to /dev/shm, then time the reading
>> of the file (as you did).
>> - on your new system do the same.
>>
>> If there is an issue inside the host or the library, then the time
>> should differ.
>> I expect this won't be the case but the problem is likely inside the
>> Lustre client.
>>
>
> Julian,
>
> Thank you for the suggestion to use /dev/shm. Eliminating the different
> lustre client versions between the two systems is a good idea, and that
> seems to be what happened.
>
> On the new cluster, in /dev/shm, the timing came out pretty much the same
> as in the shared lustre filesystem.
> n0:shm$ time ncks test.nc out.nc
>
> real    0m29.509s
> user    0m28.605s
> sys    0m0.466s
>
> On the Cray, in /dev/shm, the timing also came out pretty much the same as
> in the shared lustre filesystem.
> fish1:shm$ time ncks test.nc out.nc
>
> real    0m4.023s
> user    0m3.152s
> sys    0m0.832s
>
> So, based on these results in /dev/shm, I infer from your comments that
> there is a difference in the host configuration, NCO, NetCDF, or HDF5
> between these two systems that is causing the performance difference. I
> think I may try installing the same, older versions of the packages on the
> new cluster as we have on the Cray and see if that changes anything. At
> least it will be a better apples to apples comparison.
>
> --
> Regards,
> -liam
>
> -There are uncountably more irrational fears than rational ones. -P. Dolan
> Liam Forbes  loforbes@xxxxxxxxxx  ph: 907-450-8618 fax: 907-450-8601
> UAF Research Computing Systems Senior HPC Engineer  LPIC1, CISSP
>
  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: