On 02/24/2017 01:19 PM, Zhiyu (Drew) Li wrote:
Thanks Rob.
I have another question. How much of performance gain would we get from
using parallel-netcdf-c (or say MPI-IO) on a mainstream desktop PC (one
multi-core cpu with one hdd or ssd)? Currently we dont have a parallel
file system environment (software/hardware) at out lab.
If your goal is peak performance on one node, then I doubt MPI-IO
approaches (either directly or through a library like Unidata NetCDF or
Northwestern/Argonne Parallel-NetCDF) will outperform the serial version.
performance should be pretty close, though! Perhaps its worth a small
serial cost so that you have code you can develop, debug, and test
locally, then pick up and put on a supercomputer some day.
When it comes to I/O, there's not much gain to be had by having more
cores. If you only have one HDD or SSD, then there's no benefit to
having multiple writers/readers, aside from whatever benefit there is to
having multiple operations in flight (serial libraries would benefit
from that too, though).
==rob
Thanks
Drew
On Fri, Feb 24, 2017 at 11:50 AM, Rob Latham <robl@xxxxxxxxxxx
<mailto:robl@xxxxxxxxxxx>> wrote:
On 02/22/2017 10:46 AM, Zhiyu (Drew) Li wrote:
Hi there,
I am playing with the parallel-netcdf-c examples to learn if I could
apply this technology to improve netcdf i/o in my project. I got
some
questions about this example tst_parallel4.c found at
https://github.com/Unidata/netcdf-c/blob/master/nc_test4/tst_parallel4.c
<https://github.com/Unidata/netcdf-c/blob/master/nc_test4/tst_parallel4.c>
<https://github.com/Unidata/netcdf-c/blob/master/nc_test4/tst_parallel4.c
<https://github.com/Unidata/netcdf-c/blob/master/nc_test4/tst_parallel4.c>>.
I saw the statements "nc_var_par_access(ncid, varid,
NC_COLLECTIVE)" and
"nc_var_par_access(ncid, varid, NC_INDEPENDENT)" are commented
out on
lines 133 and 134
(https://github.com/Unidata/netcdf-c/blob/master/nc_test4/tst_parallel4.c#L133
<https://github.com/Unidata/netcdf-c/blob/master/nc_test4/tst_parallel4.c#L133>
<https://github.com/Unidata/netcdf-c/blob/master/nc_test4/tst_parallel4.c#L133
<https://github.com/Unidata/netcdf-c/blob/master/nc_test4/tst_parallel4.c#L133>>).
Q1: Is this nc_var_par_access() statement optional?
It's optional. I like to add it to make explicit if I am requesting
indepnedent i/o or collective i/o. Long ago the docs and the
implementation differed on what was the default. I make it
explicit and don't have to worry.
Q2: I enabled each of the two lines one at a time to test
NC_COLLECTIVE
mode and NC_INDEPENDENT mode separately. Each test was ran with 4
processes (mpiexec -np 4 ./tst_parallel4). Then I used jumpshot to
visualize the clog2 files they produced. The snapshots are attached
below. The green bars represent "Write to netcdf file" events (I
turn
off other bars (other mpi events) in visualization).
Inline image 1
NC_INDEPENDENT mode
In NC_INDEPENDENT mode, the Write events occurred at different time
steps in the 4 processes (the x-axis is time step). If I
understood it
correctly, although we had 4 processes running in parallel, the
Write
events still happened in sequence, not in parallel, because p0 wrote
first, then p1 wrote, and then p2, and then p3 wrote last. Is it
supposed to be like this???
It is. Look a few lines above where the test inserts some sleep
calls if USE_MPE is defined (I guess to make it more visually
interesting?)
https://github.com/Unidata/netcdf-c/blob/master/nc_test4/tst_parallel4.c#L130
<https://github.com/Unidata/netcdf-c/blob/master/nc_test4/tst_parallel4.c#L130>
NC_COLLECTIVE mode
In NC_COLLECTIVE mode, p0 started writing first but its Write event
lasted until the fourth process p3 finished writing. I thought
all the
four process should start and stop writing at the same time in
NC_COLLECTIVE mode???
If there are sleep calls in the test, then some processes will reach
the collective call later. The test does demonstrate the one big
drawback of collective calls: if there is skew, then a
"pseudo-synchronization" occurs as the first process cannot make
progress until the last process enters the collective.
(note: in this case all processes leave the collective at about the
same time. that's not necessarily guaranteed by a collective
operation, not even MPI_BARRIER).
The MPE traces you have shown are consistent with the test.
I'm so pleased you are using MPE. We haven't had funding to work on
it for a few years, but it still comes in handy!
==rob
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
<http://www.unidata.ucar.edu/mailing_lists/>
