MVAPICH/HYDRA demonstrator script

[andre-merzky]

MVAPICH/HYDRA demonstrator script

[andre-merzky]

demonstrator.txt

The above shell script is a poor-man's rendering of running a homogeneous workload. It executes 1000 tasks (each being a sleep 1), by running at most 100 of them at the same time (those numbers can be changed in line 4 and 5). This is a small scale version of what we run on clusters: the amount of resources limits the number of tasks we can run concurrently, and the total number of tasks is usually much larger than that.

The script also takes some timestamps during execution and computes the overhead. A typical output would be:

$ bash ./demonstrator.txt 
tasks: started  :       1000 
       running  :          0 
       completed:       1000 
time : expected :  10.000000 sec 
       measured :  10.176738 sec 
       overhead :   0.176738 sec (1.7 %)

The ideal runtime for that workload would be 10 seconds. Here we measure 10.17 seconds, i.e. an overhead of 1.7%. Plots for this run are also attached (you find them in /tmp/demo*.pdf after the run, iff gnuplot is installed to create them).

demo.dur.2.pdf
demo.dur.1.pdf

In the real world, instead of running sleep from a shell, we want to run mpirun <some science code. This is where MVAPICH/Hydra enter the picture: What is the best / fastest way (with the least overhead) to run many short running tasks suing HYDRAA?

[andre-merzky]

Hi all - we are going to close this ticket as the demo script was provided. Please use the discussion thread for any follow-up exchange. Thanks!