uiuc-ppl/charmlite

A lightweight facsimile of Charm++

charmlite's Issues

Oversubscribed SMP Performance is Ludicrously Bad

In particular, for jacobi, cbench, and pingpong.

Cbench is the worst of the bunch, taking the CI more than a minute to complete:

8: Test command: /home/runner/work/charmlite/charmlite/charm/bin/charmrun "/home/runner/work/charmlite/charmlite/build/bin/pgm_cbench_benchmark" "+p2" "++ppn2"
8: Test timeout computed to be: 120
8: 
8: Running as 1 OS processes: /home/runner/work/charmlite/charmlite/build/bin/pgm_cbench_benchmark ++ppn2
8: charmrun> /usr/bin/setarch x86_64 -R mpirun -np 1 /home/runner/work/charmlite/charmlite/build/bin/pgm_cbench_benchmark ++ppn2
8: Charm++> Running in SMP mode: 1 processes, 2 worker threads (PEs) + 1 comm threads per process, 2 PEs total
8: Charm++> The comm. thread both sends and receives messages
8: Converse/Charm++ Commit ID: v7.1.0-devel-122-g064b48915
8: Charm++> Using STL-based msgQ:
8: Charm++> Message priorities have been turned off and will not be respected.
8: main> rep 1 of 16
8: main> rep 2 of 16
8: main> rep 3 of 16
8: main> rep 4 of 16
8: main> rep 5 of 16
8: main> rep 6 of 16
8: main> rep 7 of 16
8: main> rep 8 of 16
8: main> rep 9 of 16
8: main> rep 10 of 16
8: main> rep 11 of 16
8: main> rep 12 of 16
8: main> rep 13 of 16
8: main> rep 14 of 16
8: main> rep 15 of 16
8: main> rep 16 of 16
8: info> interleaved 129 broadcasts and reductions across 8 chares
8: info> average time per repetition: 4453.8 ms
8: info> average time per broadcast+reduction: 34525.6 ns
8: [Partition 0][Node 0] End of program
 8/10 Test  #8: pgm_cbench_benchmark_pe2 .........   Passed   72.52 sec

It's not uncommon to see these 34525.6 ns broadcasts+reductions on an over-subscribed PC either! We should probably try to determine what's going on here, and why the performance is so bad for these configurations.

What I've tried so far:

• Enabling or disabling +CmiSleepOnIdle.
• Enabling or disabling cpu topology/affinity.
• Using the lockless queue (--enable-lockless-queue).

Nothing seemed to improve the situation.

Collective Benchmark Sometimes Hangs in SMP Mode

I believe this is the issue @NK-Nikunj observed. The benchmark sometimes hangs in SMP mode. I have been unable to reproduce this hang in non-SMP builds. Presumably, we're misusing CmiRankOf, CmiPushPE, or another SMP-sensitive function somewhere.

❯ ./bin/pgm_cbench_benchmark +p2 -r 256                                                                                                                                                                                                           build -> jszaday/fixReductionOverflow ? ! $
Charm++: standalone mode (not using charmrun)
Charm++> Running in SMP mode: 1 processes, 2 worker threads (PEs) + 1 comm threads per process, 2 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: v7.1.0-devel-122-g064b48915
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Charm++> Using STL-based msgQ:
Charm++> Message priorities have been turned off and will not be respected.
Charm++> scheduler running in netpoll mode.
main> rep 1 of 384
^C

Multi-worker SMP Builds Nullify Command-line Arguments

Running with ++ppn ≠ 1 in SMP builds nullifies argv and sets argc to a random value. This may be a Converse-related problem, but more research is necessary.

This is demonstrable by running cbench, which crashes when it tries to call parse_arguments:

% charmrun ./pgm_cbench_benchmark +p2 ++ppn 2

Running as 1 OS processes: ./pgm_cbench_benchmark +ppn 2
charmrun> /usr/bin/setarch x86_64 -R mpirun -np 1 ./pgm_cbench_benchmark +ppn 2
Charm++> Running in SMP mode: 1 processes, 2 worker threads (PEs) + 1 comm threads per process, 2 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: v7.1.0-devel-120-g8ad2f984b
Charm++> Using STL-based msgQ:
Charm++> Message priorities have been turned off and will not be respected.
[cordelia:604511:0:604511] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace (tid: 604511) ====
 0  /home/szaday2/workspace/ucx/build/lib/libucs.so.0(ucs_handle_error+0x2e4) [0x7ffff7daf534]
 1  /home/szaday2/workspace/ucx/build/lib/libucs.so.0(+0x2d76f) [0x7ffff7daf76f]
 2  /home/szaday2/workspace/ucx/build/lib/libucs.so.0(+0x2da56) [0x7ffff7dafa56]
 3  /lib/x86_64-linux-gnu/libc.so.6(+0x46520) [0x7ffff756d520]
 4  /lib/x86_64-linux-gnu/libc.so.6(+0x10b9d5) [0x7ffff76329d5]
 5  /lib/x86_64-linux-gnu/libc.so.6(+0x10bd5b) [0x7ffff7632d5b]
 6  /lib/x86_64-linux-gnu/libc.so.6(getopt+0x17) [0x7ffff7632da7]
 7  ./pgm_cbench_benchmark(_Z15parse_argumentsiPPc+0x50) [0x427fd0] 🠔
 8  ./pgm_cbench_benchmark(main+0x64) [0x427b24]
 9  /lib/x86_64-linux-gnu/libc.so.6(+0x2dfd0) [0x7ffff7554fd0]
10  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x7d) [0x7ffff755507d]
11  ./pgm_cbench_benchmark(_start+0x25) [0x4279e5]
=================================
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node cordelia exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Implement QD to automatically call `cmk::exit` when idle

An empty charmlite program such as follows fails to run to completion with non-SMP (and potentially SMP) builds:

#include <charmlite/charmlite.hpp>

int main(int argc, char* argv[])
{
    cmk::initialize(argc, argv);

    {
        // Some code OR no code
    }

    cmk::finalize();

    return 0;
}

Finish migration support

There are a couple remaining tasks for migration:

• Add chare<...>::can_migrate for emigrate to call -- reject migration request if it fails.
• Trigger on_departure and on_arrival functions during migration events.
• Ensure tree-builder handles these new "triggers" correctly.
• Ensure flush_buffers stops flushing if the target "got disappeared" midway through.
• Ensure location updates are consistent with Charm++.

(Does Charm++ send a location update from the PE an element emigrates to?)

Collective Benchmark Hangs w/ More Than 256 Repetitions

Unsure why. Needs diagnosing.

% charmrun ./pgm_cbench_benchmark -r 256                                                                                                       

Running as 1 OS processes: ./pgm_cbench_benchmark -r 256
charmrun> /usr/bin/setarch x86_64 -R mpirun -np 1 ./pgm_cbench_benchmark -r 256
Charm++> Running in non-SMP mode: 1 processes (PEs)
Converse/Charm++ Commit ID: v7.1.0-devel-120-g8ad2f984b
Charm++> Using STL-based msgQ:
Charm++> Message priorities have been turned off and will not be respected.
main> rep 1 of 384
main> rep 2 of 384
.
.
.
main> rep 256 of 384
^C