sg0/rmacxx

C++ interface over MPI-3 RMA

RMACXX

RMACXX is a prototypical C++14 interface over MPI-3 RMA, which focuses on providing a thin layer above MPI RMA, and at the same time aims to improve the programmability of MPI RMA.

Paper on RMACXX:

Sayan Ghosh, Yanfei Guo, Pavan Balaji, Assefaw Gebremedhin. RMACXX: An Efficient High-Level C++ Interface over MPI-3 RMA. 21st IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGrid 2021).

Configuration

Note: RMACXX is built and tested with this configuration, so it may not work as specified with other configurations.

rmacxx's Issues

Higher Dimensional Testing (Dimensions 4-6)

We need to design and implement multidimensional tests. Ideally, we should create a new test file for each of the dimensions being tested and implement the appropriate tests.

Concept "CopyConstructor" throws errors

When I moved the concept "CopyConstructor" to the concepts file, the IDE started throwing errors. Could be an implementation error on my part.

Higher Dimensional Testing (Dimensions 2-3)

We need to design and implement multidimensional tests. Ideally, we should create a new test file for each of the dimensions being tested and implement the appropriate tests.

Unexpected behavior when performing operations involving multiple window reads

Reproduction:

int main() {
    // -- SNIP --
    win.fill(1);
    std::vector<int> result(25);

    // Problem here:
    4*win(1,{0,0},{4,4})*win(1,{0,0},{4,4}) >> result.data();

    win.flush();
    // -- SNIP --
}

Expected behavior:

result[0] == 4

Observed behavior:

result[0] == 64

Temporary solution:

    // Quick fix; add parenthesis around the window operations
    4*(win(1,{0,0},{4,4})*win(1,{0,0},{4,4}))>> result.data();

Higher Dimensional Testing (Dimensions 7-9)

We need to design and implement multidimensional tests. Ideally, we should create a new test file for each of the dimensions being tested and implement the appropriate tests.

Replace Makefile with CMake

Replace Makefile with CMake and expose better CI stats.

Higher Dimensional Testing

We need to write tests that cover higher dimensional windows. I believe we only cover up to 2Ds right now but we want to expand this.

Test simple_gv_bulk_3D_subarray failing on WSL and GitHub Actions

The test dimensions/simple_gv_bulk_3D_subarray.4.cpp builds and passes correctly on MacOS and virtualized Ubuntu 22.10, but fails on GitHub Actions and on WSL instances running Ubuntu 20.04. The error on GitHub is MPI_ERR_ARG: invalid argument of some other kind.

`NO_FLUSH` window starting accumulation operations without flush call

Reproduction:

int main(){
    // -- SNIP --
    // fill window
    win.fill(1);

    // create a vector and fill
    std::vector<int> twos(4); // 2(h) x 2(w) 
    std::fill(twos.begin(), twos.end(), 2);

    // Accumulation
    win(1, {0,0},{1,1}, SUM) << twos.data(); 
    win.flush(1);
    win.barrier();

    std::vector<int> results(4);
    win(1, {0,0},{1,1}) >> results.data(); 

    // -- SNIP --
}

Expected behavior:

// 2 added from rank 1 only
results[0] == 3; // 1 + 2 

Observed behavior:

// 2 Added from both ranks
results[0] == 5; // 1 + 2 + 2 

It is unclear if this behavior is in error, but from experience with RMACXX, where other expressions would not even begin to compute in a NO_FLUSH window, this may at least be a consistency issue across the API.

Enable testing pipeline to work with different-sized communicators

Currently, the makefile used assumes 2 processes for each test case. A handful of them, though, are designed to run on exactly 4 processes, and are currently failing as such. This flexibility (and more) should be integrated into the testing pipeline for more accurate reads on RMACXX's current state

`RMACXX_Subarray_t` not being obeyed

Reproduction:

int main(){
    // -- SNIP --
    win.fill(1);

    // create a vector, initially 0s
    std::vector<int> a(10); 
    int *abuf = a.data();
    
    // create subarray type
    // Note: Output is unchanged by changes to these values
    rmacxx::RMACXX_Subarray_t<int> foo({1},{2},{10}); 
    
    // put
    win(1,{0},{9}) << foo(abuf);
    win.flush(1);
    std::vector<int> results(10); 
    win(1,{0},{9}) >> results.data();
    win.flush(1);
    // -- SNIP --
}

Expected behavior:

a = { 0, 0, 1, 1, 1, 1, 1, 1, 1, 1};

Observed behavior:

// Note: Output is unchanged by values of subarray_t
a = { 0, 1, 1, 1, 1, 1, 1, 1, 1, 1};

Suspected cause:
include/rmacxx-mpi-rma.hpp

       #define RMACXX_BULK_PUT_NC(origin)\
194|        do{\
...|            /* -- SNIP */\
222|            MPI_Accumulate(origin.ptr_, 1, origin.dtype_, target_, \
223|              /*disp*/ 0, 1, sarr_type, MPI_REPLACE, win_); \
...|            /* -- SNIP */\
265|        }while(0)

No members of the RMACXX_Subarray_t origin, not even the MPI_Subarray_t that was already created, are included in the actual MPI_Accumulate call, likely meaning that all calls are relying on the exact same implementation, regardless of user inputs

Higher Dimensional Testing 4D Breaking

Test seems to be breaking at the initialization of the window.

Concept "SameSize" needs implementation

In the last commit, I pushed skeleton code for a concept "SameSize". This concept is supposed to check that two windows are the same size before performing calculations on them. This concept still needs full implementation (tensor-dev-debug branch - concepts.hpp)