dash-project/dash

DASH, the C++ Template Library for Distributed Data Structures with Support for Hierarchical Locality for HPC and Data-Driven Science

DASH

A C++ Template Library for Distributed Data Structures with Support for Hierarchical Locality for HPC and Data-Driven Science.

Summary

Exascale systems are scheduled to become available in 2018-2020 and will be characterized by extreme scale and a multilevel hierarchical organization.

Efficient and productive programming of these systems will be a challenge, especially in the context of data-intensive applications. Adopting the promising notion of Partitioned Global Address Space (PGAS) programming the DASH project develops a data-structure oriented C++ template library that provides hierarchical PGAS-like abstractions for important data containers (multidimensional arrays, lists, hash tables, etc.) and allows a developer to control (and explicitly take advantage of) the hierarchical data layout of global data structures.

In contrast to other PGAS approaches such as UPC, DASH does not propose a new language or require compiler support to realize global address space semantics. Instead, operator overloading and other advanced C++ features are used to provide the semantics of data residing in a global and hierarchically partitioned address space based on a runtime system with one-sided messaging primitives provided by MPI or GASNet.

As such, DASH can co-exist with parallel programming models already in widespread use (like MPI) and developers can take advantage of DASH by incrementally replacing existing data structures with the implementation provided by DASH. Efficient I/O directly to and from the hierarchical structures and DASH-optimized algorithms such as map-reduce are also part of the project. Two applications from molecular dynamics and geoscience are driving the project and are adapted to use DASH in the course of the project.

Funding

DASH is funded by the German Research Foundation (DFG) under the priority programme "Software for Exascale Computing - SPPEXA" (2013-2018).

Community

Project Website:

http://www.dash-project.org

GitHub:

https://github.com/dash-project

Documentation Wiki

http://doc.dash-project.org

Repository:

• HTTPS: https://github.com/dash-project/dash.git
• SSH: [email protected]:dash-project/dash.git

Contact:

• Slack channel: https://dash-project.slack.com
• Support: [email protected]
• Developer mailing list: [email protected]

Contributing

See guidelines in CONTRIBUTING.md.

Installation

DASH installations are available as Docker containers or build from source using CMake.

Docker Containers

For pre-build Docker container images, see the DASH project on Docker Hub.

Building from Source

DASH is built using CMake.

Build scripts are provided for typical DASH configurations and can serve as starting points for custom builds:

Script file name	Description
build.sh	Standard release build
build.dev.sh	Development / debug build
build.mic.sh	Release build for Intel MIC (Xeon Phi)

Prerequisites

• CMake version 2.8.5 or greater (3.0.0 or greater recommended)
• C compiler supporting the C99 standard
• C++ compiler supporting the C++11 standard

Optional third-party libraries directly supported by DASH:

• PAPI
• libnuma
• hwloc
• BLAS implementation like Intel MKL, ATLAS
• LIKWID
• HDF5

Getting the sources

DASH is hosted on Github at https://github.com/dash-project/dash and makes use of git submodules to include third-party software (mainly the GoogleTest framework required for building the tests). The build environment will take care of cloning the submodules upon first invocation. However, there might be cases where recursive cloning is required, e.g., if there is no internet access available during the configuration step.

In that case, please use a recursive clone:

(dash/)$ git clone --recursive https://github.com/dash-project/dash.git

Building DASH from Source

To build the DASH project using CMake with default build settings, run:

(dash/)$ cmake --build .

Or, specify a new directory used for the build:

(dash/)$ mkdir build && cd ./build

For a list of available CMake parameters:

(build/)$ cmake .. -L

Parameters can be set using -D flags. As an example, these parameters will configure the build process to use icc as C compiler instead of the default compiler:

(build/)$ cmake -DCMAKE_C_COMPILER=icc ..

To configure build parameters using ccmake:

(build/)$ ccmake ..

1. Choosing a DASH runtime (DART)

DASH provides the following variants:

• MPI: The Message Passing Interface, requiring a MPI 3.0 compliant implementation
• CUDA: nNvidia's Compute Unified Device Architecture (contributor distribution only)
• SHMEM: Symmetric Hierarchical Memory access (contributor distribution only)

The build process creates the following libraries:

• libdart-mpi
• libdart-cuda
• libdart-shmem

By default, DASH is configured to build all variants of the runtime. You can specify which implementations of DART to build using the cmake option

(build/)$ cmake -DDART_IMPLEMENTATIONS=mpi,shmem ...

Programs using DASH select a runtime implementation by linking against the respective library.

Specifying the MPI implementation for the DART-MPI runtime

The most reliable method to build DART-MPI with a specific MPI installation is to specify the CMake options MPI_<lang>_COMPILER:

(build/) $ cmake -DMPI_C_COMPILER=/path/to/mpi/bin/mpicc \
                 -DMPI_CXX_COMPILER=/path/to/mpi/bin/mpiCC \
                 ...

3. Examples and Unit Tests

Source code of usage examples of DASH are located in dash/examples/. Examples each consist of a single executable and are built by default. Binaries from examples and unit tests are deployed to the build direcory but will not be installed. To disable building of examples, specify the cmake option

(build/)$ cmake -DBUILD_EXAMPLES=OFF ...

To disable building of unit tests, specify the cmake option

(build/)$ cmake -DBUILD_TESTS=OFF ...

The example applications are located in the bin/ folder in the build directory.

4. Installation

The default installation path is $HOME/opt as users on HPC systems typically have install permissions in their home directory only.

To specify a different installation path, use

(build/)$ cmake -DINSTALL_PREFIX=/your/install/path ../

The option -DINSTALL_PREFIX=<DASH install path> can also be given in Step 1.

The installation process copies the 'bin', 'lib', and 'include' directories in the build directory to the specified installation path.

(dash/)$ cmake --build . --target install

Or manually using make:

(build/)$ cmake <build options> ../
(build/)$ make
(build/)$ make install

Running DASH Applications

Building a DASH application

DASH provides wrapper scripts that ensure correct include paths as well as linking for all required DASH, DART, and third-party libraries.

The wrappers are named as dash-<variant>c++, depending on the DASH variant activated at build time, e.g., for the MPI variant the wrapper will be called dash-mpic++ (and its aliases dash-mpiCC and dash-mpicxx).

To build a DASH (MPI) application, replace the call to the MPI C++ compiler with a call to dash-mpicxx (or its alias dash-mpiCC and dash-mpic++):

$ CXX=dash-mpicxx make

The compiler wrapper currently provides two options:

• --dash:verbose will cause all invocations of the underlying compiler to be printed to the console.
• --dash:nocppflags disables passing DASH-related precompiler flags to the underlying compiler, including flags that control DASH verbosity (if enabled during DASH build) and assertions.

All other parameters will be passed to the underlying compiler, allowing you to control optimization flags and pass precompiler options.

Note that the compiler wrappers also set the language standard to C++11.

Running a DASH application

With the MPI variant, applications are spawn by MPI:

$ mpirun <MPI args> <app>-mpi

For CUDA and SHMEM, use

$ dartrun-cuda <dartrun-args> <app>-cuda

and respectively

$ dartrun-shmem <dartrun-args> <app>-shmem

Running Tests

Launch the DASH unit test suite using dash-test-shmem or dash-test-mpi:

(dash/shmem/bin/)$ dartrun-shmem <dartrun args> dash-test-shmem <gtest args>

or

(dash/mpi/bin/)$ mpirun <MPI args> dash-test-mpi <gtest args>

For example, you would all unit tests of matrix data structures on 4 units using the MPI runtime with:

(dash/mpi/bin/)$ mpirun -n 4 dash-test-mpi --gtest_filter="MatrixTest*"

or all tests except for the Array test suite:

(dash/mpi/bin/)$ mpirun -n 4 dash-test-mpi --gtest_filter="-ArrayTest*"

Profiling DASH Applications using IPM

Use LD_PRELOAD to run a DASH application built with the DART-MPI backend:

$ LD_PRELOAD=/$IPM_HOME/lib/libipm.so mpirun -n <nproc> <DASH executable>

Available options for IPM are documented in the IPM user guide.

Links

The DASH project homepage: http://www.dash-project.org

The Munich Network Management homepage: http://www.mnm-team.org

dash's Issues

Include minimal build configuration in CI tests

Motivated by issue #34, standard build configurations (Release, Debug) x Minimal should be covered by integration test.

Testsuite hangs in collective operations when using MPICH

I am running the testsuite on our Linux cluster (Laki), this time using MPICH v3.4.1. There seems to be a race condition that leads one process to wait in MPI_Win_free while others are waiting in an allgather. DDT shows the following stack trace:

Processes,Threads,Function
4,4,gomp_thread_start (team.c:121)
4,4,  gomp_barrier_wait_end
4,4,main (main.cc:39)
4,4,  RUN_ALL_TESTS (gtest.h:2233)
4,4,    testing::UnitTest::Run (TeamLocality.cc:15)
4,4,      bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (TeamLocality.cc:15)
4,4,        bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (TeamLocality.cc:15)
4,4,          testing::internal::UnitTestImpl::RunAllTests (TeamLocality.cc:15)
4,4,            testing::TestCase::Run (TeamLocality.cc:15)
4,4,              testing::TestInfo::Run (TeamLocality.cc:15)
4,4,                testing::Test::Run (TeamLocality.cc:15)
4,4,                  void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (TeamLocality.cc:15)
4,4,                    void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (TeamLocality.cc:15)
3,3,                      MinElementTest_TestFindArrayDefault_Test::TestBody (MinElementTest.cc:33)
3,3,                        dash::GlobIter<long, dash::Pattern<1, (dash::MemArrange)1, long>, dash::GlobMem<long, dash::allocator::CollectiveAllocator<long> >, dash::GlobPtr<long, dash::Pattern<1, (dash::MemArrange)1, long> >, dash::GlobRef<long> > dash::min_element<long, dash::Pattern<1, (dash::MemArrange)1, long> > (MinMax.h:241)
3,3,                          dart_allgather
3,3,                            PMPI_Allgather
3,3,                              MPIR_Allgather_impl
3,3,                                MPIR_Allgather
3,3,                                  MPIR_Allgather_intra
3,3,                                    MPIC_Sendrecv
3,3,                                      MPIC_Wait
3,3,                                        MPIDI_CH3I_Progress
1,1,                      MinElementTest_TestFindArrayDefault_Test::TestBody (MinElementTest.cc:42)
1,1,                        dash::Array<long, long, dash::Pattern<1, (dash::MemArrange)1, long> >::~Array (Array.h:781)
1,1,                          dash::Array<long, long, dash::Pattern<1, (dash::MemArrange)1, long> >::deallocate (Array.h:1116)
1,1,                            dash::GlobMem<long, dash::allocator::CollectiveAllocator<long> >::~GlobMem (GlobMem.h:165)
1,1,                              dash::allocator::CollectiveAllocator<long>::deallocate (CollectiveAllocator.h:210)
1,1,                                dart_team_memfree
1,1,                                  PMPI_Win_free
1,1,                                    MPIDI_CH3_SHM_Win_free
1,1,                                      MPIR_Reduce_scatter_block_impl
1,1,                                        MPIR_Reduce_scatter_block_intra
1,1,                                          MPIC_Sendrecv
1,1,                                            MPIC_Wait
1,1,                                              MPIDI_CH3I_Progress

I'm not sure whether the following message is related to this issue so I post it here for completeness:

[    0 ERROR ] [ 20713 ] UnitLocality.h           :56   | dash::exception::AssertionFailed             | [ Unit 0 ] Assertion failed: Expected 0 /zhome/academic/HLRS/hlrs/hpcjschu/src/dash/dash-development/dash/include/dash/util/UnitLocality.h:56 

When filtering this test, some tests run through until I find another test that hangs. A random example is this:

Processes,Threads,Function
4,4,gomp_thread_start (team.c:121)
4,4,  gomp_barrier_wait_end
4,4,main (main.cc:39)
4,4,  RUN_ALL_TESTS (gtest.h:2233)
4,4,    testing::UnitTest::Run (TeamLocality.cc:15)
4,4,      bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (TeamLocality.cc:15)
4,4,        bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (TeamLocality.cc:15)
4,4,          testing::internal::UnitTestImpl::RunAllTests (TeamLocality.cc:15)
4,4,            testing::TestCase::Run (TeamLocality.cc:15)
4,4,              testing::TestInfo::Run (TeamLocality.cc:15)
4,4,                testing::Test::Run (TeamLocality.cc:15)
4,4,                  void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (TeamLocality.cc:15)
4,4,                    void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (TeamLocality.cc:15)
4,4,                      CopyTest_BlockingGlobalToLocalBarrierUnaligned_Test::TestBody (CopyTest.cc:351)
4,4,                        dash::Array<int, long, dash::Pattern<1, (dash::MemArrange)1, long> >::~Array (Array.h:781)
3,3,                          dash::Array<int, long, dash::Pattern<1, (dash::MemArrange)1, long> >::deallocate (Array.h:1107)
3,3,                            dash::Array<int, long, dash::Pattern<1, (dash::MemArrange)1, long> >::barrier (Array.h:1017)
3,3,                              dash::Team::barrier (Team.h:421)
3,3,                                dart_barrier
3,3,                                  PMPI_Barrier
3,3,                                    MPIR_Barrier_impl
3,3,                                      MPIR_Barrier
3,3,                                        MPIR_Barrier_intra
3,3,                                          MPIR_Barrier_impl
3,3,                                            MPIR_Barrier
3,3,                                              MPIR_Barrier_intra
3,3,                                                MPIC_Sendrecv
3,3,                                                  MPIC_Wait
3,3,                                                    MPIDI_CH3I_Progress
1,1,                                                      MPIDU_Sched_are_pending
1,1,                                                      MPID_nem_network_poll
1,1,                                                      MPID_nem_tcp_connpoll
1,1,                                                        poll
1,1,                          dash::Array<int, long, dash::Pattern<1, (dash::MemArrange)1, long> >::deallocate (Array.h:1116)
1,1,                            dash::GlobMem<int, dash::allocator::CollectiveAllocator<int> >::~GlobMem (GlobMem.h:165)
1,1,                              dash::allocator::CollectiveAllocator<int>::deallocate (CollectiveAllocator.h:210)
1,1,                                dart_team_memfree
1,1,                                  PMPI_Win_free
1,1,                                    MPIDI_CH3_SHM_Win_free
1,1,                                      MPIR_Reduce_scatter_block_impl
1,1,                                        MPIR_Reduce_scatter_block_intra
1,1,                                          MPIC_Sendrecv
1,1,                                            MPIC_Wait
1,1,                                              MPIDI_CH3I_Progress

Used compiler was GCC 4.9.1, PAPI was used in version 5.4.2.0.

Update Doxygen

Everyone (yes, you @dash-project/developers):

• Read the DASH API at https://codedocs.xyz/dash-project/dash/
• Complete it.

Reorganize header includes in test cases

Each test case includes the libdash.h header which is not necessary and results in a long build time. The libdash contains all headers of the dash project, it would be much easier to have each test include just the specific header it really needs.

Fix linker errors of hdf5 driver

Currently the implementation of dash's hdf5 driver does not link, as it breaks the multiple definition rule.

Corrupt free() in dart_segment

The CI runs fail in dart_exit because of a corrupt free in dart_segment_clear (dart-impl/mpi/src/dart_segment.c).

See log at https://travis-ci.org/dash-project/dash/builds/172487710

The CI run is reported as PASSED due to an unrelated bug in the CI script (see #45).

** Error in `/opt/dash/build-ci/20161031-091110/Release/bin/dash/test/mpi/dash-test-mpi': double free or corruption (fasttop): 0x0000000001b4e010 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f5450b9c7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x7fe0a)[0x7f5450ba4e0a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f5450ba898c]
/opt/dash/build-ci/20161031-091110/Release/bin/dash/test/mpi/dash-test-mpi(dart_segment_clear+0x66)[0x6946c6]
/opt/dash/build-ci/20161031-091110/Release/bin/dash/test/mpi/dash-test-mpi(dart_exit+0x2eb)[0x696b9b]
/opt/dash/build-ci/20161031-091110/Release/bin/dash/test/mpi/dash-test-mpi(_ZN4dash8finalizeEv+0x101)[0x6794d1]
/opt/dash/build-ci/20161031-091110/Release/bin/dash/test/mpi/dash-test-mpi(main+0x21b)[0x4bdb2b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f5450b45830]
/opt/dash/build-ci/20161031-091110/Release/bin/dash/test/mpi/dash-test-mpi(_start+0x29)[0x4c6f59]

Unified performance reports

For automated performance reports in CI, benchmark applications should provide an option to print performance measurements in a unified format.

For this, utility types/concepts dash::bench::BenchmarkResult, dash::bench::BenchmarkResultPrinter should be introduced to decouple benchmark parameters and performance metrics from output.

Illustrating example:

$ mpirun -n 32 -genv DASH_MAX_THREADS_PER_UNIT=4 \
               ./bin/bench.12.time-dilatation.mpi --nd=11 --self-aware
$ # output:
$ [bench.12.time-dilatation.mpi] [n:32;tpu:4] [nd:11;self-aware] [iteration:0] [123.4]
$ [bench.12.time-dilatation.mpi] [n:32;tpu:4] [nd:11;self-aware] [iteration:1] [226.6]
$ [bench.12.time-dilatation.mpi] [n:32;tpu:4] [nd:11;self-aware] [iteration:2] [559.0]

... with 123.4, 226.6, 559.0 as measures in a "unified" performance metric (higher is better) specific to
the benchmark.
The overall goal is to emit warnings if these values decrease in a nightly build.

dash::Matrix local column and row access and copy

I am trying to run a simple test using dash::Matrix local row() and col() views to copy a subset of a 2D matrix into a std::vector. I am making the following observations:

1. I can copy the whole local part of the Matrix into the local vector using dash::copy.

2. I can get a row using matrix.local.row(0) and correctly print its content on unit 0. However it fails with a std::out_out_range exception on unit 1. I later realized that when using the operator[] instead of operator() it works as expected. There seems to be a problem with the operator().

3. I am trying to copy a local row into a local std::vector like this:

{
    size_t team_size = dash::Team::All().size();
    size_t myid = dash::Team::All().myid();

    dash::TeamSpec<2> teamspec_2d(team_size, 1);
    teamspec_2d.balance_extents();

    dash::SizeSpec<2> sspec(20, 20);
    dash::DistributionSpec<2> dspec(dash::BLOCKED, dash::BLOCKED);

    dash::Matrix<double, 2> matrix(sspec,dspec, dash::Team::All(), teamspec_2d);

    initialize(matrix);
    dash::barrier();

    auto row = matrix.local.row(0);
    std::vector<double> tmp(row.end() - row.begin());
    dash::copy(row.begin(), row.end(), tmp.data());
}

This code causes memory to be corrupted, e.g., on unit 0:

*** Error in `/home/joseph/src/dash/workspace_2dheat/2dheat/dash_row_test': free(): invalid next size (fast): 0x00000000008de4c0 ***
[...]
Thread 1 "dash_row_test" received signal SIGABRT, Aborted.
0x00007ffff686a418 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff686a418 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007ffff686c01a in __GI_abort () at abort.c:89
#2  0x00007ffff68ac72a in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ffff69c56b0 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007ffff68b4f4a in malloc_printerr (ar_ptr=, ptr=, str=0x7ffff69c5728 "free(): invalid next size (fast)", action=3) at malloc.c:5007
#4  _int_free (av=, p=, have_lock=0) at malloc.c:3868
#5  0x00007ffff68b8abc in __GI___libc_free (mem=) at malloc.c:2969
#6  0x0000000000405d32 in main (argc=1, argv=0x7fffffffd9d8) at dash_row_test.cc:119
(gdb) 

Valgrind says:

==793== Invalid write of size 8
==793==    at 0x4C323E3: memcpy@GLIBC_2.2.5 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==793==    by 0x4120A7: double* std::__copy_move::__copy_m(double const*, double const*, double*) (stl_algobase.h:384)
==793==    by 0x410C17: double* std::__copy_move_a(double*, double*, double*) (stl_algobase.h:402)
==793==    by 0x40E102: double* std::__copy_move_a2(double*, double*, double*) (stl_algobase.h:440)
==793==    by 0x40997D: double* std::copy(double*, double*, double*) (stl_algobase.h:472)
==793==    by 0x407736: double* dash::copy, dash::GlobMem >, dash::GlobPtr >, dash::GlobRef > >(dash::GlobViewIter, dash::GlobMem >, dash::GlobPtr >, dash::GlobRef >, dash::GlobViewIter, dash::GlobMem >, dash::GlobPtr >, dash::GlobRef >, double*) (Copy.h:880)
==793==    by 0x405D14: main (dash_row_test.cc:118)
==793==  Address 0x1a18a870 is 0 bytes after a block of size 80 alloc'd
==793==    at 0x4C2E80F: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==793==    by 0x405C2E: main (dash_row_test.cc:116)

Since I am new to DASH, I cannot rule out that the code itself is incorrect. However, looking at the API it looks plausible to me.

4. The same is true when trying to copy a local column into a local vector (exchanging test 2 and 3 in the code). Note that in this case the access through operator() fails in all cases.

I ran all my tests using

$ mpirun -n 2 ./dash_row_test

I am attaching my test case. Please let me know if you think that the test case is malformed, I am happy to correct it. Please also let me know if you need additional information.

Tests fail if running on mutliple nodes

I am running the testsuite on with 4 processes on 2 nodes using MPICH v3.2 and see the following error in the Shared test:

Fatal error in PMPI_Bcast: Message truncated, error stack:
PMPI_Bcast(1600)....................: MPI_Bcast(buf=0x7fff5850cb88, count=16, MPI_BYTE, root=0, MPI_COMM_WORLD) failed
MPIR_Bcast_impl(1452)...............:
MPIR_Bcast(1476)....................:
MPIR_Bcast_intra(1249)..............:
MPIR_SMP_Bcast(1081)................:
MPIR_Bcast_binomial(239)............:
MPIC_Recv(353)......................:
MPIDI_CH3U_Request_unpack_uebuf(568): Message truncated; 280 bytes received but buffer size is 16
MPIR_SMP_Bcast(1088)................:
MPIR_Bcast_binomial(310)............: Failure during collective

The 16 bytes are a dart gptr that is sent to everyone in the team. I tried to find out where the 280 Byte are coming from and it seems that they stem from dart__base__host_topology__init where data is send from one leader unit to all others. Notice the following two log lines:

[    0 TRACE ] [  5492 ] host_topology.c          :355  :   DART: dart__base__host_topology__init: broadcasting module locations from leader unit 0 to units in team 0
[    1 TRACE ] [  5493 ] host_topology.c          :355  :   DART: dart__base__host_topology__init: broadcasting module locations from leader unit 0 to units in team 0
[    2 TRACE ] [ 22421 ] host_topology.c          :355  :   DART: dart__base__host_topology__init: broadcasting module locations from leader unit 2 to units in team 0
[    3 TRACE ] [ 22422 ] host_topology.c          :355  :   DART: dart__base__host_topology__init: broadcasting module locations from leader unit 2 to units in team 0`
and 

Notice the different leader units operating on the same team eventually calling a single broadcast. Looking at the code, I see the following comment that is highly suspicious:

if (DART_UNDEFINED_UNIT_ID != local_leader_unit_id) {
    /*
     * TODO: Use local_team instead of team if num_hosts > 1
     */

Maybe someone familiar with this code can comment on whether this might be causing the failing runs on multiple nodes?

DART initialization fails if no third-party library is enabled

If none of PAPI, libnuma, LIKWID, hwloc are enabled, DART initialization fails because no locality domains are constructed between scope GLOBAL and UNIT.

See secondary bug report in #24

Cleanup / Finalize DART Locality Interface

Including type semantics, see
https://github.com/dash-project/dash/compare/feat-dart-loc
from PR #217

Request official dockerhub account

Apparently, only "official" organizations can have a "-" in their name on dockerhub,

Here is the list of requirements for "official repositories". Since our latest efforts, we satisfy all of them.

https://github.com/docker-library/official-images/blob/master/NEW-IMAGE-CHECKLIST.md
https://github.com/docker-library/official-images

HDF5 matrix groups do not work

Currently the hdf5 matrix writer stores all matrices in the root group instead of the selected one. Hence the HDF5MatrixTest.GroupTest does not succeed.

MatrixTest.DelayedAlloc fails in Release build

If I run the CI script locally the test MatrixTest.DelayedAlloc fails for Release builds:

[  RUN     ] MatrixTest.DelayedAlloc
[=   0  LOG =]             MatrixTest.h :  20 | >>> Test suite: MatrixTest 
[=   0  LOG =]             MatrixTest.h :  21 | >>> Hostname: beryl PID: 25459 
[=   3  LOG =]             MatrixTest.h :  21 | >>> Hostname: beryl PID: 25462 
[=   1  LOG =]             MatrixTest.h :  33 | ===> Running test case with 4 units ... 
[=   2  LOG =]             MatrixTest.h :  33 | ===> Running test case with 4 units ... 
[=   3  LOG =]             MatrixTest.h :  33 | ===> Running test case with 4 units ... 
[=   0  LOG =]             MatrixTest.h :  33 | ===> Running test case with 4 units ... 
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.011
To be equal to: actual
      Which is: 0.011
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0301
To be equal to: actual
      Which is: 0.0301
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0302
To be equal to: actual
      Which is: 0.0302
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0306
To be equal to: actual
      Which is: 0.0306
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0307
To be equal to: actual
      Which is: 0.0307
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0308
To be equal to: actual
      Which is: 0.0308
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0312
To be equal to: actual
      Which is: 0.0312
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0501
To be equal to: actual
      Which is: 0.0501
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0506
To be equal to: actual
      Which is: 0.0506
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0508
To be equal to: actual
      Which is: 0.0508
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0513
To be equal to: actual
      Which is: 0.0513
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0124
To be equal to: actual
      Which is: 0.0124
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0316
To be equal to: actual
      Which is: 0.0316
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.022
To be equal to: actual
      Which is: 0.022
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0321
To be equal to: actual
      Which is: 0.0321
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0326
To be equal to: actual
      Which is: 0.0326
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0328
To be equal to: actual
      Which is: 0.0328
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0518
To be equal to: actual
      Which is: 0.0518
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0523
To be equal to: actual
      Which is: 0.0523
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0525
To be equal to: actual
      Which is: 0.0525
Unit 0
[=   3  LOG =]             MatrixTest.h :  39 | <=== Finished test case with 4 units 
[=   3  LOG =]             MatrixTest.h :  25 | <<< Closing test suite: MatrixTest 
[=   2  LOG =]             MatrixTest.h :  39 | <=== Finished test case with 4 units 
[=   2  LOG =]             MatrixTest.h :  25 | <<< Closing test suite: MatrixTest 
[=   1  LOG =]             MatrixTest.h :  39 | <=== Finished test case with 4 units 
[=   1  LOG =]             MatrixTest.h :  25 | <<< Closing test suite: MatrixTest 
[=   0  LOG =]             MatrixTest.h :  39 | <=== Finished test case with 4 units 
[=   0  LOG =]             MatrixTest.h :  25 | <<< Closing test suite: MatrixTest 
[  FAILED  ] MatrixTest.DelayedAlloc
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/TestPrinter.h:140
Failed
Testcase failed at least on one unit
[  RUN     ] MatrixTest.DelayedAlloc
[=   0  LOG =]             MatrixTest.h :  20 | >>> Test suite: MatrixTest 
[=   0  LOG =]             MatrixTest.h :  21 | >>> Hostname: beryl PID: 25459 
[=   3  LOG =]             MatrixTest.h :  21 | >>> Hostname: beryl PID: 25462 
[=   1  LOG =]             MatrixTest.h :  33 | ===> Running test case with 4 units ... 
[=   2  LOG =]             MatrixTest.h :  33 | ===> Running test case with 4 units ... 
[=   3  LOG =]             MatrixTest.h :  33 | ===> Running test case with 4 units ... 
[=   0  LOG =]             MatrixTest.h :  33 | ===> Running test case with 4 units ... 
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.011
To be equal to: actual
      Which is: 0.011
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0301
To be equal to: actual
      Which is: 0.0301
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0302
To be equal to: actual
      Which is: 0.0302
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0306
To be equal to: actual
      Which is: 0.0306
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0307
To be equal to: actual
      Which is: 0.0307
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0308
To be equal to: actual
      Which is: 0.0308
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0312
To be equal to: actual
      Which is: 0.0312
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0501
To be equal to: actual
      Which is: 0.0501
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0506
To be equal to: actual
      Which is: 0.0506
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0508
To be equal to: actual
      Which is: 0.0508
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0513
To be equal to: actual
      Which is: 0.0513
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0124
To be equal to: actual
      Which is: 0.0124
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0316
To be equal to: actual
      Which is: 0.0316
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.022
To be equal to: actual
      Which is: 0.022
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0321
To be equal to: actual
      Which is: 0.0321
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0326
To be equal to: actual
      Which is: 0.0326
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0328
To be equal to: actual
      Which is: 0.0328
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0518
To be equal to: actual
      Which is: 0.0518
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0523
To be equal to: actual
      Which is: 0.0523
Unit 0
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/MatrixTest.cc:891
      Expected: expected
      Which is: 0.0525
To be equal to: actual
      Which is: 0.0525
Unit 0
[=   3  LOG =]             MatrixTest.h :  39 | <=== Finished test case with 4 units 
[=   3  LOG =]             MatrixTest.h :  25 | <<< Closing test suite: MatrixTest 
[=   2  LOG =]             MatrixTest.h :  39 | <=== Finished test case with 4 units 
[=   2  LOG =]             MatrixTest.h :  25 | <<< Closing test suite: MatrixTest 
[=   1  LOG =]             MatrixTest.h :  39 | <=== Finished test case with 4 units 
[=   1  LOG =]             MatrixTest.h :  25 | <<< Closing test suite: MatrixTest 
[=   0  LOG =]             MatrixTest.h :  39 | <=== Finished test case with 4 units 
[=   0  LOG =]             MatrixTest.h :  25 | <<< Closing test suite: MatrixTest 
[  FAILED  ] MatrixTest.DelayedAlloc
[  ERROR   ] [UNIT 0] in /home/joseph/src/dash/dash/dash/test/TestPrinter.h:140
Failed
Testcase failed at least on one unit

Note that expected and actual always match up to the precision displayed. I assume that the difference appears in a low-significance bit (comparing double values for equality is never a good idea). However, I was unable to extract more information, even setting -DENABLE_TRACE_LOGGING=ON didn't help. Also note that the Debug builds pass successfully (I assume due to lower compiler optimization levels).

The used compiler was GCC 5.4.0 and OpenMPI 1.10.2 locally on my laptop.

Extent of a matrix local row

Consider the following code:
Consider the row of a two-dimensional matrix in DASH. My expectation was that a row has one dimension less than the number of dimensions of the matrix it belongs to, i.e., 1 for a 2D matrix.

However, the following example shows that a row is still two-dimensional with the extent in one dimension set to 1:

{
    size_t team_size = dash::Team::All().size();
    size_t myid = dash::Team::All().myid();

    dash::TeamSpec<2> teamspec_2d(team_size, 1);
    teamspec_2d.balance_extents();

    dash::SizeSpec<2> sspec(200, 30);
    dash::DistributionSpec<2> dspec(dash::BLOCKED, dash::BLOCKED);

    dash::Matrix<double, 2> matrix(sspec,dspec, dash::Team::All(), teamspec_2d);

    initialize(matrix);
    dash::barrier();

    auto row = matrix.local.row(0);
    auto num_elements = std::distance(row.begin(), row.end());
    std::cout << "[" << dash::myid() << "] local row 0 has " << num_elements << " elements" << std::endl;
    std::cout << "[" << dash::myid() << "] local row 0 has extent(0) " << row.extent(0) << std::endl;
    std::cout << "[" << dash::myid() << "] local row 0 has extent(1) " << row.extent(1) << std::endl;
    std::cout << "[" << dash::myid() << "] local row 0 has " << row.extents().size() << " dimensions" << std::endl;
}

which results in the following output:

[0] local row 0 has 30 elements
[0] local row 0 has extent(0) 1
[0] local row 0 has extent(1) 30
[0] local row 0 has 2 dimensions

The same holds true for a column:

[0] local column 0 has 100 elements
[0] local column 0 has extent(0) 100
[0] local column 0 has extent(1) 1
[0] local column 0 has 2 dimensions

Is this expected behavior? From how I understand the code in LocalMatrixRef, a row is meant to have N-1 columns:

template<typename T, dim_t NumDim, dim_t CUR, class PatternT>
inline LocalMatrixRef<T, NumDim, NumDim-1, PatternT>
LocalMatrixRef<T, NumDim, CUR, PatternT>::row(
  size_type n)
{
  return sub<0>(n);
}

dash::Array initializer_list constructor initializes with wrong values

The dash::Array constructor initializes the Array with the wrong values.
Array(
size_type nelem,
std::initializer_list<value_type> local_elements,
Team & team = dash::Team::All())

Example: A call with dash::Array(4, {1,2,3,4}) would result in a dash::Array with values {1,1,1,1}.

Update CHANGELOG

Some metric tonnes of features and bug fixes have been added since dash-0.2.0.
It's already hard to keep track, we have to update the CHANGELOG now and then to document our progress.

Also, there are some known limitations in the upcoming version 0.3.0 that must be documented in CHANGELOG before release.
The related issues are tagged as known-limitation:0.3.0:

https://github.com/dash-project/dash/labels/known-limitation%3A0.3.0

Integrate valgrind in CircleCI build target

Valgrind should be added in at least a minimal CI target to check for memory leaks and consistency defects.

Matrix.allocate(pattern) does not work if pattern leaves context

Use cases like that does not work:

matrix_t matrix;
{
   pattern_t pattern = ...
   matrix.allocate(pattern);
}
matrix[0] = 1;

This functionality is essential for the hdf5 API.

memory leak when accessing matrix with [][]

In an example application I noticed a huge memory consumption. The following is a minimal example that demonstrates this behavior. Like this, it exceeds my 8 gig memory and is killed. Make sure to disable your swap before testing.

#include <unistd.h>
#include <iostream>
#include <libdash.h>

int main( int argc, char* argv[]) {

    dash::init(&argc, &argv);

    size_t myid= dash::myid();
    size_t team_size= dash::Team::All().size();
    dash::TeamSpec<2> teamspec_2d(team_size, 1);
    teamspec_2d.balance_extents();

    uint32_t w= 8000;
    uint32_t h= 8000;

    dash::Matrix<uint32_t, 2> matrix( dash::SizeSpec<2>( w, h ),
        dash::DistributionSpec<2>( dash::BLOCKED, dash::BLOCKED ),
        dash::Team::All(), teamspec_2d );

    if ( 0 == myid ) {

        auto it= matrix.begin(); // only needed for NOLEAK case
        for(size_t i = 0; i < w; ++i) {
            for(size_t k = 0; k < h; ++k) {

#define LEAK
#ifdef LEAK
                matrix[i][k] = 100; // me makes mighty memory leak
# else /* LEAK*/
                *it= 100; ++it;
#endif /* LEAK */
            }
        }
    }

    return 0;
}

Change '#define LEAK' into '#define NOLEAK' to get a perfectly working code. This makes me think that there is no other cause of a memory leak. Did I confuse something? Anything that should not be used like this in a regular code?

Enable compiler report output for ICC

Intel compilers (icc, icpc) support reports for inlining and vectorization that should be enabled if available.

Remove dynamic memory allocation in dash::Matrix view types

See #22 and comments by @devreal

Use dash::io Concepts in dash's HDF5 writer

The following tasks have to be done:

• rename classes and files (remove HDF5 in names, as already in namespace)
• change api to implement dash::io concept
• adopt tests and examples
• Testing
• verify that dash::io concept is implemented correctly
• documentation

Branch: feat-hdf5-cleanup

DARTLocalityTest.UnitLocality fails

Hi Team!

the DARTLocalityTest.UnitLocality fails for mpirun -n 2. See travis build #98 for details.

[----------] run 2 out of 2 tests from DARTLocalityTest
[  ERROR   ] [UNIT 1] in /opt/dash/dash/test/DARTLocalityTest.cc:31
Expected: (ul->hwinfo.num_cores) > (0), actual: 0 vs 0
Unit 1
[  RUN     ] DARTLocalityTest.UnitLocality
[  FAILED  ] DARTLocalityTest.UnitLocality

Branch bug-dart-locality

dash::make_team_spec based on locality hierarchy

In its current intermediate implementation, the number of nodes and NUMA domains has to be specified explicitly for dash::make_team_spec.
Previously, global (homogeneous) hardware locality settings have been used implicitly.

It should be updated to the new locality concepts such that blocking factors are derived from the unit's lowest common locality domain.

Finally, this would then allow to optimize a team's logical Cartesian topology from pattern traits and hierarchical locality.

Add CI configuration using NastyMPI

Some defects are only visible by chance, like missing barriers after local initialization.
I think we could drastically reduce false positives in CI builds by integrating NastyMPI.

Migrate User Guide to dash/doc

The DASH user guide, currently written in Latex in the dash-papers/userguide repository, should be migrated to the DASH distribution documentation in Markdown format.

I migrated the NArray section for starters.

The user guide now is automatically generated from dash/doc on readthedocs:
https://dash.readthedocs.io/en/latest

DART locality valgrind issues

Valgrind reports the following memory access issues related to DART locality when running the DASH test suite

==19862== Conditional jump or move depends on uninitialised value(s)
==19862==    at 0x5A7CCAE: std::ostreambuf_iterator<char, std::char_traits<char> > std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<long>(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==19862==    by 0x5A7CEDC: std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::do_put(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==19862==    by 0x5A893F9: std::ostream& std::ostream::_M_insert<long>(long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==19862==    by 0x812341: std::enable_if<std::is_integral<int>::value, dash::util::LocalityJSONPrinter&>::type dash::util::operator<< <int>(dash::util::LocalityJSONPrinter&, int const&) (LocalityJSONPrinter.h:76)
==19862==    by 0x810645: dash::util::LocalityJSONPrinter::print_domain(int, dart_domain_locality_s const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (LocalityJSONPrinter.cc:87)
==19862==    by 0x70038B: dash::util::LocalityJSONPrinter::operator<<(dart_domain_locality_s const&) (LocalityJSONPrinter.h:41)
==19862==    by 0x6FD932: LocalRangeTest_ArrayBlockcyclic_Test::TestBody() (LocalRangeTest.cc:14)
==19862==    by 0x7F57C2: void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==    by 0x7EF71C: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==    by 0x7D3DB3: testing::Test::Run() (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==    by 0x7D474B: testing::TestInfo::Run() (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==    by 0x7D4E3E: testing::TestCase::Run() (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==
==19862== Use of uninitialised value of size 8
==19862==    at 0x5A7BB13: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==19862==    by 0x5A7CCD9: std::ostreambuf_iterator<char, std::char_traits<char> > std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<long>(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==19862==    by 0x5A7CEDC: std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::do_put(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==19862==    by 0x5A893F9: std::ostream& std::ostream::_M_insert<long>(long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==19862==    by 0x812341: std::enable_if<std::is_integral<int>::value, dash::util::LocalityJSONPrinter&>::type dash::util::operator<< <int>(dash::util::LocalityJSONPrinter&, int const&) (LocalityJSONPrinter.h:76)
==19862==    by 0x810645: dash::util::LocalityJSONPrinter::print_domain(int, dart_domain_locality_s const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (LocalityJSONPrinter.cc:87)
==19862==    by 0x70038B: dash::util::LocalityJSONPrinter::operator<<(dart_domain_locality_s const&) (LocalityJSONPrinter.h:41)
==19862==    by 0x6FD932: LocalRangeTest_ArrayBlockcyclic_Test::TestBody() (LocalRangeTest.cc:14)
==19862==    by 0x7F57C2: void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==    by 0x7EF71C: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==    by 0x7D3DB3: testing::Test::Run() (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==    by 0x7D474B: testing::TestInfo::Run() (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==
==19862== Conditional jump or move depends on uninitialised value(s)
==19862==    at 0x5A7BB1F: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==19862==    by 0x5A7CCD9: std::ostreambuf_iterator<char, std::char_traits<char> > std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<long>(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==19862==    by 0x5A7CEDC: std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::do_put(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==19862==    by 0x5A893F9: std::ostream& std::ostream::_M_insert<long>(long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==19862==    by 0x812341: std::enable_if<std::is_integral<int>::value, dash::util::LocalityJSONPrinter&>::type dash::util::operator<< <int>(dash::util::LocalityJSONPrinter&, int const&) (LocalityJSONPrinter.h:76)
==19862==    by 0x810645: dash::util::LocalityJSONPrinter::print_domain(int, dart_domain_locality_s const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (LocalityJSONPrinter.cc:87)
==19862==    by 0x70038B: dash::util::LocalityJSONPrinter::operator<<(dart_domain_locality_s const&) (LocalityJSONPrinter.h:41)
==19862==    by 0x6FD932: LocalRangeTest_ArrayBlockcyclic_Test::TestBody() (LocalRangeTest.cc:14)
==19862==    by 0x7F57C2: void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==    by 0x7EF71C: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==    by 0x7D3DB3: testing::Test::Run() (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==    by 0x7D474B: testing::TestInfo::Run() (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)

==19862== Conditional jump or move depends on uninitialised value(s)
==19862==    at 0x5A7CD0C: std::ostreambuf_iterator<char, std::char_traits<char> > std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<long>(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==19862==    by 0x5A7CEDC: std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::do_put(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==19862==    by 0x5A893F9: std::ostream& std::ostream::_M_insert<long>(long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==19862==    by 0x812341: std::enable_if<std::is_integral<int>::value, dash::util::LocalityJSONPrinter&>::type dash::util::operator<< <int>(dash::util::LocalityJSONPrinter&, int const&) (LocalityJSONPrinter.h:76)
==19862==    by 0x810645: dash::util::LocalityJSONPrinter::print_domain(int, dart_domain_locality_s const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (LocalityJSONPrinter.cc:87)
==19862==    by 0x70038B: dash::util::LocalityJSONPrinter::operator<<(dart_domain_locality_s const&) (LocalityJSONPrinter.h:41)
==19862==    by 0x6FD932: LocalRangeTest_ArrayBlockcyclic_Test::TestBody() (LocalRangeTest.cc:14)
==19862==    by 0x7F57C2: void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==    by 0x7EF71C: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==    by 0x7D3DB3: testing::Test::Run() (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==    by 0x7D474B: testing::TestInfo::Run() (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==    by 0x7D4E3E: testing::TestCase::Run() (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==19862==

IMO we should aim at being valgrind-clean to prevent bugs caused by unexpected or undefined behavior.

Test UnorderedMapTest.Initialization fails in MPI_Win_detach for OpenMPI 2.0

[UPDATE:] Forgot to change to github repository before starting the testing. New errors arise on the code from the github repo so I updated the bug report to reflect them instead of the obsolete code.[/UPDATE]

Executing the test suite on our Linux cluster (Laki) using OpenMPI fails with the following message:

$ mpirun -n 4 build/dash/dash-test-mpi                                                                                    
#### Starting test on unit 0 (n041701 PID: 27191)
#### Starting test on unit 1 (n041701 PID: 27192)
#### Starting test on unit 2 (n041701 PID: 27193)
#### Starting test on unit 3 (n041701 PID: 27194)
[----------] 39 tests will be run.
[----------] run 4 out of 4 tests from UnorderedMapTest
[  RUN     ] UnorderedMapTest.Initialization
[=   0  LOG =]       UnorderedMapTest.h :  20 | >>> Test suite: UnorderedMapTest 
[=   0  LOG =]       UnorderedMapTest.h :  21 | >>> Hostname: n041701 PID: 27191 
[=   0  LOG =]       UnorderedMapTest.h :  32 | ===> Running test case with 4 units ... 
[=   1  LOG =]       UnorderedMapTest.h :  20 | >>> Test suite: UnorderedMapTest 
[=   1  LOG =]       UnorderedMapTest.h :  21 | >>> Hostname: n041701 PID: 27192 
[=   1  LOG =]       UnorderedMapTest.h :  32 | ===> Running test case with 4 units ... 
[=   2  LOG =]       UnorderedMapTest.h :  20 | >>> Test suite: UnorderedMapTest 
[=   2  LOG =]       UnorderedMapTest.h :  21 | >>> Hostname: n041701 PID: 27193 
[=   2  LOG =]       UnorderedMapTest.h :  32 | ===> Running test case with 4 units ... 
[=   3  LOG =]       UnorderedMapTest.h :  20 | >>> Test suite: UnorderedMapTest 
[=   3  LOG =]       UnorderedMapTest.h :  21 | >>> Hostname: n041701 PID: 27194 
[=   3  LOG =]       UnorderedMapTest.h :  32 | ===> Running test case with 4 units ... 
[n041701:27193] *** An error occurred in MPI_Win_detach
[n041701:27193] *** reported by process [1999765505,2]
[n041701:27193] *** on win rdma window 6
[n041701:27193] *** MPI_ERR_OTHER: known error not in list
[n041701:27193] *** MPI_ERRORS_ARE_FATAL (processes in this win will now abort,
[n041701:27193] ***    and potentially your MPI job)
[n041701:27186] 3 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[n041701:27186] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

Used compiler was GCC 6.1.0 and for OpenMPI the 2.0.1 release was used. With the latest master branch (3.0.0a) the output is more specific:

$ mpirun -n 4 build/dash/dash-test-mpi 
#### Starting test on unit 0 (n041701 PID: 29608)
#### Starting test on unit 1 (n041701 PID: 29609)
#### Starting test on unit 2 (n041701 PID: 29610)
#### Starting test on unit 3 (n041701 PID: 29611)
[----------] 39 tests will be run.
[----------] run 4 out of 4 tests from UnorderedMapTest
[=   1  LOG =]       UnorderedMapTest.h :  20 | >>> Test suite: UnorderedMapTest 
[=   1  LOG =]       UnorderedMapTest.h :  21 | >>> Hostname: n041701 PID: 29609 
[=   3  LOG =]       UnorderedMapTest.h :  20 | >>> Test suite: UnorderedMapTest 
[=   3  LOG =]       UnorderedMapTest.h :  21 | >>> Hostname: n041701 PID: 29611 
[  RUN     ] UnorderedMapTest.Initialization
[=   2  LOG =]       UnorderedMapTest.h :  20 | >>> Test suite: UnorderedMapTest 
[=   2  LOG =]       UnorderedMapTest.h :  21 | >>> Hostname: n041701 PID: 29610 
[=   1  LOG =]       UnorderedMapTest.h :  32 | ===> Running test case with 4 units ... 
[=   3  LOG =]       UnorderedMapTest.h :  32 | ===> Running test case with 4 units ... 
[=   0  LOG =]       UnorderedMapTest.h :  20 | >>> Test suite: UnorderedMapTest 
[=   0  LOG =]       UnorderedMapTest.h :  21 | >>> Hostname: n041701 PID: 29608 
[=   0  LOG =]       UnorderedMapTest.h :  32 | ===> Running test case with 4 units ... 
[=   2  LOG =]       UnorderedMapTest.h :  32 | ===> Running test case with 4 units ... 
[  ERROR   ] [UNIT 0] in /zhome/academic/HLRS/hlrs/hpcjschu/src/dash/dash-development/dash/test/UnorderedMapTest.cc:50
Value of: existing.second
  Actual: true
Expected: false
Unit 0
[n041701:29609] *** An error occurred in MPI_Get
[n041701:29609] *** reported by process [1856176129,1]
[n041701:29609] *** on win rdma window 6
[n041701:29609] *** MPI_ERR_RMA_RANGE: invalid RMA address range
[n041701:29609] *** MPI_ERRORS_ARE_FATAL (processes in this win will now abort,
[n041701:29609] ***    and potentially your MPI job)
[n041701:29603] 2 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[n041701:29603] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

Inconsistent IndexType and SizeType across classes

While scanning through the dash::Matrix code, I realized that there are inconsistencies among classes with regards to the IndexType and SizeType used.

While dash::Matrix passes these types on to some members, e.g., Pattern, other classes in the hierarchy are agnostic of the IndexType the user has chosen as template parameter and instead use the default index type (internal::default_signed_index). This will likely cause problems for larger matrices, esp. since the default is 32 bit.
All classes along the member hierarchy should use the same types to avoid issues that are a nightmare to debug.

Moreover, I propose to use 64 bit indexes on 64 bit architectures and 32 bit indexes on 32 bit architectures by default. Implicitly limiting the IndexType to 32 bit on 64 bit architectures is counter-intuitive and will lead to confusion among users. My point regarding debuggability holds true here as well.

Re-consolidate CI tasks in CircleCI

Each task (line) in the circleci.yml file adds an implicit barrier to the parallel execution of the containers.
As the trend in DASH CI is towards a subset of targets executed in each environment, this massively rises CI time.

For example the valgrind check is run if and only if the container is openmpi2_vg and build type is debug. All other containers do not even run the debug target.

Yes, I know the output is much more pretty if one target per line is set, but this can also be accomplished by copying the logs to the artifacts directory.

We should discuss this.

CopyTests fail in minimal build target

If building the dash minimal target, dash::copy is not working properly. This can also be reproduced on a non virtualized / CI environment.

• CopyTest.AsyncGlobalToLocalBlock

/opt/dash/dash/test/CopyTest.cc:764
      Expected: num_elem_per_unit
      Which is: 20
To be equal to: dest_end.get() - local_copy
      Which is: 0
Unit 0

See this build for details

Tests hang using OpenMPI 2.0

The following tests hang when using openmpi 2.0. For testing and verification the new docker container dashproject/ci:openmpi2 can be used. The container provides a openmpi 2.0 environment which is build from sources.

It is also possible to ssh into the container on circleci and debug there.

OpenMPI v2.0

2 Units

• SharedTest.AtomicAdd
• TransformTest.Array*

3 Units

• CopyTest.AsyncLocalToGlobPtr

Locality Tests fail in minimal build target

If no hardware locality lib is loaded, some hardware metrices are set to "-1". However the DARTLocalityTest checks that these values are greater 0. Maybe some other test failures of the minimal target are symptoms of that bug.

See this example build.

ul->hwinfo.max_threads = -1

Evaluate interprocedural (icc) / link-time (gcc) optimization

Add compiler / linker flags and evaluate benchmark results (dash/examples/bench.*):

• gcc: -lto
• icc: -qipo or -ipo depending on version

Specific Testcase as a build target

Build time matters for developers. If if implement a feature or fix a bug I immediately want to have a new build and the unit test result. I also do not want to build all the other unit tests, which might not affect my ticket at all. All I need is a usable possibility to build an run just one specific unit test.

dart_gptr not fully initialized

Throwing valgrind at the DASH test suite I see reports similar to the one below:

==16575== Conditional jump or move depends on uninitialised value(s)
==16575==    at 0x81E6EE: operator==(dart_gptr_t const&, dart_gptr_t const&) (GlobPtr.cc:22)
==16575==    by 0x7C8BD5: dash::allocator::DynamicAllocator<int>::deallocate(dart_gptr_t)::{lambda(std::pair<int*, dart_gptr_t>)#1}::operator()(std::pair<int*, dart_gptr_t>) (DynamicAllocator.h:324)
==16575==    by 0x7CA0BA: dash::allocator::DynamicAllocator<int>::deallocate(dart_gptr_t)::{lambda(std::pair<int*, dart_gptr_t>)#1} std::for_each<__gnu_cxx::__normal_iterator<std::pair<int*, dart_gptr_t>*, std::vector<std::pair<int*, dart_gptr_t>, std::allocator<std::pair<int*, dart_gptr_t> > > >, dash::allocator::DynamicAllocator<int>::deallocate(dart_gptr_t)::{lambda(std::pair<int*, dart_gptr_t>)#1}>(__gnu_cxx::__normal_iterator<std::pair<int*, dart_gptr_t>*, std::vector<std::pair<int*, dart_gptr_t>, std::allocator<std::pair<int*, dart_gptr_t> > > >, dash::allocator::DynamicAllocator<int>::deallocate(dart_gptr_t)::{lambda(std::pair<int*, dart_gptr_t>)#1}, dash::allocator::DynamicAllocator<int>::deallocate(dart_gptr_t)::{lambda(std::pair<int*, dart_gptr_t>)#1}) (stl_algo.h:3767)
==16575==    by 0x7C8CA0: dash::allocator::DynamicAllocator<int>::deallocate(dart_gptr_t) (DynamicAllocator.h:320)
==16575==    by 0x7C5A1B: dash::GlobDynamicMem<int, dash::allocator::DynamicAllocator<int> >::commit_detach() (GlobDynamicMem.h:1038)
==16575==    by 0x7C33FE: dash::GlobDynamicMem<int, dash::allocator::DynamicAllocator<int> >::commit() (GlobDynamicMem.h:666)
==16575==    by 0x7BF782: GlobDynamicMemTest_LocalVisibility_Test::TestBody() (GlobDynamicMemTest.cc:446)
==16575==    by 0x7F57C2: void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==16575==    by 0x7EF71C: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==16575==    by 0x7D3DB3: testing::Test::Run() (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==16575==    by 0x7D474B: testing::TestInfo::Run() (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)
==16575==    by 0x7D4E3E: testing::TestCase::Run() (in /home/joseph/src/dash/dash/build/bin/dash-test-mpi)

The reason seems to be that none of the allocation functions in dart_globmem.c initialize the flag field of the gptr argument. Working on a fix.

Travis add support for long running Tests

Currently the test script has a maximum execution time of 10 minutes. Travis cancels the build if no output is generated within 10 minutes.

Currently the test selection runs ~ 5 minutes.

Todo: Output test results for each test in ci script.

Support PAPI versions greater 5.4.1

Currently it is not possible to compile DASH with a PAPI version > 5.4.1. The problem cannot be easily fixed, as functions from string.h and papi.h clash.

In file included from /usr/include/c++/5/cstring:42:0,
                 from /opt/dash/dash/include/dash/internal/StreamConversion.h:14,
                 from /opt/dash/dash/include/dash/internal/Logging.h:5,
                 from /opt/dash/dash/include/dash/util/Timer.h:20,
                 from /opt/dash/dash/src/util/Timer.cc:1:
/usr/include/string.h:525:40: error: declaration of 'int ffsll(long long int) throw ()' has a different exception specifier
      __THROW __attribute__ ((__const__));
                                        ^
In file included from /opt/dash/dash/include/dash/util/internal/TimestampPAPI.h:9:0,
                 from /opt/dash/dash/include/dash/util/Timer.h:14,
                 from /opt/dash/dash/src/util/Timer.cc:1:
/opt/papi/include/papi.h:1024:8: error: from previous declaration 'int ffsll(long long int)'
    int ffsll(long long lli); //required for --with-ffsll and used in extras.c/papi.c

some examples in "root/dash/examples" don't build

The array examples series don't build in my VM on development branch.
In contrast ex.01.hello for example does build.
When I execute "build.sh" they are build successfully and are found in the folders:
"root/build/bin" and "root/build/dash".

When I try building the examples after building DASH with make in the folder "root/dash/examples/.."
I get an error. The outut of my building ex.02.array attempt is in the attachment.

ex.02.array make log.txt

MPI versions of travis ci do not support MPI3

Hi Team!

unfortunately, no travis build environment currently supports packages for MPI3 [1]. Hence our CI scripts will fail as we depend on MPI3 [2]. Compiling a current version of MPI inside the travis env is not a possibility, as the compilation time reaches hours.
The recommended workaround for this problem is to use a docker container [3].

How should we proceed?

References

• [1] https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments
• [2] Build Output
• [3] travis-ci/travis-ci#5821

Buggy dash testing in docker CI environment

faulty tests

The following tests report errors in the docker test environment, using the build config from dash-ci.sh

mpich

• ArrayLargeStructTest (segfault)
• HDF5 (See Issue #1)

4 Units

• TeamLocalityTest.* (Assert)

8 Units

• DARTOnesidedTest.GetBlockingTwoBlocks (Assert)

openmpi

3 Units

• FillTest.TestAllItemsFilled (segfault)
• ForEachTest.ForEachWithIndex (segfault)

4 Units

• TeamLocalityTest.*(Assert)
• MatrixTest.DelayedAlloc (segfault)

8 Units

• TransformTest.ArrayGlobalPlusLocalBlocking

host_topology: usage of strncat might overflow buffer

GCC issues the following warning for the function dart__base__host_topology__module_locations:

In function ‘strncat’,
    inlined from ‘dart__base__host_topology__module_locations’ at /home/joseph/src/dash/dash/dart-impl/base/src/internal/host_topology.c:103:13:
/usr/include/x86_64-linux-gnu/bits/string3.h:156:10: warning: call to __builtin___strncat_chk might overflow destination buffer
   return __builtin___strncat_chk (__dest, __src, __len, __bos (__dest));

The way strncat is used here is dangerous since the size parameter does not describe the size of the destination buffer. The man page reads:

The strncat() function is similar, except that
* it will use at most n bytes from src; [...]

A safe way would be to use snprintf or the like to control the amount of data written and thus to avoid memory corruption.

Integrate Huan's Async Progress Extension

Huan implemented an asynchronous progress engine for the DART-MPI runtime.

See: "Asynchronous progress design for a MPI-based PGAS one-sided communication system". Huan Zhou, Jose Gracia. 2016.

The implementation is in branch feat-mpi-progress and must be merged.
Instead of adding another #ifdef cascade, the progress-runtime should be implemented as alternative
runtime implementation.

Execution Policies for Algorithms

Execution policies should be introduced that allow to specify the behavior of DASH algorithms.

Usage should correspond to execution policies as proposed for the STL:
http://en.cppreference.com/w/cpp/experimental/transform_reduce

For example:

// The specialization of dash::transform for one-sided execution results in
// MPI_Accumulate + MPI_Win_flush:

// defaults to one-sided execution policy:
dash::transform(
  in_a_first, in_a_last,
  in_b_first,
  out_first);
// specify one-sided execution explicitly:
dash::transform(
  dash::execution::onesided,
  in_a_first, in_a_last,
  in_b_first,
  out_first);

// For collective execution, dash::transform resolves to MPI_Allreduce which
// is more efficient but requires a coordinated schedule: 
dash::transform(
  dash::execution::collective,
  in_a_first, in_a_last,
  in_b_first,
  out_first);

FindTest.* Testgroup fails

Hi,

the tests in FindTest.* currently fail:

[  RUN     ] FindTest.TestSimpleFind
*** stack smashing detected ***: /opt/dash/build-ci/20161015-103653/Release/bin/dash/test/mpi/dash-test-mpi terminated
[aaae69326d4c:02207] *** Process received signal ***
[aaae69326d4c:02207] Signal: Aborted (6)
[aaae69326d4c:02207] Signal code:  (-6)

See (travis build #97)[https://travis-ci.org/dash-project/dash/builds/167857538] for details. Branch bug-stl-find

Compiler errors in 32-bit build

Some template specializations are duplicate on 32-bit targets, like:

struct dash::dart_datatype<unsigned int>
struct dash::dart_datatype<size_t>

Also, compiler warnings have been reported for gcc-6.4.

Document CI procedure in DASH distribution docs

To enable other contributors to extend the continuous integration builds, the current CI procedure should be explained in the DASH distribution docs (/doc/CI/...).

CI documentation should be comprehensible to an audience without prior experience in container configurations and summarize the relevant aspects from Travis/CircleCI documentation by example.

CI does not report critical failures

See log output of several recent CI tests, for example:
https://travis-ci.org/dash-project/dash/builds/171929653

Critical failures are recognized by the CI script but overall result is "PASSED".

Also, there is currently a corrupt free in dart-mpi/dart_segment.

DART-IF: Send/Receive

Extend dart-if by send/receive communication, such as:

dart_ret_t dart_send(
  void      * sendbuf,
  size_t      nbytes,
  dart_unit_t unit);

dart_ret_t dart_recv(
  void      * recvbuf,
  size_t      nbytes,
  dart_unit_t unit);

dart_ret_t dart_team_send(
  void      * sendbuf,
  size_t      nbytes,
  int         tag,
  dart_team_t team);

dart_ret_t dart_team_recv(
  void      * recvbuf,
  size_t      nbytes,
  int         tag,
  dart_team_t team);