nek5000/gslib

sparse communication library

GSLIB

• Scalable Many-to-Many (neighborhood) gather-scatter collectives
• Robust GLL interpolation on hexahedral spectral element meshes

Build Instructions

The build system relies on GNU Make with the make command. To compile gslib just run:

make CC=mpicc

Different make options are available (see Makefile).

Applications

[1] Nek5000: Nek5000 open-source, spectral element code.

[2] CEED: Co-design center for Efficient Exascale Discretizations.

[3] Nektar++: Nektar++ open-source spectral/hp element code.

[4] Libparanumal: Accelerated finite element flow solvers .

gslib's Issues

Add hard exit condition in newton_vol()

Just in case convergence of the residual stalls

Add async gs_op()

rdtsc not supported on non-86X

The tests

• findpts_el_2_test2
• findpts_el_3_test2
• poly_test2
• sort_test2

are using a time step counter which is not supported on non-86X systems. Note, the core-library is not affected.

Add one sided MPI exchange method

Add a PushErrorHandler

Sets a user routine to be called on detection of errors. Currently we just call exit().

Make unit tests producing a test answer (successful/failed)

Run unit tests on Travis

Run all existing test in src e.g. on Travis

Compiling error for non-blocking allreduce on Blue Gene/Q

Hi,

The non-blocking allreduce function MPI_Iallreduce belongs to the MPI-3 standard, which is not supported on Blue Gene/Q (and possibly other machines), thus causing a compilation error for gslib version 1.0.3. Is there a workaround?

Nicolas

Add additional exchange methods for sparse collective communication

Currently we support crystal router only. This PR suggests to add pairwise and all_reduce.

Fix expected identifier bug on clang builds

Hi,

I am trying to build gslib using mvapich2 installed with clang and I get the following error:

 /opt/local/include/mpich-mp/mpi.h:116:56: error: expected identifier

 static const MPI_Datatype mpich_mpi_char
 MPICH_ATTR_TYPE_TAG(char)

                = MPI_CHAR;

                                                        ^

 /opt/local/include/mpich-mp/mpi.h:24:106: note: expanded from macro
 'MPICH_ATTR_TYPE_TAG'

 #    define MPICH_ATTR_TYPE_TAG(type)
 __attribute__((type_tag_for_datatype(MPI,type)))

Turns out this error has affected others before trying to build on apple machines - all the way back in 2015: https://lists.mcs.anl.gov/pipermail/nek5000-users/2015-May/003325.html

The problem is nek5000 has #defined MPI to be 1, but the mpich header file
'mpi.h' uses the symbol 'MPI' (for a completely different reason).

I have a branch locally that resolves this problem. All that needs to be done is change the macro name MPI to something more specific, such as GSLIB_USE_MPI.

Add MPI_Neighbor_alltoall exchange method

Check if MPI supports required NBCs

Add microbenchmark

So far MPI benchmark packages do not assess the performance of sparse collectives. Possible benchmarks need to include:

• different message sizes and virtual topologies
• blocking and non-blocking performance

Document C and Fortran interface in README

negating unsigned int

https://github.com/Nek5000/gslib/blob/master/src/sort_imp.h#L282

Dependencies on Nek5000

We have to get rid of some remaining dependencies to Nek5000 to make it an independent library:

• call to nek_exit() in fail.c
• call to nek_mxm in tensor.h

Add proper C API

Unsupported assignments between restricted pointers

IBM compilers complains about:

"src/lob_bnd.c", line 153.29: 1506-1418 (E) Assignment between restrict pointers "br" and "br_" is not allowed. Only outer-to-inner scope assignments between restrict pointers are allowed.

Cray compiler doesn't complain but optimization seems to break the code.

Add non-blocking crystal router

I can think of two possible implementations:

1. Progress thread running in the background
2. User level thread (ULT) with network interrupts

Add GPUDirect RDMA support

Add OCCA support

Extend findpts_eval() to perform interpolation on the source rank

Currently the interpolation step is performed on the target rank i.e. not on the source rank where interpolation points are stored. For large ratios of source/target ranks the interpolation step becomes a critical bottleneck. To resolve this we should support a different mode where the interpolation step is performed on the source rank.

LDFLAGS when BLAS=1

Hi Stefan,

Currently, if the user set BLAS=1, he has to set LDFLAGS="-lcblas" as well to make
sure the inking process is successful. Shall we edit the Makefile so that we do this
on our end? simply adding "-lcblas" to LDFLAGS variable?

Test Fortran bindings