hypre-space/hypre

Parallel solvers for sparse linear systems featuring multigrid methods.

HYPRE is a library of high performance preconditioners and solvers featuring multigrid methods for the solution of large, sparse linear systems of equations on massively parallel computers.

For documentation, see our readthedocs page.

For information on code development, build requirements, publications, and more, see our Wiki page.

To install HYPRE, please see either the documentation or the file INSTALL.md.

An overview of the HYPRE release history can be found in the file CHANGELOG.

Support information can be found in the file SUPPORT.md.

License

HYPRE is distributed under the terms of both the MIT license and the Apache License (Version 2.0). Users may choose either license, at their option.

All new contributions must be made under both the MIT and Apache-2.0 licenses.

See LICENSE-MIT, LICENSE-APACHE, COPYRIGHT, and NOTICE for details.

SPDX-License-Identifier: (Apache-2.0 OR MIT)

LLNL-CODE-778117

hypre's Issues

Accessing the matrix solution in Fortran

Hi,

I was wondering how I can get the matrix solution in Fortran. In reference to the Example 5, the solutions obtained are printed out with the HYPRE_IJVectorPrint subroutine. But if I want to access the values inside the main program (in the form of HYPRE_GetIJVector(x,ierr)), which subroutine do I use?

Thank you very much.

CUDA API differences?

We are experimentally exploring the use of the CUDA enabled version of Hypre in an application already using it.

Because there is no documentation readily available (#69), we are not sure what (if any) API changes have been made, and (if so) what changes we need make on our end to use it.

Is that something someone can help us with?
Thanks.

New hypre release 2.18.2

Hello @hypre-space/developers,

We will do another patch release, so please hold off on making changes to master.

Thanks!

-Rob

Interest in contributing CMake improvements?

I'm currently in the process of adding hypre support to another project that uses CMake, and noticed that hypre currently does not install a CMake config file. Doing so would make consuming hypre in other CMake projects trivial, as CMake can automatically bring in all the dependencies:

# find pre-compiled HYPRE
find_package(HYPRE)
# link my_target against HYPRE and dependencies, e.g. MPI, SuperLU
target_link_libraries(my_target PUBLIC HYPRE)

There's some easy boilerplate that enables this, but to make it useful by bundling all the information about dependencies requires some other changes, namely requiring a more recent version of CMake and using the target_* interface.

I've made most of the required changes already, as they didn't take long. Would it be useful if I finished it off and submitted a PR?

Left preconditioner for solvers

For some problems, left preconditioner i.e., solving MAx = Mb instead of Ax =b (M is the preconditioner) works much better than using right preconditoner.

However, by default all preconditioners in GMRES, etc is applied from the right.

I was wandering if left/right option can be added??

Thank you,

GPU Guidance

Hi, I'm looking to use Hypre's GPU capabilities, but since this seems experimental / undocumented for now, I'm asking here if you might be able to give me some guidance. I'm working on a Fortran codebase and access Hypre's C interface through ISO C bindings. We use two interfaces: Struct's hybrid solve, and BoomerAMG with SStruct.

1. First questions are about which interfaces support which GPU backends. Is OpenMP device offloading supported for both Struct hybrid and SStruct BoomerAMG? If not, how about the CUDA backend? Worst case (for us), does the IJ interface support OpenMP device offloading or CUDA?

2. How portable is OpenMP-enabled Hypre (e.g., Nvidia, AMD, and upcoming Intel GPUs)? I noticed during build that --with-cuda seems to be required when --with-device-openmp is enabled. For now I only need to support Nvidia GPUs, but would like to keep portability in mind.

3. Should memory already be on the GPU when we call Hypre routines? If so, does this only apply to matrix and vector values, or to the grid, graph, and stencil as well?

4. Do you have any examples or documentation you could point me to for GPU-enabled Hypre use?

Thanks!

ex5.c erro: cannot convert ‘double*’ to ‘HYPRE_Real* {aka long double*}’ for argument ‘2’ to ‘HYPRE_Int HYPRE_PCGGetFinalRelativeResidualNorm(HYPRE_Solver, HYPRE_Real*)’

Dear sir:
I compile ex5 with kdevelop, but i met the following error.
cannot convert ‘double*’ to ‘HYPRE_Real* {aka long double*}’ for argument ‘2’ to ‘HYPRE_Int HYPRE_PCGGetFinalRelativeResidualNorm(HYPRE_Solver, HYPRE_Real*)’
How to resolve this problem.

builds internal BLAS and LAPACK regardless of --with-blas options

The HYPRE source contains an internal copy of blas (also lapack) which it builds and uses in case blas is not already available.

If blas is available on the system then it can be identified at configure time with --with-blas and --with-blas-lib flags (also --with-lapack --with-lapack-lib).

But the problem is that even if --with-blas is used, the hypre build proceeds to build and link the internal copy of blas anyway. You can see it with nm -D libHYPRE-2.17.0.so | less, which shows symbols like hypre_dgemm, hypre_dtrmm etc.

The internal blas and lapack source should not be built and should not be linked, if configure flags have identified that a system BLAS and LAPACK is available.

QUESTION: example files

I had a look on the example files, and found that running for e.g. ex5 with BoomerAMG on a single core takes less time, than running it on all 4 cores of my processor, although all four of them are running at 100%. I stopped the time for Setup() and Solve().

I used standard settings, with a mesh-size of 2048x2048 on a core i7 7700.
Is there so much overhead that it is less efficient for this mesh size, or am I doing something wrong?

thanks, Chris

Summit GPU Run

Hi,

I'm running the code on Summit with GPU support but, while it compiles and runs fine, the run itself seems to be ignoring the GPU and giving the wrong output. Here is the detailed process of what I did:

$ git clone https://github.com/hypre-space/hypre.git
$ cd hypre/src
$ ./configure --prefix=`pwd`/build/opt --with-cuda CUDA_HOME=/sw/summit/cuda/10.1.105
$ make install -j
$ cd test
$ make struct
$ bsub -nnodes 1 -W 10 -env "all" -P XXXXXX -o struct.out jsrun -n 1 -c 1 -g 1 ./struct

On completion, the output reads:

Running with these driver parameters:
  (nx, ny, nz)    = (10, 10, 10)
  (istart[0],istart[1],istart[2]) = (-3, -3, -3)
  (Px, Py, Pz)    = (1, 1, 1)
  (bx, by, bz)    = (1, 1, 1)
  (px, py, pz)    = (0, 0, 0)
  (cx, cy, cz)    = (1.000000, 1.000000, 1.000000)
  (conx,cony,conz)= (0.000000, 0.000000, 0.000000)
  (n_pre, n_post) = (1, 1)
  dim             = 3
  skip            = 0
  sym             = 1
  rap             = 0
  relax           = 1
  jump            = 0
  solver ID       = 0
=============================================
Struct Interface:
=============================================
Struct Interface:
  wall clock time = 0.210000 seconds
  wall MFLOPS     = 0.000000
  cpu clock time  = 0.200000 seconds
  cpu MFLOPS      = 0.000000

=============================================
Setup phase times:
=============================================
SMG Setup:
  wall clock time = 0.020000 seconds
  wall MFLOPS     = 0.000000
  cpu clock time  = 0.010000 seconds
  cpu MFLOPS      = 0.000000

=============================================
Solve phase times:
=============================================
SMG Solve:
  wall clock time = 0.000000 seconds
  wall MFLOPS     = 0.000000
  cpu clock time  = 0.000000 seconds
  cpu MFLOPS      = 0.000000


Iterations = 0
Final Relative Residual Norm = 0.000000e+00

I did the same on my computer with a GPU and the output runs fine with the last line reading:

...
Iterations = 5
Final Relative Residual Norm = 1.375560e-07

If you have any Summit specific ways to compile or run the code I would love to hear about them since this seems to be a Summit specific issue.

Thanks!
Pedro

bootstrap: provide path to update-release.sh

src/config/bootstrap now invokes update-release.sh in the docs subdir.

Since it's fairly common to not include This Directory (.) in a user's PATH, the bootstrap script will be more robust with
(cd docs; ./update-release.sh)
instead of
(cd docs; update-release.sh)

documenting ABI soname policy

The current 2.18.1 hypre code sets the soname for HYPRE shared libraries using HYPRE_RELEASE_VERSION,
e.g. SONAME = libHYPRE-${HYPRE_RELEASE_VERSION}${HYPRE_LIB_SUFFIX} in src/lib/Makefile

HYPRE_RELEASE_VERSION is set from HYPRE_VERSION in src/config/Makefile.config.in. HYPRE_VERSION is the full major.minor.patch version set in src/CMakeLists.txt or src/config/configure.in. So the current value of HYPRE_VERSION is 2.18.1.

In principle it means 2.18.1 is not ABI compatible with 2.18.0. Is this the intention, or should the 2.18 series be considered compatible (in which case the soname could be libHYPRE-2.18.so rather than libHYPRE-2.18.1.so).

If each patch release is intended to be considered incompatible, then could I trouble you to consider putting a comment in INSTALL.md or README.md to document that? That would help avoid confusion about the ABI policy in the future, since usually libraries are ABI compatible across patch releases.

New hypre release 2.18.0

Hello @hypre-space/developers,

We are in the process of doing a new hypre release, so please hold off on making changes to the master branch until it is done, except for @liruipeng who will be merging branch amg-setup today. Also, don't make changes to the amg-setup branch without first talking to @liruipeng. There are a number of things we were hoping to get into this release, but we can do another one really soon (say within 2 weeks), so no worries if you miss this release.

Thanks!

-Rob

CUDA test error

Hi,
I'm looking to use Hypre's GPU capabilities. But when I type hypre/src/configure to configure the package for my system, The CUDA test example in hypre/src/test/ works fine. But the test example in hypre/src/example/ will have error. In fact, the other software I use, PARFLOW, requires HYPRE. When I compiled PARFLOW, I encountered the same error as when I compiled the test example in hypre/src/example/. Here are some details：
./configure --prefix=$HYPRE_DIR --with-MPI --with-cuda --enable-shared CC=mpicc CXX=mpicxx FC=mpif90 F77=mpif77

hypre-2.18.2/src/examples$ make
mpicc -g -Wall -I../hypre/include -DHAVE_CONFIG_H -DHYPRE_TIMING -c ex1.c
In file included from /usr/local/cuda-9.2/include/thrust/execution_policy.h:24:0,
                 from ../hypre/include/_hypre_utilities.h:1129,
                 from ../hypre/include/interpreter.h:11,
                 from ../hypre/include/HYPRE_struct_ls.h:1105,
                 from ex1.c:32:
/usr/local/cuda-9.2/include/thrust/detail/execution_policy.h:21:1: error: unknown type name ‘namespace’
 namespace thrust
 ^
/usr/local/cuda-9.2/include/thrust/detail/execution_policy.h:22:1: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ tok
en {
 ^
。。。

In file included from /usr/local/cuda-9.2/include/thrust/detail/type_traits/pointer_traits.h:23:0,
                 from /usr/local/cuda-9.2/include/thrust/detail/raw_pointer_cast.h:20,
                 from /usr/local/cuda-9.2/include/thrust/detail/execute_with_allocator.h:20,
                 from /usr/local/cuda-9.2/include/thrust/execution_policy.h:25,
                 from ../hypre/include/_hypre_utilities.h:1129,
                 from ../hypre/include/interpreter.h:11,
                 from ../hypre/include/HYPRE_struct_ls.h:1105,
                 from ex1.c:32:
/usr/local/cuda-9.2/include/thrust/iterator/iterator_traits.h:34:20: fatal error: iterator: No such file or directory
compilation terminated.
Makefile:53: recipe for target 'ex1.o' failed
make: *** [ex1.o] Error 1

when I change ex1.c to ex1.cpp , it will report another error :

hypre-2.18.2/src/examples$ make
mpicxx -g -Wall -Wno-deprecated -I../hypre/include -I.. -DHAVE_CONFIG_H -DHYPRE_TIMING    -c -o ex1.o ex1.cpp
In file included from ../hypre/include/interpreter.h:11:0,
                 from ../hypre/include/HYPRE_struct_ls.h:1105,
                 from ex1.cpp:32:
../hypre/include/_hypre_utilities.h: In function ‘hypre_int hypre_cuda_get_num_threads()’:
../hypre/include/_hypre_utilities.h:1223:18: error: ‘blockDim’ was not declared in this scope
          return (blockDim.x);
                  ^
../hypre/include/_hypre_utilities.h: In function ‘hypre_int hypre_cuda_get_thread_id()’:
../hypre/include/_hypre_utilities.h:1241:18: error: ‘threadIdx’ was not declared in this scope
          return (threadIdx.x);
                  ^
../hypre/include/_hypre_utilities.h:1243:32: error: ‘blockDim’ was not declared in this scope
          return (threadIdx.y * blockDim.x + threadIdx.x);

how to set number of threads?

I installed hypre 2.18.2 in my pc ( hypre openmp supported ).
then in my test.c , i can see HYPRE_USING_OPENMP = 1.
then ,how can i set number of threads per process used?

Compiling ex5f.f without MPI

Hi,

I am trying to compile one of the Fortran examples (ex5f.f) without the MPI implementation, but I can't figure out how to tweak the code to safely ignore the MPI-related subroutines.

Just so you know, I have installed Hypre on my Windows machine with CMake, with the following configuration options in CMakeLists.txt:

---

option(HYPRE_SHARED "Build a shared library" OFF)
option(HYPRE_BIGINT "Use long long int for HYPRE_Int" OFF)
option(HYPRE_SEQUENTIAL "Compile without MPI" ON)
option(HYPRE_TIMING "Use HYPRE timing routines" OFF)
option(HYPRE_USING_HYPRE_BLAS "Use internal BLAS library" ON)
option(HYPRE_USING_HYPRE_LAPACK "Use internal LAPACK library" ON)
option(HYPRE_NO_GLOBAL_PARTITION "Use assumed partition" ON)
option(HYPRE_PRINT_ERRORS "Print HYPRE errors" OFF)
option(HYPRE_USING_OPENMP "Use OpenMP" ON)
option(HYPRE_USING_FEI "Use FEI" ON)
option(HYPRE_USING_CALIPER "Use Caliper" OFF)

Now, before compiling ex5f.f (which comes with the Hypre distribution) in Visual Studio, I noticed that it makes use of "mpif.h". Because I turned off MPI, I do not have mpif.h (please let me know if I am wrong here or anywhere else). As a work-around, I commented out the include 'mpif.h' line and other calls to the subroutines that begin with "MPI..".

When I try to build the solution with these edits, as you probably have guessed, I get error messages due to unresolved external symbols (e.g., _HYPRE_IJMATRIXCREATE, _HYPRE_IJMATRIXSETOBJECTTYPE, etc.).

Is there anything I can do to build this example file without MPI? Any help will be greatly appreciated.

Thank you.

CUDA Status and Documentation

Can someone tell me what the status of CUDA support is, and maybe what are common changes necessary to make existing Hypre using code compatible with the CUDA version of the library?

I've been tasked with incorporating the CUDA enabled version of Hypre into an application already using Hypre.
While there seems to be a small number of people using and discussing using CUDA-hypre in github issues, there isn't any documentation (as far as I can tell).

Thanks!

Wrong if-else syntax in main CMakeLists.txt

I tried to configure HYPRE 2.18.2 with CMake 3.15 and specified from command-line -DHYPRE_WITH_MPI=OFF but the option was ignored.

I guess the reason is that CMake options HYPRE_ENABLE_GLOBAL_PARTITION and HYPRE_WITH_MPI are wrongly handled in the main CMakeLists.txt. Since they are boolean, the if-block should have a else() statement, not elseif():

# 2.18.2
if(HYPRE_WITH_MPI)
  ...
elseif()
  # This is never reached
  ...
endif()

# Correct
if(HYPRE_WITH_MPI)
  ...
else()
  ...
endif()

Building with XL on an IBM AC922 system

I'm attempting to install Hypre on an IBM AC922 (i.e., Summit, Sierra, Lassen) and would like to use the IBM XL 16.1 compilers and Spectrum MPI 10.3. My first attempt at the install was through Spack. That build failed while trying to compile the openblas dependency; openblas does not support XL. My Spack 'spec' for the Hypre install is:

[email protected]%[email protected]~complex~debug~int64~internal-superlu~mixedint+mpi~openmp+shared~superlu-dist

The Spack package file for Hypre is here:
https://github.com/spack/spack/blob/v0.13.3/var/spack/repos/builtin/packages/hypre/package.py

Is there a way through Spack, or otherwise, to build Hypre with the XL compilers that produces an install similar to the one requested via Spack?

Building shared lib when CUDA is enabled

I was experimenting building HYPRE (tag v.2.17.0) with CUDA support from within PETSc configure @BarrySmith @balay . PETSc generates lots of -Wl,-rpath instructions needed for the creation of the so file.
nvcc, which is used by HYPRE when linking, does not tolerate this. Attached the error.

Building libHYPRE.so ...
nvcc -shared -gencode arch=compute_60,"code=sm_60" -o libHYPRE-2.17.0.so /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/IJ_mv/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/distributed_ls/Euclid/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/distributed_ls/ParaSails/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/distributed_ls/pilut/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/distributed_matrix/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/krylov/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/matrix_matrix/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/multivector/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/parcsr_ls/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/parcsr_mv/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/parcsr_block_mv/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/seq_mv/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/sstruct_ls/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/sstruct_mv/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/struct_ls/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/struct_mv/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/utilities/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/blas/.o /home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/lapack/*.o -Wl,-rpath,/opt/ecrc/intel/2018/compilers_and_libraries/linux/mkl/lib/intel64 -L/opt/ecrc/intel/2018/compilers_and_libraries/linux/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -liomp5 -ldl -lpthread -lstdc++ -ldl -Wl,-rpath,/opt/ecrc/intel/2018/compilers_and_libraries/linux/mpi/intel64/lib/debug_mt -L/opt/ecrc/intel/2018/compilers_and_libraries/linux/mpi/intel64/lib/debug_mt -Wl,-rpath,/opt/ecrc/intel/2018/compilers_and_libraries/linux/mpi/intel64/lib -L/opt/ecrc/intel/2018/compilers_and_libraries/linux/mpi/intel64/lib -lmpifort -lmpi -lmpigi -lrt -lpthread -Wl,-rpath,/opt/ecrc/cuda/10.0/lib64 -L/opt/ecrc/cuda/10.0/lib64 -Wl,-rpath,/opt/ecrc/cuda/10.0/lib -L/opt/ecrc/cuda/10.0/lib -Wl,-rpath,/opt/ecrc/valgrind/3.14.0-gcc-5.5.0/ub16/lib -L/opt/ecrc/valgrind/3.14.0-gcc-5.5.0/ub16/lib -Wl,-rpath,/opt/ecrc/valgrind/3.14.0-gcc-5.5.0/ub16/lib/valgrind -L/opt/ecrc/valgrind/3.14.0-gcc-5.5.0/ub16/lib/valgrind -Wl,-rpath,/opt/ecrc/cmake/3.11.1/ub16/lib -L/opt/ecrc/cmake/3.11.1/ub16/lib -Wl,-rpath,/opt/ecrc/intel/2018/compilers_and_libraries/linux/lib/intel64 -L/opt/ecrc/intel/2018/compilers_and_libraries/linux/lib/intel64 -Wl,-rpath,/opt/ecrc/intel/2018/compilers_and_libraries/linux/ipp/lib/intel64 -L/opt/ecrc/intel/2018/compilers_and_libraries/linux/ipp/lib/intel64 -Wl,-rpath,/opt/ecrc/intel/2018/compilers_and_libraries/linux/mkl/lib/intel64 -L/opt/ecrc/intel/2018/compilers_and_libraries/linux/mkl/lib/intel64 -Wl,-rpath,/opt/ecrc/intel/2018/compilers_and_libraries/linux/tbb/lib/intel64/gcc4.4 -L/opt/ecrc/intel/2018/compilers_and_libraries/linux/tbb/lib/intel64/gcc4.4 -Wl,-rpath,/opt/ecrc/intel/2018/compilers_and_libraries/linux/daal/lib/intel64_lin -L/opt/ecrc/intel/2018/compilers_and_libraries/linux/daal/lib/intel64_lin -Wl,-rpath,/opt/ecrc/intel/2018/compilers_and_libraries/linux/tbb/lib/intel64_lin/gcc4.4 -L/opt/ecrc/intel/2018/compilers_and_libraries/linux/tbb/lib/intel64_lin/gcc4.4 -Wl,-rpath,/opt/ecrc/intel/2018/compilers_and_libraries/linux/lib/intel64_lin -L/opt/ecrc/intel/2018/compilers_and_libraries/linux/lib/intel64_lin -Wl,-rpath,/opt/ecrc/intel-it/2018/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64_lin -L/opt/ecrc/intel-it/2018/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64_lin -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/5 -L/usr/lib/gcc/x86_64-linux-gnu/5 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/i386-linux-gnu -L/usr/lib/i386-linux-gnu -Wl,-rpath,/opt/ecrc/intel/2018/compilers_and_libraries/linux/mpi/intel64/lib/debug_mt -Wl,-rpath,/opt/ecrc/intel/2018/compilers_and_libraries/linux/mpi/intel64/lib -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib -lifport -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -lm -L/opt/ecrc/cuda/10.0//lib64 -lcusparse -lcudart -lcublas -lnvToolsExt -Xlinker=-soname,libHYPRE-2.17.0.so -Xlinker=-z,defs -ccbin=/opt/ecrc/intel/2018/compilers_and_libraries/linux/mpi/intel64/bin/mpiicpc -gencode arch=compute_60,"code=sm_60" -Xcompiler ""
nvcc fatal : Unknown option 'Wl,-rpath,/opt/ecrc/intel/2018/compilers_and_libraries/linux/mkl/lib/intel64'
Makefile:101: recipe for target 'libHYPRE.so' failed
make[1]: *** [libHYPRE.so] Error 1
make[1]: Leaving directory '/home/zampins/petsc/arch-gpu-double/externalpackages/git.hypre/src/lib'
Makefile:86: recipe for target 'all' failed
make: *** [all] Error 1

Accessing entries of a hypreCSR matrix

Dear Hypre team,

I have the following question.

Given a HypreCSR matrix A, is there an easy way to read a particular entry A(i,j), where i and j are global indices.
This is assuming that only the processor that "owns" index i will read the entry.

Thanks,

Socratis

Bug for large number of d.o.f./MPI task?

As I am running tests on Summit, I got a segfault running the 'struct' driver:

[g04n11:54194] *** Process received signal ***
[g04n11:54194] Signal: Segmentation fault (11)
[g04n11:54194] Signal code: Address not mapped (1)
[g04n11:54194] Failing at address: (nil)
[g04n11:54194] [ 0] [0x2000000504d8]
[g04n11:54194] [ 1] /ccs/home/jeanluc/GIT/hypre-space/hypre/src/hypre/lib/libHYPRE-2.16.0.so(+0x549e84)[0x200000639e84]
[g04n11:54194] [ 2] /ccs/home/jeanluc/GIT/hypre-space/hypre/src/hypre/lib/libHYPRE-2.16.0.so(+0x54f384)[0x20000063f384]
[g04n11:54194] [ 3] /ccs/home/jeanluc/GIT/hypre-space/hypre/src/hypre/lib/libHYPRE-2.16.0.so(hypre_PFMGSetup+0xab4)[0x2000005470c4]
[g04n11:54194] [ 4] /ccs/home/jeanluc/GIT/hypre-space/hypre/src/hypre/lib/libHYPRE-2.16.0.so(HYPRE_StructPFMGSetup+0x18)[0x200000500
7d4]
[g04n11:54194] [ 5] /ccs/home/jeanluc/hypre/bin/struct_gpu[0x1000b08c]
[g04n11:54194] [ 6] /lib64/libc.so.6(+0x25100)[0x20000b3b5100]
[g04n11:54194] [ 7] /lib64/libc.so.6(__libc_start_main+0xc4)[0x20000b3b52f4]

I am running the following command line:

jsrun -n 1 -g 1 -a 1 struct -n 512 512 512 -P 1 1 1 -c 2.0 3.0 4.0 -solver 1

that is, running 1 MPI task on a single GPU, for a mesh 512x512x512.
All my other (smaller) tests, using smaller meshes, or several MPI tasks, work.

Jean-Luc Fattebert

New hypre release 2.16.0

Hello @hypre-space/developers ,

We are in the process of doing a new hypre release, so please hold off on making changes to the master branch until it is done. It should be completed by tomorrow or Thu, but I will let you know.

Thanks!

-Rob

.

Access to the final residual norm

This is the remaining part of an issue from the old github project https://github.com/LLNL/hypre/issues/39

The various ParCSR Krylov solvers employ a combination of the relative residual norm and absolute residual norm for their convergence criterion. They currently provide a method for getting the final relative residual norm (e.g., HYPRE_ParCSRHybridGetFinalRelativeResidualNorm) but not the final (absolute) residual norm. The latter is also needed.

hypre_IJMatrixSetValuesParCSR bug?

Hi:

There might be a bug around line 1032 of IJMatrix_parcscr.c. If it's a bug the same bug exists in the release version also.

                if (not_found)
                 {
                    if (cnt_diag < diag_space)
                    {
                       diag_j[cnt_diag] = col_j;
                       diag_data[cnt_diag++] = values[indx];
                    }
                    else
                    {
                       hypre_error(HYPRE_ERROR_GENERIC);
                       if (print_level)
                       {
                          hypre_printf("Error in row %b ! Too many elements !\n",
                                       row);
                       }
                       /* return 1; */
                       return hypre_error_flag;
                    }
                 }

col_j is not initialized for this if condition. I believe line 1032 should be
diag_j[cnt_diag] = (HYPRE_Int)(cols[indx]-col_0);

Issue with configure

Hi
I'm encountering the following issue when I run configure in the src folder:

checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for mpxlc... no
checking for mpixlc... no
checking for mpiicc... mpiicc
checking for mpxlC... no
checking for mpixlcxx... no
checking for mpixlC... no
checking for mpiicpc... mpiicpc
checking for mpxlf... no
checking for mpixlf77... no
checking for mpiifort... mpiifort
checking whether make sets $(MAKE)... yes
checking for ranlib... ranlib
checking whether the C compiler works... no
configure: error: in /****/****/hypre/src': configure: error: C compiler cannot create executables See config.log' for more details

I'm wondering if all the mp compilers are needed or I have to comment some of them out?
I'm fairly new to complex codes so I can't really figure this out by myself for now and I'd really appreciate some help
Thank you

Missing MPI files when Installing Hypre on Windows with CMake for Visual Studio

Hello,

I am trying to install Hypre (ver. 2.11.2) on my Windows machine using CMake (ver. 3.14.0), but I am having some trouble due to missing mpi.h files. I was wondering if I could receive any help resolving this issue. I am new to Hypre and I am just trying to get any example code to run.

When I build the binaries (configured for Visual Studio 2011; x64) on CMake following the installation instruction, I receive the following messages:

---

The C compiler identification is MSVC 17.0.50727.1
The CXX compiler identification is MSVC 17.0.50727.1
Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 11.0/VC/bin/x86_amd64/cl.exe
Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 11.0/VC/bin/x86_amd64/cl.exe -- works
Detecting C compiler ABI info
Detecting C compiler ABI info - done
Detecting C compile features
Detecting C compile features - done
Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio 11.0/VC/bin/x86_amd64/cl.exe
Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio 11.0/VC/bin/x86_amd64/cl.exe -- works
Detecting CXX compiler ABI info
Detecting CXX compiler ABI info - done
Detecting CXX compile features
Detecting CXX compile features - done
Could NOT find MPI_C (missing: MPI_C_LIB_NAMES MPI_C_HEADER_DIR MPI_C_WORKS)
Could NOT find MPI_CXX (missing: MPI_CXX_LIB_NAMES MPI_CXX_HEADER_DIR MPI_CXX_WORKS)
Could NOT find MPI (missing: MPI_C_FOUND MPI_CXX_FOUND)
Configuring done

Then, as instructed, I generate the build files.

When I open up Visual Studio and try to build the "ALL_BUILD" target, I run into 395 errors, all of which say the following line:
"error C1083: Cannot open include file: 'mpi.h': No such file or directory"
The associated File is "HYPRE_utilities.h" and Line = 25.

Based on the error messages I get, both on CMake and Visual Studio, I think this issue has to do with MPI. Is there any way to bypass MPI? My ultimate intention is to utilize OpenMP-enabled Linear-Algebraic System Interface in Fortran, and I do NOT need MPI for my applications.

Thank you.

problem build hypre

Think I need some help, can not buildHypre anymore, see error message below.
bjorn@ray23 79) make
Making fei-hypre ...
make[1]: Entering directory '/usr/WS2/bjorn/codes/hypres/hypre.ray2/src/FEI_mv/fei-hypre'
make[1]: *** No rule to make target '/g/g91/bjorn/workspace/codes/hypres/hypre.ray2/src/FEI_mv/fei-base/fei_Data.hpp', needed by 'HYPRE_LSI_Dsuperlu.o'. Stop.
make[1]: Leaving directory '/usr/WS2/bjorn/codes/hypres/hypre.ray2/src/FEI_mv/fei-hypre'
make: *** [Makefile:9: all] Error 1

boomeramg crash with openmp

Hi:

I'm getting the following crash with hypre when I turn on OpenMP. I'm running hypre 2.18.2 compiled with intel, MPI is mpich, and OMP_NUM_THREADS=16 and I'm running on KNL's

hypre is configured with"
./configure --enable-mixedint --with-openmp

======= Backtrace: =========
(hypre_BoomerAMGSetup+0x26a2)[0x3982ef2]
double free or corruption (fasttop): 0x00000000056a6420 ***
free(): invalid next size (normal): 0x0000000006ad4f60 ***

gdb backtrace gives:

(gdb) bt
#0 0x0000000000000000 in ?? ()
#1
#2 0x00002aaacd6521c2 in int_free () from /lib64/libc.so.6
#3 0x0000000003a6533c in hypre_BoomerAMGCreateSHost (A=, strength_threshold=<error reading variable: Unable to access DWARF register number 67>, max_row_sum=0, num_functions=0, dof_func=0x0, S_ptr=) at par_strength.c:522
#4 hypre_BoomerAMGCreateS (A=0x2aaacd977640 <main_arena>, strength_threshold=0, max_row_sum=0, num_functions=0, dof_func=0x2c1030, S_ptr=0x0) at par_strength.c:554
#5 0x0000000003982ef2 in hypre_BoomerAMGSetup (amg_vdata=0x2aaacd977640 <main_arena>, A=0x0, f=0x2c1030, u=0x0) at par_amg_setup.c:988
#6 0x0000000003b66852 in hypre_PCGSetup (pcg_vdata=0x2aaacd977640 <main_arena>, A=0x0, b=0x2c1030, x=0x0) at pcg.c:219
#7 0x0000000003977a6c in hypre_parcsrpcgsetup (solver=0x2aaacd977640 <main_arena>, A=0x0, b=0x2c1030, x=0x0, ierr=0x792fa00) at F90_HYPRE_parcsr_pcg.c:64

Multivectors

By default, the Krylov space in GMRES is stored as an array of ParVectors. This is very inefficient esp if the orthogonalization is performed "in bulk" and on the GPU (either one-sycnh modified Gram Schmidt or Classical Gram Schmidt with reorthogonalization), because data access is very scattered.
I implemented this instead storing the space as one long multivector in which the data is stored columnwise as one long contiguous chunk of memory. I was wandering if this can be changed in the main hypre branch.

Thanks

Compiling

New hypre release 2.17.0 - license change to MIT/Apache-2.0

Hello @hypre-space/developers,

We are in the process of doing a new hypre release, so please hold off on making changes to the master branch until it is done. I hope to have it completed by Wednesday, but I will let you know.

This release will incorporate a license change to dual MIT/Apache-2.0. Please take a look at the new source in the 'relicense' branch at https://github.com/hypre-space/hypre/tree/relicense.

Thanks!

-Rob

Problem while installation

I'm new user of Hypre. I downloaded a zip file then used unzip command to unzip the file.
Then I went to the src folder to install it.
I typed ./configure
then I typed make install
that what I got,
Makefile:16: config/Makefile.config: No such file or directory
make: *** No rule to make target 'config/Makefile.config'. Stop.

config.log file is attached

Could you please help me to figure out this problem?
Thank in advance.

config.log

Build peculiarities, MPICC and AR flags

When building I have a few questions/requests:

The configure tool does not get the MPICC variable even though --with-mpi is defined. This would make it easier to use since you don't need to do CC=$MPICC ./configure

Secondly, if AR is defined it is used as-is which is not typical (at least for the makefile world). Typically AR is only the ar command without flags. The flags should be given in ARFLAGS. Perhaps this could be incorporated?

Just ideas! :)

CUDA and ESSL

On Lassen, configure with

HYPRE_CUDA_SM=70 ./configure --with-cuda --enable-unified-memory  --with-lapack-lib="-L/usr/tcetmp/packages/essl/essl-6.2/lib64 -R/usr/tcetmp/packages/essl/ess-6.2/lib64 -lessl -llapackforessl"

Error:

nvcc -O2 -ccbin=mpixlC -gencode arch=compute_70,"code=sm_70" -expt-extended-lambda -dc -std=c++11 --x cu -Xcompiler "-O2 " -DHAVE_CONFIG_H -I../.. -I. -I./../.. -I./../../blas -I./../../lapack -I./../../utilities -I./../../distributed_matrix -I/usr/tce/packages/cuda/cuda-10.1.243/include   -c ParaSails.c 

ParaSails.c(1317): error: initial value of reference to non-const must be an lvalue

Scalablity for 1D tri-diagonal solver

Hi All,

I'm trying to solve a 1D Poisson-like equation which corresponds to a tridiagonal linear system. Now I'm using the Struct interface with cyclic reduction solver but found it has poor parallel scalability. When I run it from 1 to 16 processors, only no more than 3 times speedup can be reached. The problem size is 8192 grid points so it corresponds to an 8192 x 8192 sparse matrix. Is the poor parallel scalability the nature of cyclic reduction method itself, or something I did wrong? Is there any suggestion on efficiently solving such problem parallelly?

Thanks,

Fei

SONAME is wrong

The version 2.16.0 installs only these files inti lib/:

lib/libHYPRE.so 
lib/libHYPRE.so.2
lib/libHYPRE.so.2.16

but this library has this SONAME:

$ elfdump -a libHYPRE.so.2.16
...
...
entry: 9
        d_tag: DT_SONAME
        d_val: libHYPRE-2.16.0.so
...

It is customary to have SONAME in the form libxx.so.N based on the installed file name.

out of bounds write: hypre_*SetMaxIter()

In hypre_*Create() max_iter is set to a default of 1000 for the Krylov solvers (some subset that includes PCG, GMRES, BICGSTAB) and the norm and rel_norm arrays are allocated to a size of (max_iter+1) if logging or print_level >0.

If the user calls hypre_*SetMaxIter() prior to a solve with max_iter > 1000 and the solver happens to take more than 1000 iterations an out of bounds memory write happens for the arrays norm and rel_norm if logging or print_level > 0.

The norm and rel_norm arrays should probably be reallocated if a call to hypre_*SetMaxIter() happens.

Convergence tolerance defaults

Hi @hypre-space/developers,

It looks like we have some inconsistencies with our convergence tolerance defaults. In most cases, the default is 1.0e-6, but there are a couple that are 1.0e-7 (par_amg.c, par_mgr.c). I think we should make them all 1.0e-6 and ensure that the header file comments also match. Any concerns about making these changes?

Thanks!

-Rob

"Mixed" Filename Mangling Issue

Hi,

I have been trying to compile ex5f.f with the Hypre library (MPI option ENABLED), and I suspect that my Fortran compiler is running into a situation where a single mangling macro cannot take care of all subroutine name formats.

When I try to compile ex5f.f with the Hypre library built with the default options, I get the following error messages:

---

Error 1 error LNK2019: unresolved external symbol MPI_INIT referenced in function MAIN__ ex5f.obj
Error 2 error LNK2019: unresolved external symbol MPI_COMM_RANK referenced in function MAIN__ ex5f.obj
Error 3 error LNK2019: unresolved external symbol MPI_COMM_SIZE referenced in function MAIN__ ex5f.obj
Error 4 error LNK2019: unresolved external symbol HYPRE_IJMATRIXCREATE referenced in function MAIN__ ex5f.obj
Error 5 error LNK2019: unresolved external symbol HYPRE_IJMATRIXSETOBJECTTYPE referenced in function MAIN__ ex5f.obj
Error 6 error LNK2019: unresolved external symbol HYPRE_IJMATRIXINITIALIZE referenced in function MAIN__ ex5f.obj
Error 7 error LNK2019: unresolved external symbol HYPRE_IJMATRIXSETVALUES referenced in function MAIN__ ex5f.obj
Error 8 error LNK2019: unresolved external symbol HYPRE_IJMATRIXASSEMBLE referenced in function MAIN__ ex5f.obj
Error 9 error LNK2019: unresolved external symbol HYPRE_IJMATRIXGETOBJECT referenced in function MAIN__ ex5f.obj
Error 10 error LNK2019: unresolved external symbol HYPRE_IJVECTORCREATE referenced in function MAIN__ ex5f.obj
Error 11 error LNK2019: unresolved external symbol HYPRE_IJVECTORSETOBJECTTYPE referenced in function MAIN__ ex5f.obj
Error 12 error LNK2019: unresolved external symbol HYPRE_IJVECTORINITIALIZE referenced in function MAIN__ ex5f.obj
Error 13 error LNK2019: unresolved external symbol HYPRE_IJVECTORSETVALUES referenced in function MAIN__ ex5f.obj
Error 14 error LNK2019: unresolved external symbol HYPRE_IJVECTORASSEMBLE referenced in function MAIN__ ex5f.obj
Error 15 error LNK2019: unresolved external symbol HYPRE_IJVECTORGETOBJECT referenced in function MAIN__ ex5f.obj
Error 16 error LNK2019: unresolved external symbol HYPRE_BOOMERAMGCREATE referenced in function MAIN__ ex5f.obj
Error 17 error LNK2019: unresolved external symbol HYPRE_BOOMERAMGSETPRINTLEVEL referenced in function MAIN__ ex5f.obj
Error 18 error LNK2019: unresolved external symbol HYPRE_BOOMERAMGSETOLDDEFAULT referenced in function MAIN__ ex5f.obj
Error 19 error LNK2019: unresolved external symbol HYPRE_BOOMERAMGSETRELAXTYPE referenced in function MAIN__ ex5f.obj
Error 20 error LNK2019: unresolved external symbol HYPRE_BOOMERAMGSETRELAXORDER referenced in function MAIN__ ex5f.obj
Error 21 error LNK2019: unresolved external symbol HYPRE_BOOMERAMGSETNUMSWEEPS referenced in function MAIN__ ex5f.obj
Error 22 error LNK2019: unresolved external symbol HYPRE_BOOMERAMGSETMAXLEVELS referenced in function MAIN__ ex5f.obj
Error 23 error LNK2019: unresolved external symbol HYPRE_BOOMERAMGSETTOL referenced in function MAIN__ ex5f.obj
Error 24 error LNK2019: unresolved external symbol HYPRE_BOOMERAMGSETUP referenced in function MAIN__ ex5f.obj
Error 25 error LNK2019: unresolved external symbol HYPRE_BOOMERAMGSOLVE referenced in function MAIN__ ex5f.obj
Error 26 error LNK2019: unresolved external symbol HYPRE_BOOMERAMGGETNUMITERATIONS referenced in function MAIN__ ex5f.obj
Error 27 error LNK2019: unresolved external symbol HYPRE_BOOMERAMGGETFINALRELTVRES referenced in function MAIN__ ex5f.obj
Error 28 error LNK2019: unresolved external symbol HYPRE_BOOMERAMGDESTROY referenced in function MAIN__ ex5f.obj
Error 29 error LNK2019: unresolved external symbol HYPRE_IJMATRIXDESTROY referenced in function MAIN__ ex5f.obj
Error 30 error LNK2019: unresolved external symbol HYPRE_IJVECTORDESTROY referenced in function MAIN__ ex5f.obj
Error 31 error LNK2019: unresolved external symbol MPI_FINALIZE referenced in function MAIN__ ex5f.obj
Error 32 error LNK2001: unresolved external symbol MPI_DUP_FN ex5f.obj
Error 33 error LNK2001: unresolved external symbol MPI_NULL_DELETE_FN ex5f.obj
Error 34 error LNK2001: unresolved external symbol MPI_NULL_COPY_FN ex5f.obj
Error 35 error LNK2001: unresolved external symbol MPI_COMM_DUP_FN ex5f.obj
Error 36 error LNK2001: unresolved external symbol MPI_COMM_NULL_DELETE_FN ex5f.obj
Error 37 error LNK2001: unresolved external symbol MPI_COMM_NULL_COPY_FN ex5f.obj
Error 38 error LNK2001: unresolved external symbol MPI_WIN_DUP_FN ex5f.obj
Error 39 error LNK2001: unresolved external symbol MPI_WIN_NULL_DELETE_FN ex5f.obj
Error 40 error LNK2001: unresolved external symbol MPI_WIN_NULL_COPY_FN ex5f.obj
Error 41 error LNK2001: unresolved external symbol MPI_TYPE_DUP_FN ex5f.obj
Error 42 error LNK2001: unresolved external symbol MPI_TYPE_NULL_DELETE_FN ex5f.obj
Error 43 error LNK2001: unresolved external symbol MPI_TYPE_NULL_COPY_FN ex5f.obj
Error 44 error LNK2001: unresolved external symbol MPI_CONVERSION_FN_NULL ex5f.obj
Error 45 fatal error LNK1120: 44 unresolved externals x64\Release\Test_MPI.exe

---

Based on my previous correspondence in this forum (thank you for the help again), I modified the fortran.h file to capitalize the subroutine names by inserting

"#define HYPRE_FMANGLE 4"

right before

/*-------------------------------------------------------
Define hypre_F90_NAME and hypre_F90_IFACE
-------------------------------------------------------/

However, when I try to compile ex5f.f again after re-building the Hypre library, I get the following errors:

---

Error 1 error LNK2019: unresolved external symbol MPI_INIT referenced in function MAIN__ ex5f.obj
Error 2 error LNK2019: unresolved external symbol MPI_COMM_RANK referenced in function MAIN__ ex5f.obj
Error 3 error LNK2019: unresolved external symbol MPI_COMM_SIZE referenced in function MAIN__ ex5f.obj
Error 4 error LNK2019: unresolved external symbol MPI_FINALIZE referenced in function MAIN__ ex5f.obj
Error 5 error LNK2001: unresolved external symbol MPI_DUP_FN ex5f.obj
Error 6 error LNK2001: unresolved external symbol MPI_NULL_DELETE_FN ex5f.obj
Error 7 error LNK2001: unresolved external symbol MPI_NULL_COPY_FN ex5f.obj
Error 8 error LNK2001: unresolved external symbol MPI_COMM_DUP_FN ex5f.obj
Error 9 error LNK2001: unresolved external symbol MPI_COMM_NULL_DELETE_FN ex5f.obj
Error 10 error LNK2001: unresolved external symbol MPI_COMM_NULL_COPY_FN ex5f.obj
Error 11 error LNK2001: unresolved external symbol MPI_WIN_DUP_FN ex5f.obj
Error 12 error LNK2001: unresolved external symbol MPI_WIN_NULL_DELETE_FN ex5f.obj
Error 13 error LNK2001: unresolved external symbol MPI_WIN_NULL_COPY_FN ex5f.obj
Error 14 error LNK2001: unresolved external symbol MPI_TYPE_DUP_FN ex5f.obj
Error 15 error LNK2001: unresolved external symbol MPI_TYPE_NULL_DELETE_FN ex5f.obj
Error 16 error LNK2001: unresolved external symbol MPI_TYPE_NULL_COPY_FN ex5f.obj
Error 17 error LNK2001: unresolved external symbol MPI_CONVERSION_FN_NULL ex5f.obj
Error 18 error LNK2019: unresolved external symbol MPI_Send referenced in function hypre_MPI_Send HYPRE.lib(mpistubs.obj)
Error 19 error LNK2019: unresolved external symbol MPI_Recv referenced in function hypre_MPI_Recv HYPRE.lib(mpistubs.obj)
Error 20 error LNK2019: unresolved external symbol MPI_Get_count referenced in function hypre_MPI_Get_count HYPRE.lib(mpistubs.obj)
Error 21 error LNK2019: unresolved external symbol MPI_Isend referenced in function hypre_MPI_Isend HYPRE.lib(mpistubs.obj)
Error 22 error LNK2019: unresolved external symbol MPI_Irsend referenced in function hypre_MPI_Irsend HYPRE.lib(mpistubs.obj)
Error 23 error LNK2019: unresolved external symbol MPI_Irecv referenced in function hypre_MPI_Irecv HYPRE.lib(mpistubs.obj)
Error 24 error LNK2019: unresolved external symbol MPI_Wait referenced in function hypre_MPI_Wait HYPRE.lib(mpistubs.obj)
Error 25 error LNK2019: unresolved external symbol MPI_Test referenced in function hypre_MPI_Test HYPRE.lib(mpistubs.obj)
Error 26 error LNK2019: unresolved external symbol MPI_Request_free referenced in function hypre_MPI_Request_free HYPRE.lib(mpistubs.obj)
Error 27 error LNK2019: unresolved external symbol MPI_Waitany referenced in function hypre_MPI_Waitany HYPRE.lib(mpistubs.obj)
Error 28 error LNK2019: unresolved external symbol MPI_Waitall referenced in function hypre_MPI_Waitall HYPRE.lib(mpistubs.obj)
Error 29 error LNK2019: unresolved external symbol MPI_Testall referenced in function hypre_MPI_Testall HYPRE.lib(mpistubs.obj)
Error 30 error LNK2019: unresolved external symbol MPI_Iprobe referenced in function hypre_MPI_Iprobe HYPRE.lib(mpistubs.obj)
Error 31 error LNK2019: unresolved external symbol MPI_Probe referenced in function hypre_MPI_Probe HYPRE.lib(mpistubs.obj)
Error 32 error LNK2019: unresolved external symbol MPI_Send_init referenced in function hypre_MPI_Send_init HYPRE.lib(mpistubs.obj)
Error 33 error LNK2019: unresolved external symbol MPI_Recv_init referenced in function hypre_MPI_Recv_init HYPRE.lib(mpistubs.obj)
Error 34 error LNK2019: unresolved external symbol MPI_Startall referenced in function hypre_MPI_Startall HYPRE.lib(mpistubs.obj)
Error 35 error LNK2019: unresolved external symbol MPI_Type_contiguous referenced in function hypre_MPI_Type_contiguous HYPRE.lib(mpistubs.obj)
Error 36 error LNK2019: unresolved external symbol MPI_Type_vector referenced in function hypre_MPI_Type_vector HYPRE.lib(mpistubs.obj)
Error 37 error LNK2019: unresolved external symbol MPI_Type_create_hvector referenced in function hypre_MPI_Type_hvector HYPRE.lib(mpistubs.obj)
Error 38 error LNK2019: unresolved external symbol MPI_Type_create_struct referenced in function hypre_MPI_Type_struct HYPRE.lib(mpistubs.obj)
Error 39 error LNK2019: unresolved external symbol MPI_Get_address referenced in function hypre_MPI_Address HYPRE.lib(mpistubs.obj)
Error 40 error LNK2019: unresolved external symbol MPI_Type_commit referenced in function hypre_MPI_Type_commit HYPRE.lib(mpistubs.obj)
Error 41 error LNK2019: unresolved external symbol MPI_Type_free referenced in function hypre_MPI_Type_free HYPRE.lib(mpistubs.obj)
Error 42 error LNK2019: unresolved external symbol MPI_Barrier referenced in function hypre_MPI_Barrier HYPRE.lib(mpistubs.obj)
Error 43 error LNK2019: unresolved external symbol MPI_Bcast referenced in function hypre_MPI_Bcast HYPRE.lib(mpistubs.obj)
Error 44 error LNK2019: unresolved external symbol MPI_Gather referenced in function hypre_MPI_Gather HYPRE.lib(mpistubs.obj)
Error 45 error LNK2019: unresolved external symbol MPI_Gatherv referenced in function hypre_MPI_Gatherv HYPRE.lib(mpistubs.obj)
Error 46 error LNK2019: unresolved external symbol MPI_Scatter referenced in function hypre_MPI_Scatter HYPRE.lib(mpistubs.obj)
Error 47 error LNK2019: unresolved external symbol MPI_Scatterv referenced in function hypre_MPI_Scatterv HYPRE.lib(mpistubs.obj)
Error 48 error LNK2019: unresolved external symbol MPI_Allgather referenced in function hypre_MPI_Allgather HYPRE.lib(mpistubs.obj)
Error 49 error LNK2019: unresolved external symbol MPI_Allgatherv referenced in function hypre_MPI_Allgatherv HYPRE.lib(mpistubs.obj)
Error 50 error LNK2019: unresolved external symbol MPI_Alltoall referenced in function hypre_MPI_Alltoall HYPRE.lib(mpistubs.obj)
Error 51 error LNK2019: unresolved external symbol MPI_Op_create referenced in function hypre_MPI_Op_create HYPRE.lib(mpistubs.obj)
Error 52 error LNK2019: unresolved external symbol MPI_Op_free referenced in function hypre_MPI_Op_free HYPRE.lib(mpistubs.obj)
Error 53 error LNK2019: unresolved external symbol MPI_Reduce referenced in function hypre_MPI_Reduce HYPRE.lib(mpistubs.obj)
Error 54 error LNK2019: unresolved external symbol MPI_Allreduce referenced in function hypre_MPI_Allreduce HYPRE.lib(mpistubs.obj)
Error 55 error LNK2019: unresolved external symbol MPI_Scan referenced in function hypre_MPI_Scan HYPRE.lib(mpistubs.obj)
Error 56 error LNK2019: unresolved external symbol MPI_Comm_group referenced in function hypre_MPI_Comm_group HYPRE.lib(mpistubs.obj)
Error 57 error LNK2019: unresolved external symbol MPI_Group_incl referenced in function hypre_MPI_Group_incl HYPRE.lib(mpistubs.obj)
Error 58 error LNK2019: unresolved external symbol MPI_Group_free referenced in function hypre_MPI_Group_free HYPRE.lib(mpistubs.obj)
Error 59 error LNK2019: unresolved external symbol MPI_Comm_size referenced in function hypre_MPI_Allgatherv HYPRE.lib(mpistubs.obj)
Error 60 error LNK2019: unresolved external symbol MPI_Comm_rank referenced in function hypre_MPI_Comm_rank HYPRE.lib(mpistubs.obj)
Error 61 error LNK2019: unresolved external symbol MPI_Comm_dup referenced in function hypre_MPI_Comm_dup HYPRE.lib(mpistubs.obj)
Error 62 error LNK2019: unresolved external symbol MPI_Comm_create referenced in function hypre_MPI_Comm_create HYPRE.lib(mpistubs.obj)
Error 63 error LNK2019: unresolved external symbol MPI_Comm_split referenced in function hypre_MPI_Comm_split HYPRE.lib(mpistubs.obj)
Error 64 error LNK2019: unresolved external symbol MPI_Comm_free referenced in function hypre_MPI_Comm_free HYPRE.lib(mpistubs.obj)
Error 65 error LNK2019: unresolved external symbol MPI_Wtime referenced in function hypre_MPI_Wtime HYPRE.lib(mpistubs.obj)
Error 66 error LNK2019: unresolved external symbol MPI_Wtick referenced in function hypre_MPI_Wtick HYPRE.lib(mpistubs.obj)
Error 67 error LNK2019: unresolved external symbol MPI_Init referenced in function hypre_MPI_Init HYPRE.lib(mpistubs.obj)
Error 68 error LNK2019: unresolved external symbol MPI_Finalize referenced in function hypre_MPI_Finalize HYPRE.lib(mpistubs.obj)
Error 69 error LNK2019: unresolved external symbol MPI_Abort referenced in function hypre_MPI_Abort HYPRE.lib(mpistubs.obj)
Error 70 fatal error LNK1120: 69 unresolved externals x64\Release\Test_MPI.exe

---

As can be seen, the compiler now has trouble finding subroutines whose names are made up of both lower case and capital letters.

Would there be anything I can do to resolve this issue? For your info, the MPI is provided by Microsoft MPI and I am using Visual Studio.

Thank you.

hypre_CSRMatrixToParCSRMatrix(...) some problem

Fatal error in PMPI_Type_create_struct: invalid argument ,error stack:
PMPI_Type_create_struct(185):MPI_Type_create_struct(count=3 ,array_of_blocklengths=0x10c94f0,array_of_displacements=0x7ffc88c1f730,array_of_types=0x7ffc88c1f700,newtype=0x10a6794) failed

  PMPI_Type_create_struct(148): Invalid value for blocken ,must be non-negative but is -449

New hypre release 2.18.1

Hello @hypre-space/developers,

We are in the process of doing a new hypre patch release, so please hold off on making changes to the master branch until it is done. We still plan on doing a minor release soon with the new features that didn't make it into 2.18.

Thanks!

-Rob

Number of OpenMP Threads

Hello,

Is there any way to specify the number of OpenMP Threads for the Hypre library? I know one can use set omp_num_threads= in the command window but I would like to hard-wire the number of threads. FYI, my codes are in Fortran and I am using Windows.

Thank you.

PETSc+hypre examples broken with 2.16.0 (works with 2.15)

@ulrikeyang,

PETSc no longer works with latest hypre. (2.16.0) We get a bunch of valgrind errors in hypre.

not ok ksp_ksp_tutorials-ex55_hypre
#   ==15518== Invalid read of size 4
#   ==15518==    at 0x62F6D1A: hypre_IJMatrixAssembleParCSR (IJMatrix_parcsr.c:2962)
#   ==15518==    by 0x62F1A04: HYPRE_IJMatrixAssemble (HYPRE_IJMatrix.c:721)
#   ==15518==    by 0x551EEBC: MatAssemblyEnd_HYPRE (mhypre.c:1040)
#   ==15518==    by 0x530BB6F: MatAssemblyEnd (matrix.c:5440)
#   ==15518==    by 0x5526EC3: MatConvert_AIJ_HYPRE (mhypre.c:355)
#   ==15518==    by 0x530EB3E: MatConvert (matrix.c:4275)
#   ==15518==    by 0x5E3E8AC: PCSetUp_HYPRE (hypre.c:171)
#   ==15518==    by 0x5E91514: PCSetUp (precon.c:932)
#   ==15518==    by 0x5FA992F: KSPSetUp (itfunc.c:391)
#   ==15518==    by 0x40DD03: main (ex55.c:224)
#   ==15518==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
etc..

The trigger is changes in 84d5010

This commit appears to have multiple changes rolled in - so we are unable to narrow down.

(gdb) p *offd
$1 = {i = 0x1e37600, j = 0x1e411d0, big_j = 0x0, num_rows = 28, num_cols = 0, num_nonzeros = 7, i_short = 0x0, j_short = 0x0, owns_data = 1, 
  data = 0x1d3afc0, rownnz = 0x0, num_rownnz = 28}

This new version now has j & big_j - but big_j is not allocate [and NULL] - hence the above crash.

Could you take a look at this?

To reproduce - one can build PETSc with hypre 2.16.0

[with pre-installed system-blas - and MPI in PATH]

export PETSC_DIR=$PWD
./configure --download-hypre=https://github.com/hypre-space/hypre/archive/v2.16.0.tar.gz
make
cd src/ksp/ksp/examples/tutorials
make ex2
mpiexec -n 2 ./ex2 -pc_type hypre

I attempted to workaround by revering to 2.15 usage with j instead of big_j with:

diff --git a/src/IJ_mv/IJMatrix_parcsr.c b/src/IJ_mv/IJMatrix_parcsr.c
index 84f9fe0b7..7a51c2f50 100644
--- a/src/IJ_mv/IJMatrix_parcsr.c
+++ b/src/IJ_mv/IJMatrix_parcsr.c
@@ -2959,7 +2959,7 @@ hypre_IJMatrixAssembleParCSR(hypre_IJMatrix *matrix)
          tmp_j = hypre_CTAlloc(HYPRE_BigInt,  nnz_offd, HYPRE_MEMORY_HOST);
          for (i=0; i < nnz_offd; i++)
          {
-            tmp_j[i] = big_offd_j[i];
+            tmp_j[i] = offd_j[i];
          }
          hypre_BigQsort0(tmp_j,0,nnz_offd-1);
          num_cols_offd = 1;
@@ -2980,7 +2980,7 @@ hypre_IJMatrixAssembleParCSR(hypre_IJMatrix *matrix)
 #endif
          for (i=0; i < nnz_offd; i++)
          {
-            offd_j[i]=hypre_BigBinarySearch(col_map_offd,big_offd_j[i],num_cols_offd);
+            offd_j[i]=hypre_BigBinarySearch(col_map_offd,offd_j[i],num_cols_offd);
          }
 
          if (base)

But how the crash is elsewhere

==24433== Invalid read of size 4
==24433==    at 0x5BF27E8: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:1006)
==24433==    by 0x5BF4704: hypre_BoomerAMGCoarsenFalgout (par_coarsen.c:1924)
==24433==    by 0x5BE62FA: hypre_BoomerAMGSetup (par_amg_setup.c:1015)
==24433==    by 0x5BDD83C: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:52)
==24433==    by 0x5731855: PCSetUp_HYPRE (hypre.c:326)
==24433==    by 0x5784BEC: PCSetUp (precon.c:932)
==24433==    by 0x5883B22: KSPSetUp (itfunc.c:391)
==24433==    by 0x588477E: KSPSolve (itfunc.c:725)
==24433==    by 0x401D51: main (ex2.c:172)
==24433==  Address 0x8026358 is 4 bytes after a block of size 116 alloc'd
==24433==    at 0x483AB1A: calloc (vg_replace_malloc.c:762)
==24433==    by 0x5CD5E36: hypre_HostMalloc (hypre_memory.c:156)
==24433==    by 0x5CD5E36: hypre_MAlloc_core (hypre_memory.c:270)
==24433==    by 0x5CD5E36: hypre_CAlloc (hypre_memory.c:307)
==24433==    by 0x5BF2771: hypre_BoomerAMGCoarsenRuge (par_coarsen.c:992)
==24433==    by 0x5BF4704: hypre_BoomerAMGCoarsenFalgout (par_coarsen.c:1924)
==24433==    by 0x5BE62FA: hypre_BoomerAMGSetup (par_amg_setup.c:1015)
==24433==    by 0x5BDD83C: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:52)
==24433==    by 0x5731855: PCSetUp_HYPRE (hypre.c:326)
==24433==    by 0x5784BEC: PCSetUp (precon.c:932)
==24433==    by 0x5883B22: KSPSetUp (itfunc.c:391)
==24433==    by 0x588477E: KSPSolve (itfunc.c:725)
==24433==    by 0x401D51: main (ex2.c:172)
etc..

Question with hypre memory use

Dear hypre developers,

I am a using hypre for preconditioning (AMG) of homogenization problem. The problem I find is that the memory requirement for AMG is high so the system would crash because it cannot allocate enough memory. I would assume the total memory is scalable for hypre. I am just wondering whether the local memory would differ too much between different processors? Another question is that is there any way to reduce the memory requirement by change the parameters in preconditioner? The parameter I use is as followed (default setting for boomeramg):

    PETScOptions::set("ksp_view");
    PETScOptions::set("ksp_monitor_true_residual");
    PETScOptions::set("ksp_pc_side", "right");
    PETScOptions::set("pc_type", "hypre");
    PETScOptions::set("pc_hypre_type", "boomeramg");

Thank you for your help all the time. Hope you reply soon.

Yours sincerely,
Qiming

CUDA exposed in interface

(Sorry for posting previously to the wrong github. Here is an update using the latest
version of hypre downloaded today from https://github.com/hypre-space/hypre)

I am using the hypre struct PFMG solver built for GPU on the Volta V 100.
I built hypre with the options:
./configure --with-cuda --enable-unified-memory --enable-shared
--prefix=$HOME/hypre/summit_xl_gpu CUDACXX=mpiCC HYPRE_CUDA_SM=70

While it is working for me, I had to work around an interface issue:
To define various hypre structure in my C++ code, such as HYPRE_StructVector that
I can pass to initialization functions such as HYPRE_StructVectorCreate(), I have to include
the file HYPRE_struct_ls.h.

Now if I use a C++ compiler (xlC here), I end up with an error like:

In file included from /ccs/home/jeanluc/GIT/PFiSM/source/PfmFACSolver.C:2:
In file included from /ccs/home/jeanluc/GIT/PFiSM/source/CellPoissonHypreSolver.h:25:
In file included from /ccs/home/jeanluc/hypre/summit_xl_gpu/include/HYPRE_struct_ls.h:1105:
In file included from /ccs/home/jeanluc/hypre/summit_xl_gpu/include/interpreter.h:11:
/ccs/home/jeanluc/hypre/summit_xl_gpu/include/_hypre_utilities.h:1260:18: error: use of undeclared
identifier 'blockDim'
return (blockDim.x);

In other words, even though I have no CUDA code in my own code, I end up having to
compile multiple C++ files with nvcc. I was wondering why CUDA code/includes in hypre are
exposed in the interface and lead to that issue? I noticed a comment in HYPRE_struct_ls.h
"These includes shouldn't be here. (RDF)", so I am guessing that that issue is already
on the developers radar? Or maybe I am missing a configure option?

Thanks,
Jean-Luc Fattebert
ORNL

BoomerAMG: `Out of memory` error on corner case

Dear hypre developers!

The deal.II community is currently making an effort to implement parallel hp-adaptive finite element methods. We were preparing a particular benchmark to verify our parallel implementation, which is based on one of the deal.II tutorial step step-27 in which sequential hp-adaptive FEM has been introduced.

However, a very certain corner configuration of our scenario yields to some errors triggered by hypre, or more specifically, by BoomerAMG. We use a backend to PETSc as our parallel linear algebra library, which provides BoomerAMG as our algebraic multigrid preconditioner.

We could only reproduce this issue with exactly 20 MPI processes assigned to a total of 49152 cells with 198144 degrees of freedom. The error occurs in the very first cycle of consecutive refinements, in which all cells were assigned a Lagrangian finite element with quadratic base functions.

Using a different preconditioner or none at all does not trigger the error. Neither does a different amount of MPI processes, nor a different number of global mesh refinements (which yields this particular amount of cells). However, I haven't tried different finite elements yet. With parallel debugging, I encountered the following issues:

---

Five processes reported the following (with petsc-3.9.1):

---

Four processes reported the following (with petsc-3.9.1):

---

Additionally, when using versions of PETSc prior to 3.8.4, error messages likely to the following appear.

bash-4.2$ mpirun -np 20 step-27new
Running with PETSc on 20 MPI rank(s)...
Cycle 0:
   Number of active cells      : 49152
   Number of degrees of freedom: 198144
   Number of constraints       : 3272
Out of memory trying to allocate 1451474080 bytes
Out of memory trying to allocate 1451474080 bytes
Out of memory trying to allocate -1938871136 bytes
Out of memory trying to allocate -1938871136 bytes
Out of memory trying to allocate 1073649184 bytes
Out of memory trying to allocate 1073649184 bytes
Out of memory trying to allocate 595867104 bytes
Out of memory trying to allocate 595867104 bytes
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 36483 on node down exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

---

We have been discussing this topic in dealii/dealii#8565. We tried to locate the error using several git bisect sessions, and the problem seems to have been introduced in one of the following commits f4e70c8...a08bfa5. The test case completes with f4e70c8 and fails with a08bfa5 for the first time. We were not able to narrow the selection of commits down any further, since we were not able to build hypre successfully for any of them.

I know that, since this is a very special corner case, it will most probably not be easy to find the source of the error, but maybe you have a very first idea on where this error may come from regarding the gdb output above. I hope that we could make a joint effort to tackle this problem together.

Please let me know if you need further information on our scenario, or if you would like to work with the aforementioned deal.II testcase and need assistance with it. You'll find the sourcecode on this branch. Pardon me if it you find it untidy - it is in some kind of work-in-progress state. If desired, I will try to boil it down to a minimal working example.

BoomerAMGSetup issue

Hi:

I'm getting an error in the BoomerAMGSetup. At present I can reproduce this on 5760 MPI processes and more and only sporadically at lower number of ranks. It looks like a comm_handle is not being consistently destroyed by all ranks. Below is the information I could extract from the DDT stack trace. A majority of ranks are destroying a comm_handle in hypre_ParCSRMatrixExtractBExt_Overlap while a couple seem to have exited the routine and are trying to destroy a different handle. I recognize this is not much to go on but would appreciate any ideas you have to debug this. I have a dedicated chunk of time tomorrow (09/10) on one of the machines so I could try out any suggestions you have. Otherwise the usual turnaround time is much slower. Interestingly, the error being reported to the command line is a oom-kill event.

From DDT the stack trace I am getting is:

Majority of MPI Ranks:

hypre_BoomerAMGSetup(par_amg_setup.c:1715)
hypre_BoomerAMGBuildExtPICCInterp(par_lr_interp.c:1982)
hypre_exchange_interp_data(aux_interp.c:601)
hypre_ParCSRMatrixExtractBExt_Overlap(par_csr_matop.c:1464)
hypre_ParCSRCommHandleDestroy(par_csr_communication.c:537)
hypre_exchange_interp_data(aux_interp.c:601)
hypre_ParCSRCommHandleDestroy(par_csr_communication.c:537)

Couple of MPI Ranks:

hypre_BoomerAMGSetup(par_amg_setup.c:1715)
hypre_BoomerAMGBuildExtPICCInterp(par_lr_interp.c:1982)
hypre_exchange_interp_data(aux_interp.c:607)
hypre_ParCSRCommHandleDestroy(par_csr_communication.c:537)

GPU version segfault

Hi,

I am trying to use Hypre GPU version but I am running into the following issue when calling HYPRE_StructVectorSetConstantValues from Parflow. Everything works fine when compiling without the GPU support.

The HYPRE vector is created as follows:

 HYPRE_StructVectorCreate(MPI_COMM_WORLD, instance_xtra->hypre_grid, 
   &instance_xtra->hypre_x)); 
 HYPRE_StructVectorSetNumGhost(instance_xtra->hypre_x, full_ghosts); 
 HYPRE_StructVectorInitialize(instance_xtra->hypre_x); 
 HYPRE_StructVectorSetConstantValues(instance_xtra->hypre_x, 0.0e0);

GDB output:

Program received signal SIGSEGV, Segmentation fault.
BoxLoopforall<__nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(hypre_StructVector_struct*, double), &hypre_StructVectorSetConstantValues, 1u>, void (int), hypre_Boxloop_struct, double*, double> >(int, int, __nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(hypre_StructVector_struct*, double), &hypre_StructVectorSetConstantValues, 1u>, void (int), hypre_Boxloop_struct, double*, double>) (
    policy=0, length=length@entry=600000, loop_body=...)
    at _hypre_struct_mv.h:802
802	      HYPRE_CUDA_LAUNCH( forall_kernel, gDim, bDim, loop_body, length );
#0  BoxLoopforall<__nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(hypre_StructVector_struct*, double), &hypre_StructVectorSetConstantValues, 1u>, void (int), hypre_Boxloop_struct, double*, double> >(int, int, __nv_hdl_wrapper_t<false, false, __nv_dl_tag<int (*)(hypre_StructVector_struct*, double), &hypre_StructVectorSetConstantValues, 1u>, void (int), hypre_Boxloop_struct, double*, double>)
    (policy=0, length=length@entry=600000, loop_body=...)
    at _hypre_struct_mv.h:802
#1  0x00002aaaab0cb22c in hypre_StructVectorSetConstantValues ()
    at nvcc_internal_extended_lambda_implementation:420
#2  0x000000000049be99 in SMGInitInstanceXtra (problem=0x0, grid=0x0, 
    problem_data=0x46a5b10, pf_matrix=0x2aaaf2299200, temp_data=0x0)
    at /p/project/cesmtst/hokkanen1/pfdir/parflow/pfsimulator/parflow_lib/pf_smg.c:365

Any ideas?

Thanks,
Jaro