Issue building on Perlmutter.

[YohannDudouit]

Issue building on Perlmutter.

[rchen20]

@YohannDudouit I'm waiting for access to the perlmutter system, and can take a closer look at this after that goes through.

[rhornung67]

@rchen20 can you take a look at this?

@YohannDudouit thank you for pointing this out. We will fix.

[YohannDudouit]

Thank you very much @rchen20 ! I will get back to this also next week, unfortunately my CMake skills are really bad, any feedback on how you build on Perlmutter will be more than appreciated!

[YohannDudouit]

We made some progress on this, by updating RAJA we got RAJA building and installing, but when including RAJA in GEOSX we get the same compilation error.

[YohannDudouit]

Okay, so the issue was that -D RAJA_ENABLE_VECTORIZATION=OFF defines RAJA_ENABLE_VECTORIZATION and since RAJA doesn't check the value but just checks if RAJA_ENABLE_VECTORIZATION is defined then it was using vectorization...

[rchen20]

Hi @YohannDudouit, I've tried -DRAJA_ENABLE_VECTORIZATION=Off on a couple different builds on LLNL systems, and it seems to remain off. In your build, would you mind checking whether /* #undef RAJA_ENABLE_VECTORIZATION */ is an existing line in your build/include/RAJA/config.hpp file? If the vectorization variable is defined there, could you try removing the entire build directory, and rebuilding again? Perhaps the past builds are affecting your current build, and I'd like to eliminate that possibility by removing the build directory and rebuilding. Thanks!

[YohannDudouit]

Hi @rchen20 , I'm a bit confused by what you are asking me to do. I have checked build/include/RAJA/config.hpp, line 179 there is /* #undef RAJA_ENABLE_VECTORIZATION */. However, I'm a bit confused by what you are asking beyond that, do you want me to add again -D RAJA_ENABLE_VECTORIZATION=OFF, and rebuild RAJA?
When testing the issue was this:

#if defined(RAJA_ENABLE_VECTORIZATION)

This line line only checks if RAJA_ENABLE_VECTORIZATION is defined, so if I set it to OFF it is defined.

Remark: set( RAJA_ENABLE_VECTORIZATION OFF CACHE BOOL "" FORCE) has the correct behavior as I suppose that CMake does not define the variable when OFF.

[rchen20]

Sorry @YohannDudouit, I'll explain a little more clearly. I'm not able to reproduce your error on LLNL systems (still awaiting Perlmutter access), and when I simply pass -DRAJA_ENABLE_VECTORIZATION=Off to the cmake line, that creates the necessary /* #undef RAJA_ENABLE_VECTORIZATION */ in build/include/RAJA/config.hpp which turns off all of these macro guards:

#if defined(RAJA_ENABLE_VECTORIZATION)

Also, set( RAJA_ENABLE_VECTORIZATION OFF CACHE BOOL "" FORCE) will also set the variable, which ought to trigger the macro guards in the same way (as defined), so it's odd that this would work for you.

This leads me to believe that it's possible that your build/include/RAJA/config.hpp was an old one generated when the vectorization variable was turned On previously, and did not get updated to have the variable undefined, so I was hoping that you could try rebuilding RAJA from scratch to confirm that theory. And if you've already done this to no avail, then I'll likely need to try debugging this on Perlmutter itself when I get an account.

[YohannDudouit]

I don't think your hypothesis is correct, otherwise removing -DRAJA_ENABLE_VECTORIZATION=Off would not have worked.

[rchen20]

I don't think your hypothesis is correct, otherwise removing -DRAJA_ENABLE_VECTORIZATION=Off would not have worked.

That's the strange thing, because when I add -DRAJA_ENABLE_VECTORIZATION=Off on our systems, it works as intended (no vectorization code generated, and the variable is marked as undefined in config.hpp).

Perhaps I have misunderstood your solution/problem. Are you saying that this alone works for you?

set( RAJA_ENABLE_VECTORIZATION OFF CACHE BOOL "" FORCE)

But, these two combined simultaneously do not work for you?

set( RAJA_ENABLE_VECTORIZATION OFF CACHE BOOL "" FORCE)
-DRAJA_ENABLE_VECTORIZATION=Off

And, when you tried this alone, it did not work?

-DRAJA_ENABLE_VECTORIZATION=Off <== This works for me.

Also, would you mind showing your final cmake line to build RAJA?

[YohannDudouit]

You're correct.
This works:

set( RAJA_ENABLE_VECTORIZATION OFF CACHE BOOL "" FORCE)

Having both fails, I haven't tried having only -DRAJA_ENABLE_VECTORIZATION=Off.
The CMake command to which this was added is:

ExternalProject_Add( raja
                     URL ${RAJA_URL}
                     PREFIX ${PROJECT_BINARY_DIR}/raja
                     INSTALL_DIR ${RAJA_DIR}
                     DEPENDS ${RAJA_DEPENDS}
                     BUILD_COMMAND ${TPL_BUILD_COMMAND} VERBOSE=1
                    #  BUILD_COMMAND make VERBOSE=1
                     INSTALL_COMMAND ${TPL_INSTALL_COMMAND}
                     CMAKE_GENERATOR ${TPL_GENERATOR}
                     CMAKE_ARGS -D CMAKE_C_COMPILER=${CMAKE_C_COMPILER}
                                -D CMAKE_CXX_COMPILER:PATH=${CMAKE_CXX_COMPILER}
                                -D CMAKE_CXX_FLAGS:PATH=${CMAKE_CXX_FLAGS}
                                -D CMAKE_BUILD_TYPE:STRING=${CMAKE_BUILD_TYPE}
                                -D CMAKE_CXX_FLAGS_RELEASE:STRING=${CMAKE_CXX_FLAGS_RELEASE}
                                # -D CMAKE_VERBOSE_MAKEFILE:BOOL=${CMAKE_VERBOSE_MAKEFILE}
                                -D CMAKE_VERBOSE_MAKEFILE:BOOL=TRUE
                                -D ENABLE_CUDA:BOOL=${ENABLE_CUDA}
                                -D CMAKE_CUDA_COMPILER=${CMAKE_CUDA_COMPILER}
                                -D CUDA_TOOLKIT_ROOT_DIR=${CUDA_TOOLKIT_ROOT_DIR}
                                -D CUDA_ARCH:CACHE=${CUDA_ARCH}
                                -D CMAKE_CUDA_FLAGS:STRING=${RAJA_CUDA_FLAGS}
                                -D ENABLE_TESTS:BOOL=OFF
                                -D ENABLE_EXAMPLES:BOOL=OFF
                                -D ENABLE_OPENMP:BOOL=${ENABLE_OPENMP}
                                -D CMAKE_INSTALL_PREFIX:PATH=<INSTALL_DIR>
                                -D BUILD_SHARED_LIBS=${BUILD_SHARED_LIBS}
                                -D CMAKE_POSITION_INDEPENDENT_CODE=${CMAKE_POSITION_INDEPENDENT_CODE}
                                -D RAJA_ENABLE_VECTORIZATION=OFF
                    )

[rchen20]

Thanks for digging that up! I'd suggest using only -DRAJA_ENABLE_VECTORIZATION=Off, but that seems moot given that you got something to work. Nothing looks wrong in your cmake line either. I assume you're using a recent version of CMake, given your usage of CUDA_ARCH? When I get access to Perlmutter, I'll try just the vectorization variable by itself.

[rhornung67]

@rchen20 is this resolved? if so, please close this