Unable to Build NALU-CFD with Trilinos Configured for OpenMP

[nalucfd]

Unable to Build NALU-CFD with Trilinos Configured for OpenMP

[alanw0]

Hi Si,
In Nalu we have our parallel_for loops hard-wired to use Kokkos::Serial because we haven't yet verified that we are thread-safe. I know we need to do that, and will probably do it soon, but haven't yet.
I'm not sure why it can't utilize a trilinos that was built with openmp enabled. I would have thought that even if openmp is enabled, Kokkos::Serial is always available.

Alan

[nmhamster]

Hey @alanw0 - I agree it should always be available but its something to do with a default assumption somewhere in the code I think (i.e. that the default backend when OpenMP is enabled is != Serial). Let me know when we can start more aggressive OpenMP checking and performance work, we are on hold at the moment due to the above although trying to push MPI only ahead.

[sayerhs]

@alanw0

Should we be forcing TeamHandleType to use Kokkos::Serial as the execution space in KokkosInterface.h?

typedef typename Kokkos::TeamPolicy<typename Kokkos::DefaultExecutionSpace, DynamicScheduleType>::member_type TeamHandleType;

I suspect this is picking up OpenMP whereas the DeviceTeamPolicy is forced to Kokkos::Serial.

[alanw0]

@sayerhs possibly so, that's worth a try, maybe @nmhamster wants to try that in his build to see if it helps.

[nmhamster]

@sayerhs @alanw0 - I can take a look at that and get back to you. I wanted to check if you were routinely compiling with OpenMP etc but it sounds like you are working on the parallelization etc still. Hope to get back to you with some good news shortly!

[sayerhs]

@nmhamster Can you please share the CMake doConfig that you are using to compile Trilinos with OpenMP flags?

[sayerhs]

@nmhamster @alanw0 Please see #157 and let me know if this fixes the compilation issues with Trilinos configured with OpenMP. I have tested this with Trilinos with the following options:

-DTrilinos_ENABLE_OpenMP:BOOL=ON \
-DKokkos_ENABLE_OpenMP:BOOL=ON \