Unrecognized OpenMP pragma

[biochem-fan]

Unrecognized OpenMP pragma

[do-jason]

@biochem-fan To remove messages for any ICC execution, one can set __INTEL_PRE_CFLAGS shell environment variable. In this case, you can set export __INTEL_PRE_CFLAGS="-diag-disable=10441" in your shell. For permanent impact, it can be added in .bashrc file.

[biochem-fan]

VERBOSE=1 make indicated that my CMake 3.16.3 was calling NVCC like this:

/home1/software/packages/cuda-12.2/bin/nvcc /home1/software/packages/relion/src/acc/cuda/cuda_ml_optimiser.cu -c -o /home1/software/packages/relion/build-5.0/src/apps/CMakeFiles/relion_gpu_util.dir/__/acc/cuda/./relion_gpu_util_generated_cuda_ml_optimiser.cu.o -ccbin /home1/software/packages/oneapi-2023.2.0/compiler/2023.2.1/linux/bin/intel64/icc -m64 -DINSTALL_LIBRARY_DIR=/usr/local/lib/ -DSOURCE_DIR=/home1/software/packages/relion/src/ -DACC_HIP=3 -DACC_CUDA=2 -DACC_CPU=1 -D_CUDA_ENABLED -DMKLFFT -DUSE_IPP -DHAVE_SINCOS -DHAVE_TIFF -DHAVE_PNG -DHAVE_JPEG -Xcompiler ,\"-O3\",\"-ip\",\"-g\",\"-xCORE-AVX2\",\"-restrict\",\"-ipp\",\"-diag-disable=10441,10412\",\"-O3\",\"-DNDEBUG\" -DNVCC -I/home1/software/packages/cuda-12.2/include -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include -I/ime1/software/packages/oneapi-2023.2.0/mkl/2023.2.0/include/fftw -I/home1/software/packages/relion -I/home1/software/packages/relion/external/fltk/include -I/usr/include/x86_64-linux-gnu -I/usr/include

There was no -fopenmp flag! By adding ,\"-fopenmp\" at the end of -Xcompiler argument, the warning was gone.

The question is: how to do this in CMake?

Until this is fixed, the ICC build of RELION is broken, at least for CUDA. I don't know if this affects ALTCPU and SYCL as well.

[biochem-fan]

FIY: @scheres

[do-jason]

@biochem-fan Could you show the whole output of cmake command after commenting out line 626~627 of CMakeLists.txt file to check if OpenMP finding is properly done? OpenMP flags should be found for host from line 573~580 of src/apps/CMakeLists.txt file.

[biochem-fan]

$ cmake .. -DMKLFFT=ON -DCMAKE_C_COMPILER=icc -DCMAKE_CXX_COMPILER=icpc -DMPI_C_COMPILER=/usr/bin/mpicc
-DMPI_CXX_COMPILER=/usr/bin/mpicxx -DCMAKE_C_FLAGS="-O3 -ip -g -xCORE-AVX2 -restrict " -DCMAKE_CXX_FLAGS="-O3 -ip -g -xCORE-AVX2 -restrict " -DCUDA_A
RCH=60 -DTORCH_HOME_PATH=/home1/software/packages/relion-torch-models -DPYTHON_EXE_PATH=/home1/software/packages/miniconda3/envs/relion-5.0/bin/pytho
n3
-- The C compiler identification is Intel 20.2.10.20230609
-- The CXX compiler identification is Intel 20.2.10.20230609
-- Check for working C compiler: /home1/software/packages/oneapi-2023.2.0/compiler/2023.2.1/linux/bin/intel64/icc
-- Check for working C compiler: /home1/software/packages/oneapi-2023.2.0/compiler/2023.2.1/linux/bin/intel64/icc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /home1/software/packages/oneapi-2023.2.0/compiler/2023.2.1/linux/bin/intel64/icpc
-- Check for working CXX compiler: /home1/software/packages/oneapi-2023.2.0/compiler/2023.2.1/linux/bin/intel64/icpc -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- BUILD TYPE set to the default type:  'Release'
-- Using provided CUDA_ARCH=60
-- CUDA enabled - Building CUDA-accelerated version of RELION
-- Setting cpu precision to double
-- Setting accelerated code precision to single
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /home1/software/packages/cuda-12.2 (found version "12.2")
-- Using cuda wrapper to compile....
-- Cuda version is >= 7.5 and single-precision build, enable double usage warning.
-- Found MPI_C: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so (found version "3.1")
-- MKL FFTW wrapper header files: /ime1/software/packages/oneapi-2023.2.0/mkl/2023.2.0/include/fftw
-- Using Intel IPP library: /ime1/software/packages/oneapi-2023.2.0/ipp/2021.9.0
-- Found X11: /usr/include
-- Looking for XOpenDisplay in /usr/lib/x86_64-linux-gnu/libX11.so
-- Looking for XOpenDisplay in /usr/lib/x86_64-linux-gnu/libX11.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Could NOT find FLTK (missing: FLTK_LIBRARIES FLTK_INCLUDE_DIR)
-- No FLTK installation was found
-- Found previously built non-system FLTK libraries that will be used.
-- FLTK_INCLUDE_DIR: /home1/software/packages/relion/external/fltk/include
-- FLTK_LIBRARIES:   /home1/software/packages/relion/external/fltk/lib/libfltk.so
-- Looking for sincos
-- Looking for sincos - found
-- Looking for __sincos
-- Looking for __sincos - not found
-- Found TIFF: /usr/lib/x86_64-linux-gnu/libtiff.so (found version "4.1.0")
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11")
-- Found PNG: /usr/lib/x86_64-linux-gnu/libpng.so (found version "1.6.37")
-- Found JPEG: /usr/lib/x86_64-linux-gnu/libjpeg.so (found version "80")
-- Using Python executable: /home1/software/packages/miniconda3/envs/relion-5.0/bin/python3
-- Will try to find Torch home directory...
-- Using Torch home: /home1/software/packages/relion-torch-models
-- Will attempts to download model weights for dependent packages...
Attempting to download weights for class ranker...
Model (v1.0) loaded successfully from checkpoint /home1/software/packages/relion-torch-models/hub/checkpoints/relion_class_ranker/v1.0.ckpt
No project directory was specified... exiting! 
Attempting to download weights for Blush...
Blush model (v1.0) loaded successfully from checkpoint /home1/software/packages/relion-torch-models/hub/checkpoints/relion_blush/v1.0.ckpt
No job target was specified for Blush regularization... exiting!
Attempting to download weights for ModelAngelo...
Please make sure you have set the environment variable TORCH_HOME
to a suitable directory, visible to all relevant users!
Successful!
Path to model is /home1/software/packages/relion-torch-models/hub/checkpoints/model_angelo_v1.0/nucleotides
Please make sure you have set the environment variable TORCH_HOME
to a suitable directory, visible to all relevant users!
Successful!
Path to model is /home1/software/packages/relion-torch-models/hub/checkpoints/model_angelo_v1.0/nucleotides_no_seq
-- Successfully downloaded model weights
-- Building static libs (larger build size and binaries)
Running apps/CMakeLists.txt...
-- CMAKE_BINARY_DIR:/home1/software/packages/relion/build-test
-- Git commit ID: 84ba16768036bfef791f40df70af87347eccb1ad
-- Found OpenMP_C: -qopenmp (found version "5.0")
-- Found OpenMP_CXX: -qopenmp (found version "5.0")
-- Found OpenMP: TRUE (found version "5.0")
-- CMAKE_C_FLAGS : -O3 -ip -g -xCORE-AVX2 -restrict -diag-disable=10441,10412 -diag-disable=10441,10412
-- CMAKE_CXX_FLAGS : -O3 -ip -g -xCORE-AVX2 -restrict -ipp -diag-disable=10441,10412
-- Configuring done
-- Generating done
-- Build files have been written to: /home1/software/packages/relion/build-test

[biochem-fan]

@do-jason
As we can see above, -qopenmp was found but not added to CMAKE_C_FLAGS.

[jonggwan]

@biochem-fan Line 574 of src/apps/CMakeLists.txt should be "if(OpenMP_FOUND)" based on https://cmake.org/cmake/help/latest/module/FindOpenMP.html. It is case sensitive. I think most other compilers ignored this OpenMP pragma since it is not enabled in compiler flag. Ignoring unsupported "#pragma" is expected behavior and quite interestingly Intel compiler tries to enables OpenMP pragma even if it is not enabled in compiler flag.

[biochem-fan]

@jonggwan
Thank you very much.

OPENMP_FOUND is acceptable:
https://gitlab.kitware.com/cmake/cmake/blob/master/Modules/FindOpenMP.cmake#L638

I uncommented the OPENMP FOUND debug message.
Regardless of OpenMP_FOUND or OPENMP_FOUND, I got this:

-- Found OpenMP_C: -qopenmp (found version "5.0") 
-- Found OpenMP_CXX: -qopenmp (found version "5.0") 
-- Found OpenMP: TRUE (found version "5.0")  
OPENMP FOUND
-- CMAKE_C_FLAGS : -O3 -ip -g -xCORE-AVX2 -restrict -diag-disable=10441,10412 -diag-disable=10441,10412
-- CMAKE_CXX_FLAGS : -O3 -ip -g -xCORE-AVX2 -restrict -ipp -diag-disable=10441,10412

The if clause is executed but -qopenmp is nonetheless ignored.

[do-jason]

@biochem-fan Now I learned more about cmake. ADD_SUBDIRECTORY is just child scope and variable set in that file is not propagated to parent scope. set() need "PARENT_SCOPE" as the last argument to make it effective in parent scope. So, 582~583 with the below will fix this problem.

set(CMAKE_C_FLAGS "-std=c99 ${CMAKE_C_FLAGS}" PARENT_SCOPE)
set(CMAKE_CXX_FLAGS "-fPIC -std=c++14 ${CMAKE_CXX_FLAGS}" PARENT_SCOPE)

[do-jason]

@biochem-fan Now I learned more about cmake. ADD_SUBDIRECTORY is just child scope and variable set in that file is not propagated to parent scope. set() need "PARENT_SCOPE" as the last argument to make it effective in parent scope. So, 582~583 with the below will fix this problem.

set(CMAKE_C_FLAGS "-std=c99 ${CMAKE_C_FLAGS}" PARENT_SCOPE)
set(CMAKE_CXX_FLAGS "-fPIC -std=c++14 ${CMAKE_CXX_FLAGS}" PARENT_SCOPE)

This may have impacted some code with OpenMP syntax so far, especially for non-Intel compilers since it looks like Intel compiler try to enable OpenMP pragma even if there is no OpenMP compiler flags.

[do-jason]

@biochem-fan This OpenMP finding in src/apps/CMakeLists.txt is different from other settings since it has target/application specific settings. How about to move FindOpenMP logic to root CMakeLists.txt file?

[biochem-fan]

@do-jason

Thank you very much. The output looks fine now.

-- CMAKE_C_FLAGS : -std=c99 -O3 -ip -g -xCORE-AVX2 -restrict -diag-disable=10441,10412 -diag-disable=10441,10412 -qopenmp
-- CMAKE_CXX_FLAGS : -fPIC -std=c++14 -O3 -ip -g -xCORE-AVX2 -restrict -ipp -diag-disable=10441,10412 -qopenmp

However, the unrecognized OpenMP #pragma persists.
The actual compilation happens at src/app/CMakeList.txt, not the top level CMakeList.txt, so whether the variables are propagated to the top level affects only the debug message, not compilation, I'm afraid.

[biochem-fan]

This OpenMP finding in src/apps/CMakeLists.txt is different from other settings since it has target/application specific settings. How about to move FindOpenMP logic to root CMakeLists.txt file?

This makes sense.

[do-jason]

@biochem-fan NVIDIA flags is set by cmake/BuileTypes.cmake from line 256 of CMakeLists.txt file while OpenMP is set after that. So, it will not have any impact to NVIDIA build. I think some reordering is required in CMakeLists.txt file.

[biochem-fan]

cmake/BuileTypes.cmake does not use CMAKE_C_FLAGS or CMAKE_CXX_FLAGS to set NVCC specific flags. So I still don't know why/where old versions of these variables are picked up.

[biochem-fan]

I confirmed that GCC without -fopenmp silently ignored #pragma. This is expected but makes this problem serious...

[biochem-fan]

By moving OpenMP related things to the top level CMakeList.txt, the pragma issue seems to be fixed (dc7a999). I also moved -diag-disable=10441 upwards.

However, NVCC (but not non-CUDA files) keeps complaining:

icc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message.

This is very strange, because the -Xcompiler argument does contain -diag-disable=10441. Actually this was present even before I rearranged CMakeList.txt as can be seen in #1038 (comment). This is very confusing. I don't really understand when NVCC captures flags and how they are passed to ICC.

[biochem-fan]

I don't really understand when NVCC captures flags and how they are passed to ICC.

This is done in the CUDA_LINK_SEPARABLE_COMPILATION_OBJECTS function https://github.com/Kitware/CMake/blob/3a1af958ee35c476b93c2003c2eebf5d2cc280ab/Modules/FindCUDA.cmake#L1893C10-L1893C49, which is called by cuda_add_library and cuda_add_executable.
Thus, OpenMP flags must be set up before them.

[biochem-fan]

I made icc-wrapper:

#!/bin/sh

echo ICC CALLED with $@ >> ICC-LOG
`which icc` $@

Analysis using nvcc test.cu -keep -ccbin ./icc-wrapper -Xcompiler "-diag-disable=10441" revealed that nvcc calls ICC several times. The first call is (probably) to test compiler compatibility and made without the flags passed in -Xcompiler. In my case, it was icc -E /tmp/tmpxft_XXXXX.cpp.

Thus, we cannot completely suppress the deprecation warnings when ICC is called from NVCC.

[biochem-fan]

Thanks everyone.

All issues have settled. I will make a commit.

[scheres]

Thanks to all!

[biochem-fan]

RELION 4.0 must be affected as well. However, no compiler warnings are displayed in 4.0.
This is because --disable-warnings is passed to NVCC. This option is not set in RELION 5.0.

# 4.0.1
$ grep NVCC_FLAGS_RELEASE CMakeCache.txt 
CUDA_NVCC_FLAGS_RELEASE:STRING=-arch=sm_60  -D__INTEL_COMPILER --default-stream per-thread --std=c++11 --disable-warnings
RELION_NVCC_FLAGS_RELEASE:STRING=-arch=sm_60  -D__INTEL_COMPILER --default-stream per-thread --std=c++11 --disable-warnings
//ADVANCED property for variable: CUDA_NVCC_FLAGS_RELEASE
CUDA_NVCC_FLAGS_RELEASE-ADVANCED:INTERNAL=1

# 5.0
$ grep NVCC_FLAGS_RELEASE CMakeCache.txt 
CUDA_NVCC_FLAGS_RELEASE:STRING=
RELION_NVCC_FLAGS_RELEASE:STRING=-arch=sm_60 --ptxas-options=-warn-double-usage -D__INTEL_COMPILER --default-stream per-thread --std=c++11 --disable-warnings
//ADVANCED property for variable: CUDA_NVCC_FLAGS_RELEASE

This is probably related to #1032.
CMake scripts underwent significant refactoring when AMD HIP port was merged.
We have to investigate.

[biochem-fan]

Removing --disable-warnings, I confirmed RELION 4.0 is also affected by this bug.

I fixed it in dcc8fe7.