kokkos/kokkos-fft

A shared-memory FFT for the Kokkos ecosystem

Kokkos: Core Libraries

Kokkos Core implements a programming model in C++ for writing performance portable applications targeting all major HPC platforms. For that purpose it provides abstractions for both parallel execution of code and data management. Kokkos is designed to target complex node architectures with N-level memory hierarchies and multiple types of execution resources. It currently can use CUDA, HIP, SYCL, HPX, OpenMP and C++ threads as backend programming models with several other backends in development.

Kokkos Core is part of the Kokkos C++ Performance Portability Programming Ecosystem.

Kokkos is a Linux Foundation project.

Learning about Kokkos

To start learning about Kokkos:

• Kokkos Lectures: they contain a mix of lecture videos and hands-on exercises covering all the important capabilities.

• Programming guide: contains in "narrative" form a technical description of the programming model, machine model, and the main building blocks like the Views and parallel dispatch.

• API reference: organized by category, i.e., core, algorithms and containers or, if you prefer, in alphabetical order.

• Use cases and Examples: a serie of examples ranging from how to use Kokkos with MPI to Fortran interoperability.

Obtaining Kokkos

The latest release of Kokkos can be obtained from the GitHub releases page.

The current release is 4.3.01.

curl -OJ -L https://github.com/kokkos/kokkos/archive/refs/tags/4.3.01.tar.gz
# Or with wget
wget https://github.com/kokkos/kokkos/archive/refs/tags/4.3.01.tar.gz

To clone the latest development version of Kokkos from GitHub:

git clone -b develop  https://github.com/kokkos/kokkos.git

Building Kokkos

To build Kokkos, you will need to have a C++ compiler that supports C++17 or later. All requirements including minimum and primary tested compiler versions can be found here.

Building and installation instructions are described here.

You can also install Kokkos using Spack: spack install kokkos. Available configuration options can be displayed using spack info kokkos.

For the complete documentation: kokkos.org/kokkos-core-wiki/

Support

For questions find us on Slack: https://kokkosteam.slack.com or open a GitHub issue.

For non-public questions send an email to: crtrott(at)sandia.gov

Contributing

Please see this page for details on how to contribute.

Citing Kokkos

Please see the following page.

License

Under the terms of Contract DE-NA0003525 with NTESS, the U.S. Government retains certain rights in this software.

The full license statement used in all headers is available here or here.

kokkos-fft's Issues

Function with execution space should be asynchronous. SYCL backend performs a barrier inside

Explore existing DDC implementation

Summary

Before starting development, we should start looking at what's already been done in DDC

End-User Goal

Business Goal

Acceptance Criteria

Measurement of Success

We understand the existing implementation & its choices.

Tasks

Update based on reviews from core team

Comments from core.

• #79
• #80
• #81
• #82
• #83
• #84
• #85
• #86
• #90

Review docs

Googletest should be parameterized over types using `::testing::Types`

nvcc version should be 11.0

Introduce RocFFT backend which may be more suitable

Seek for non-contiguous FFT plans which may be available in some cases

Kokkos version check

Hi all another issue appears to be the lack of check on the Kokkos Version being used when building against a pre-installed Kokkos library.
I am suspecting that for the time being it works with 4.2.0 and maybe up to 4.0.0 but likely not against anything older than that?
It would be good to add something in CMakeLists.txt to warn users if they attempt to install against an unsupported version.
Also updating the documentation to reflect what is actually supported would be great!

Consider an implementation not relying on `Kokkos_ENABLE_CUDA_CONSTEXPR`

Use uninitialize mode for temporary views in order to improve performance

Create tests for FFT

Preparation for release

Here are the current lists from #8 (comment).

• Add AUTHORS, who should be? Just myself for the moment (#23)
• Unuse double underscore identifiers like __KOKKOSFFT_NORMALIZATION_HPP__ (#8)
• Rename FFT_Normalization to Normalization because they are already under KokkosFFT namespace (#8)
• Add installation part to CMakeLists.txt. Also add installation test in GitHub actions (#22). Thanks a lot for your help! @cedricchevalier19
• Support transforms over odd number. The number of points for FFT direction is limited to be even number (so that the size after transform would be n/2+1) (#21)
• More static assertions. Particularly, we assume that the In and Out Views have same Layout and ranks, which should be checked. (#26)
• Introduce ExecSpace as a template argument as well as DDC. This allows the coexistence of fft helpers for both host and device (#12):

// As a template parameter
KokkosFFT::fft<Kokkos::OpenMP>(a, out); // a and out are on Host or Device is CPU
KokkosFFT::fft<Kokkos::Cuda>(a, out); // a and out are on Device

// As an argument
KokkosFFT::fft(Kokkos::OpenMP(), a, out); // a and out are on Host or Device is CPU
KokkosFFT::fft(Kokkos::Cuda(), a, out); // a and out are on Device

We can also make a check for memory and exec space consistency by

static_assert(
  Kokkos::SpaceAccessibility<ExecSpace, MemorySpace>::accessible,
  "MemorySpace has to be accessible for ExecutionSpace."
);

• Add capability to reuse plans. This is particularly important for NVIDIA GPUs, which has some overheads to create plans (#19)
• Management of default arguments. As well as numpy, I would like to allow the function calls with minimum overloading (#19)

KokkosFFT::fft(execution_space(), a, out);
KokkosFFT::fft(execution_space(), a, out, /*axis=*/-1);
KokkosFFT::fft(execution_space(), a, out, /*n=*/n0);
KokkosFFT::fft(execution_space(), a, out, KokkosFFT::FFT_Normalization::BACKWARD);
KokkosFFT::fft(execution_space(), a, out, KokkosFFT::FFT_Normalization::BACKWARD, /*axis=*/*-1);
KokkosFFT::fft2(execution_space(), a, out, KokkosFFT::FFT_Normalization::BACKWARD, /*axes=*/axes_type{-2, -1});
KokkosFFT::fft(execution_space(), a, out, plan, KokkosFFT::FFT_Normalization::BACKWARD, /*axis=*/-1);

Particularly, it is difficult to distinguish n and axis, where n is a size_t type and axis is an int type.

• Introduce the Impl namespace to disallow users to access implementation details (#13)
• Add missing functions such as hfft (#16)
• Add missing helper functions such as fftfreq (#17)
• Add capability to allow optimal argument n. (See fft) (#16)
• Googletest should be parameterized over types using ::testing::Types (#14)
• Add docs to explain what are our interfaces and how to use them (#42)
• introduce a clang format file ideally from Kokkos repo (#20)
• Adding perf_test with googlebenchmark (#41)
• Add capability to call KokkosFFT APIs from both host and device. This is not quite useful so I will make it optional. (#18)

using execution_space = Kokkos::DefaultExecutionSpace;
using host_execution_space = Kokkos::DefaultHostExecutionSpace;
KokkosFFT::fft(execution_space(), a, out); // Calling FFTs on Device
KokkosFFT::fft(host_execution_space(), h_a, h_out); // Calling FFTs on Host

• Improve docker files and add CI on GPUs (#29)
  Thanks a lot for your help! @pzehner
• Kokkos::Threads support (#31, #46)
• Adding 4D to 8D capabilities (#51)
• #63

The remaining issues which may be important in the future

• #57
• #58
• Add Intel GPU support based on oneMKL (#38)
• Add nvidia-hpc-sdk env. Build issue on the container (#50).
• #59
• #60
• #61
• #62
• #65

Recover GPU runner

Self-hosted runner needs to be recovered as we have not used it for 30 days. (See actions/runner#756)

Error handling let them know which line causes the error

Submodules are actually plain directories

Dependency submodules in tpls are actually duplicates of the target repositories and occupy 21 Mb of space. They should be replaced by actual submodules.

Cannot compile if CMAKE_BUILD_TYPE is not set or set to Debug

When building with CMAKE_BUILD_TYPE unset or set to Debug, I got the following error:

/local/home/user/Projets/kokkos-fft/fft/src/KokkosFFT_Helpers.hpp(27): error: no instance of function template "KokkosFFT::Impl::is_out_of_range_value_included" matches the argument list
            argument types are: (std::vector<int, std::allocator<int>>, const int)
   !KokkosFFT::Impl::is_out_of_range_value_included(axes, rank)

Automatic base images cleanup fails

As seen here, the cleanup seems ill-configured.

Avoid variables or functions starting from under score

Support in-place transform

Create a Spack recipe for KokkosFFT

Create a spack recipe for KokkosFTT, locally and then upstream it to spack.

Rename KokkosFFT_OpenMP_*.hpp to KokkosFFT_Host_*.hpp

KokkosFFT_OpenMP_*.hpp handles OpenMP, Threads and Serial.
It is better to rename Host as suggested in #109

support plan: latest two releases

Cleanup Plan class. Disallow copies.

Epic FFT Goal

Description

The goal is to design a performance portable FFT library with a Kokkos-compatible interface.

At first we target shared memory and exclude distributed memory.

Related User Stories

• #3
• #2
• #11

Looks for FFTW when Kokkos::Threads is enabled

It would be nice to modify the config to include the use case of Kokkos::Threads, it works fine and is actually used commonly on MacOS since OpenMP is not available by default.

Allow to access from Views with strided Layouts. Internally, the input and/or output buffers with `LayoutRight` are created and the strided Views are copied into these buffers.

Testing Kokkos-FFT Sycl/oneMKL backend on Intel PVC

SYCL support with oneMKL backend has been implemented. It is confirmed that all tests passed. Now implementing a CI for SYCL backend.

LICENCSE: Apache 2 LLVM exception

Pre-build Kokkos on CI

Build and install Kokkos on Release mode in the images.

Kokkos would be discovered as an installed library, backend parameters should be deduced and not passed again.

The test example build at the end of the workflow should properly use this installed version of Kokkos.

Optionally, KokkosFFT_INTERNAL_Kokkos should be removed.

Write a small example of numpy fft derivative

3 Execution spaces can coexist, SFINAE needs to be improved

Pertinence of Kokkos_Version_Check

I think the feature in cmake/Kokkos_Version_Check.cmake should be easily replaced by the CMake standard find_package version option:

find_package(Kokkos 4.2.0 REQUIRED)

Maybe we don't need to specify the version up to the patch level (i.e. 4.2 should be enough).

Otherwise, the cmake/Kokkos_Version_Check.cmake is invalid and should be:

diff --git a/cmake/Kokkos_Version_Check.cmake b/cmake/Kokkos_Version_Check.cmake
index 3f3f5cb..98a38f5 100644
--- a/cmake/Kokkos_Version_Check.cmake
+++ b/cmake/Kokkos_Version_Check.cmake
@@ -1,11 +1,11 @@
 function(check_minimum_required_kokkos kokkos_required_version)
-    if(${Kokkos_VERSION} STREQUAL "")
+    if(Kokkos_VERSION STREQUAL "")
         message(FATAL_ERROR "Kokkos_VERSION not set. Cannot check Kokkos satisfies the minimum required version.")
     else()
-        if(${Kokkos_VERSION} VERSION_GREATER_EQUAL ${kokkos_required_version})
+        if(Kokkos_VERSION VERSION_GREATER_EQUAL ${kokkos_required_version})
             message(STATUS "Found Kokkos version ${Kokkos_VERSION} at ${Kokkos_DIR}")
         else()
             message(FATAL_ERROR "Kokkos FFT ${KOKKOSFFT_VERSION} requires ${kokkos_required_version} or later.")
         endif()
     endif()

Embed licence

Stricter checks in the CI

Ideally we should not allow warnings.
I am in favor to add CMAKE_CXX_FLAGS="-Wall -Werror" to the CI.

`_fft` should be refined not to include Execution space Use execspace from Plan.

API design

Summary

As a code developer, I want to know the list of features supported by all back-ends, on:

• CPU,
• AMD GPU,
• Nvidia GPU,
• Intel GPU.

End-User Goal

• I'd like a list of function calls to be available that only differ in terms of template parameters describing the execution space and memory space, and that I can use on all four back-ends.
• I'd accept calls to not be available when the memory space where my data resides is not accessible from the execution space I select.
• I'd expect the configuration options to be similar to that of Kokkos so that I don't have much to do by default when already using Kokos to use the FFT.

Business Goal

Long term goal for the project

Acceptance Criteria

Measurement of Success

Tasks

Add more advanced examples. Derivatives in Fourier space, simulations with spectral method?

Support distributed plan with MPI decomposition