kokkos/pykokkos-base

Python bindings for data interoperability with Kokkos (View, DynRankView)

Kokkos: Core Libraries

Kokkos Core implements a programming model in C++ for writing performance portable applications targeting all major HPC platforms. For that purpose it provides abstractions for both parallel execution of code and data management. Kokkos is designed to target complex node architectures with N-level memory hierarchies and multiple types of execution resources. It currently can use CUDA, HIP, SYCL, HPX, OpenMP and C++ threads as backend programming models with several other backends in development.

Kokkos Core is part of the Kokkos C++ Performance Portability Programming Ecosystem.

Kokkos is a Linux Foundation project.

Learning about Kokkos

To start learning about Kokkos:

• Kokkos Lectures: they contain a mix of lecture videos and hands-on exercises covering all the important capabilities.

• Programming guide: contains in "narrative" form a technical description of the programming model, machine model, and the main building blocks like the Views and parallel dispatch.

• API reference: organized by category, i.e., core, algorithms and containers or, if you prefer, in alphabetical order.

• Use cases and Examples: a serie of examples ranging from how to use Kokkos with MPI to Fortran interoperability.

Obtaining Kokkos

The latest release of Kokkos can be obtained from the GitHub releases page.

The current release is 4.3.01.

curl -OJ -L https://github.com/kokkos/kokkos/archive/refs/tags/4.3.01.tar.gz
# Or with wget
wget https://github.com/kokkos/kokkos/archive/refs/tags/4.3.01.tar.gz

To clone the latest development version of Kokkos from GitHub:

git clone -b develop  https://github.com/kokkos/kokkos.git

Building Kokkos

To build Kokkos, you will need to have a C++ compiler that supports C++17 or later. All requirements including minimum and primary tested compiler versions can be found here.

Building and installation instructions are described here.

You can also install Kokkos using Spack: spack install kokkos. Available configuration options can be displayed using spack info kokkos.

For the complete documentation: kokkos.org/kokkos-core-wiki/

Support

For questions find us on Slack: https://kokkosteam.slack.com or open a GitHub issue.

For non-public questions send an email to: crtrott(at)sandia.gov

Contributing

Please see this page for details on how to contribute.

Citing Kokkos

Please see the following page.

License

Under the terms of Contract DE-NA0003525 with NTESS, the U.S. Government retains certain rights in this software.

The full license statement used in all headers is available here or here.

pykokkos-base's Issues

breaking behavior with kokkos 3.7

I have had a stable code using pykokkos base + kokkos 3.6 that is broken with kokkos 3.7 release and pykokkos-base starting with commit a8f4ca6.

I use structs to hold kokkos arrays, and pybind11 to allow these structs to serve as inheritable classes on a python side. Then use the kokkos module (python side) to set the kokkos array attributes of my c++ structs.

It seems the signature of the kokkos arrays have changed and pybind11 does not recognize the kokkos module's arrays as c++ kokkos arrays anymore. It is failing with

(): incompatible function arguments. The following argument types are supported:
    1. (self: example.thtrdat_, arg0: Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace>) -> None

Invoked with: <__main__.thtrdat object at 0x7fe4c2f40c70>, (<kokkos.libpykokkos.KokkosView_float64_HostSpace_LayoutRight_1 object at 0x7fe4bcee74b0>,)
Traceback (most recent call last):
  File "/home/kschau6/test/test.py", line 29, in <module>
    simulate()
  File "/home/kschau6/test/test.py", line 14, in simulate
    test.MW = (
TypeError: (): incompatible function arguments. The following argument types are supported:
    1. (self: example.thtrdat_, arg0: Kokkos::View<double*, Kokkos::LayoutRight, Kokkos::HostSpace>) -> None

Invoked with: <__main__.thtrdat object at 0x7fe4c2f40c70>, (<kokkos.libpykokkos.KokkosView_float64_HostSpace_LayoutRight_1 object at 0x7fe4bcee74b0>,)

A small reproducer is available at https://github.com/kaschau/kokkos37Issue

Using stable branch of pybind11, python 3.9

Does not work with:
kokkos3.7, any pykokkos-base branch a8f4ca6 and after.

Works with:
kokkos3.6, and pykokkos-base before a8f4ca6

Support for memory layouts

Would it be possible to add support for memory layouts? i.e.:

view = kokkos.array("view", [10], dtype=kokkos.double, space=kokkos.HostSpace, layout=kokkos.LayoutLeft)

inline static members in Kokkos 4.0 class not persistent with CUDA backend

Kokkos 4.0 changed many class members set with Kokkos::initialize() to inline static T types. With this change it seems there is an issue with pybind11 and setting these members persistently when called from python.

Whenever using cuda, the TileSizeProperties attribute maxThreads is being set to zeros, and causes an abort at the first MDRange execution.

When Kokkos::initialize() is called (from python bound function), cudaProp.maxThreadsPerMultiProcessor (from here ) reports 1024, however, by the time we get to the MDRange policy here, the space.impl_internal_space_instance()->m_maxThreadsPerSM is 0. This causes an abort at this check here.

I am only having an issue with CUDA, and it works fine with OpenMP and Serial backends. It has been consistent with every host/device compiler I have tried.

Primarily gcc 9.4.0/intel19.04 + CUDA 11.7

Multiple definitions at link time

Hi,

I have an object model using Kokkos views I want to export to python using pybind11.
The export code is in fargOCApy library. So I added the following link directive in my CMake files:

pybind11_add_module(fargOCApy
  fargOCApy.cpp
  exportDisk.cpp
  exportEnvironment.cpp
  exportScalarField.cpp
  exportSimulation.cpp
  )

target_link_libraries(fargOCApy PUBLIC fargoseqrt libpykokkos-core)
set_target_properties(fargOCApy PROPERTIES
  POSITION_INDEPENDENT_CODE 1
  COMPILE_DEFINITIONS FARGO_SEQ
  )

fargoseqrt contains the object model.

If I do that, the link fails with:

../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/pool_variants/pools.cpp.o: In function `generate_pool_variants(pybind11::module_&)':
/work/aminiussi/fargo/pybind11/dbg/external/pykokkos-base/src/pool_variants/pools.cpp:11: multiple definition of `generate_pool_variants(pybind11::module_&)'
CMakeFiles/fargOCApy.dir/__/__/external/pykokkos-base/src/pool_variants/pools.cpp.o:/work/aminiussi/fargo/pybind11/dbg/external/pykokkos-base/src/pool_variants/pools.cpp:11: first defined here
../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/variants/atomics.cpp.o: In function `generate_atomic_variants(pybind11::module_&)':
/work/aminiussi/fargo/pybind11/external/pykokkos-base/src/variants/atomics.cpp:47: multiple definition of `generate_atomic_variants(pybind11::module_&)'
CMakeFiles/fargOCApy.dir/__/__/external/pykokkos-base/src/variants/atomics.cpp.o:/work/aminiussi/fargo/pybind11/external/pykokkos-base/src/variants/atomics.cpp:47: first defined here
../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/variants/views.cpp.o: In function `std::_Vector_base<Kokkos::Tools::Experimental::Impl::ValueHierarchyNode<long, void>, std::allocator<Kokkos::Tools::Experimental::Impl::ValueHierarchyNode<long, void> > >::_M_deallocate(Kokkos::Tools::Experimental::Impl::ValueHierarchyNode<long, void>*, unsigned long)':
/work/aminiussi/fargo/pybind11/dbg/external/pykokkos-base/src/variants/views.cpp:42: multiple definition of `generate_view_variants(pybind11::module_&)'
CMakeFiles/fargOCApy.dir/__/__/external/pykokkos-base/src/variants/views.cpp.o:/work/aminiussi/fargo/pybind11/dbg/external/pykokkos-base/src/variants/views.cpp:42: first defined here
make[2]: *** [src/fargOCApy/fargOCApy.cpython-36m-x86_64-linux-gnu.so] Error 1

A verbose build indicates that the pool.o object does indeed appears twice on the link command:

cd /work/aminiussi/fargo/pybind11/dbg/src/fargOCApy && /panfs/panasas/softs/occigen/tools/cmake/3.16.1/bin/cmake -E cmake_link_script CMakeFiles/fargOCApy.dir/link.txt --verbose=1
/opt/software/common/intel/compilers_and_libraries_2019.4.243/linux/bin/intel64/icpc -fPIC  -use-intel-optimized-headers -shared-intel -wd9 -qopenmp -g -DKOKKOS_DEPENDENCE -fopenmp -shared  -o fargOCApy.cpython-36m-x86_64-linux-gnu.so CMakeFiles/fargOCApy.dir/fargOCApy.cpp.o CMakeFiles/fargOCApy.dir/exportDisk.cpp.o CMakeFiles/fargOCApy.dir/exportEnvironment.cpp.o CMakeFiles/fargOCApy.dir/exportScalarField.cpp.o CMakeFiles/fargOCApy.dir/exportSimulation.cpp.o CMakeFiles/fargOCApy.dir/__/__/external/pykokkos-base/src/pool_variants/pools.cpp.o CMakeFiles/fargOCApy.dir/__/__/external/pykokkos-base/src/variants/atomics.cpp.o CMakeFiles/fargOCApy.dir/__/__/external/pykokkos-base/src/variants/views.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/libpykokkos.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/backend_version.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/enumeration.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/available.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/common.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/tools.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/pool_variants/pools.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/variants/atomics.cpp.o ../../external/pykokkos-base/CMakeFiles/libpykokkos-core.dir/src/variants/views.cpp.o   -L/panfs/panasas/cnt0026/oca7233/SHARED/boost-1.73.0/lib  -Wl,-rpath,/panfs/panasas/cnt0026/oca7233/SHARED/boost-1.73.0/lib:/work/aminiussi/fargo/pybind11/dbg/src:/work/aminiussi/fargo/pybind11/dbg/src/noopmpi:/panfs/panasas/cnt0026/oca7233/SHARED/hdf5-1.12.0-1-intel-2019.4/lib:/work/aminiussi/fargo/pybind11/dbg/src/symba7:/work/aminiussi/fargo/pybind11/dbg/src/swift:/work/aminiussi/fargo/pybind11/dbg/external/kokkos/containers/src:/work/aminiussi/fargo/pybind11/dbg/external/kokkos/core/src ../libfargoseqrt.so ../noopmpi/libnoopmpi.so /panfs/panasas/cnt0026/oca7233/SHARED/boost-1.73.0/lib/libboost_program_options.so.1.73.0 /panfs/panasas/cnt0026/oca7233/SHARED/boost-1.73.0/lib/libboost_chrono.so.1.73.0 /panfs/panasas/cnt0026/oca7233/SHARED/boost-1.73.0/lib/libboost_filesystem.so.1.73.0 /panfs/panasas/cnt0026/oca7233/SHARED/boost-1.73.0/lib/libboost_system.so.1.73.0 /panfs/panasas/cnt0026/oca7233/SHARED/boost-1.73.0/lib/libboost_regex.so.1.73.0 /panfs/panasas/cnt0026/oca7233/SHARED/hdf5-1.12.0-1-intel-2019.4/lib/libhdf5_cpp.so /panfs/panasas/cnt0026/oca7233/SHARED/hdf5-1.12.0-1-intel-2019.4/lib/libhdf5.so /usr/lib64/libz.so /usr/lib64/libdl.so /usr/lib64/libm.so -lm ../symba7/libsymba7seq.so ../swift/libswift.so ../../external/kokkos/containers/src/libkokkoscontainers.so.3.6.00 ../../external/kokkos/core/src/libkokkoscore.so.3.6.00 /usr/lib64/libdl.so

If I do not link with libpykokkos-core, the python test script will fail, as it won't find the Kokkos views binding.

Am I doing something wrong ?

Thx

Compilation error in libpykokkos.hpp

Compiler version: g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

Compilation of libpykokkos.hpp fails with the following error:

libpykokkos.hpp:278:32: error: ‘array’ in namespace ‘std’ does not name a template type
           typename RetT = std::array<size_t, sizeof...(Idx)>>

A long list of errors also follows. I fixed it by adding #include <array> at the top of libpykokkos.hpp. I can open a PR if needed.

Cupy/Numpy arrays for CudaSpace views

Is there a way to get a cupy or numpy array from a CudaSpace view? This code results in a segfault when I try to create the cupy array. I also tried variations with numpy arrays and copy=True, but that didn't work either.

import cupy as cp
import kokkos

def test():
    view = kokkos.array("view", [5], dtype=kokkos.double, space=kokkos.CudaSpace)
    arr = cp.array(view, copy=False)

if __name__ == "__main__":
    kokkos.initialize()
    test()
    kokkos.finalize()

Python version: 3.8.3
Numpy version: 1.18.5
Cupy version: 7.8.0
CUDA version: 11.0
I also built the project with -DKokkos_ENABLE_CUDA=ON

Query: compilation time for unity_dynamic_Left_cxx.cxx.o

I do see some notes in the README about slow compilation times, but I'm getting 1 hour, 48 minute build times for:

python setup.py install -- -DENABLE_INTERNAL_KOKKOS=ON -DENABLE_LAYOUTS=ON -DENABLE_MEMORY_TRAITS=OFF -DKokkos_ARCH_PASCAL61=ON -DKokkos_ENABLE_OPENMP=ON -DKokkos_ENABLE_CUDA=ON -DBUILD_SHARED_LIBS=ON

Mostly gets stuck here:

[32/36] Building CXX object src/variants/CMakeFiles/libpykokkos-variants.dir/Unity/unity_dynamic_Left_cxx.cxx.o

Is this normal? ENABLE_MEMORY_TRAITS is OFF, which is the main slow one mentioned in the docs. In general, this has been the hardest piece to install for me so far (Kokkos/pykokkos were not too bad). Working on Ubuntu Linux for now, which is the only "official" platform mentioned in pykokkos README.

Defaulting layout to Right versus following Kokkos' default

Is there a reason layouts are defaulted to LayoutRight for kokkos.arrays instead of being exec space preference based like native Kokkos?

Currently, if one creates a CudaSpace array without specifying LayoutLeft, a LayoutRight array is created, and the predictable performance hit is experienced. We could add some simple logic to the array creation like is already done if a numpy array is passed. Or better yet just access the space's preferred layout straight from kokkos source (nor sure if that's feasible or not). I guess at this point its more or less as simple as GPU <-> Left CPU <-> Right so its not too tedious to just specify on the python side.

Unmanaged Views from Numpy Arrays

@jrmadsen I extended the bindings by adding support for unmanaged views that can be initialized from numpy arrays. This would help avoid allocating memory twice and copying the data. Would you be interested in a pull request?

Error compiling with CUDA 11.0

I have been using pykokkos-base successfully on summit with CUDA 11.4, however due to other issues (unrelated to pykokkos-base) I am reverting to CUDA 11.0.

When I attempt to compile with CUDA 11.0 I get the following compiler errors:

[  9%] Building CXX object CMakeFiles/libpykokkos-core.dir/src/enumeration.cpp.o
pykokkos-base/src/enumeration.cpp(188): error: incomplete type is not allowed

pykokkos-base/src/enumeration.cpp(187): error: incomplete type is not allowed

pykokkos-base/src/enumeration.cpp(192): error: incomplete type is not allowed

pykokkos-base/src/enumeration.cpp(191): error: incomplete type is not allowed

and

kokkos/include/Kokkos_Concepts.hpp(410): error: static assertion failed with "template arguments must be memory spaces"
          detected during:
            instantiation of class "Kokkos::Impl::MemorySpaceAccess<DstMemorySpace, SrcMemorySpace> [with DstMemorySpace=Kokkos::HostSpace, SrcMemorySpace=Kokkos::CudaSpace]" 
pykokkos-base/src/available.cpp(101): here
            instantiation of class "memory_space_access<Tp, true> [with Tp=Kokkos::CudaSpace]" 
pykokkos-base/src/available.cpp(122): here
            instantiation of "auto get_host_accessible<SpecT,Idx,Tail...,<unnamed>>(size_t, std::index_sequence<Idx, Tail...>) [with SpecT=MemorySpaceSpecialization, Idx=1UL, Tail=<2UL, 3UL, 4UL, 5UL, 6UL, 7UL, 8UL, 9UL>, <unnamed>=0]" 

... and more of the same

Using gcc/9.3.0, Kokkos 3.5.00

Support for atomic views

Would it be possible to add support for atomic views? Something like this:

view = kokkos.array("view", [10], dtype=kokkos.double, space=kokkos.HostSpace, atomic=True)

Error in CMakeLists.txt include of CheckCXXSourceCompiles

cmake version: 3.18.2

Running

mkdir build
cd build
cmake -DBUILD_EXAMPLES=ON -DKokkos_DIR=$path_to_install/lib/cmake/Kokkos ..

gives the following error

  CMake Error at CMakeLists.txt:8 (INCLUDE):
  INCLUDE could not find load file:

    CheckCxxSourceCompiles

I fixed this by replacing CheckCxxSourceCompiles with CheckCXXSourceCompiles (capitalizing xx) on line 8 in CMakeLists.txt. I can open a pull request if needed.

BLD: providing binary wheels on PyPI

@namehta4 @NaderAlAwar

Since pykokkos-base is not actually needed to build pykokkos (distinction of genuine build time vs. runtime dependencies), and because PEP518-based pip installs will build pykokkos on its own in an isolated env before installing it to a local user env, regardless of how we install pykokkos with pip, it will still be up to the user to provide a suitable version of pykokkos-base in their environment (the same would apply for providing a suitable version of NumPy when working with SciPy for example).

So, in the pip/PyPI ecosystem, I suspect the only way for us to reduce build/install friction is to:

• make it easier install pykokkos-base from source (but building from source is never really "easy" for end users)
• providing binary wheels on PyPI using i.e., https://cibuildwheel.readthedocs.io/en/stable/

The latter would likely be a substantial lift, and I'm not sure how we'd handle OMP, CUDA backend library shipping, though some libs like pytorch or tensorflow could likely be used as inspiration for that.

MAINT: python 3.11 support

Python 3.11 RC1 is guaranteed to be ABI stable, so it is safe to start thinking/testing for compatibility there.

Currently, I see some pybind11-related failures like this during pykokkos-base build:

| [ 75%] Building CXX object CMakeFiles/libpykokkos-core.dir/Unity/unity_1_cxx.cxx.o
| In file included from /tmp/pykokkos-base/external/pybind11/include/pybind11/cast.h:16,
|                  from /tmp/pykokkos-base/external/pybind11/include/pybind11/attr.h:13,
|                  from /tmp/pykokkos-base/external/pybind11/include/pybind11/pybind11.h:45,
|                  from /tmp/pykokkos-base/include/common.hpp:54,
|                  from /tmp/pykokkos-base/include/traits.hpp:47,
|                  from /tmp/pykokkos-base/_skbuild/linux-x86_64-3.11/cmake-build/src/variants/views.cpp:3,
|                  from /tmp/pykokkos-base/_skbuild/linux-x86_64-3.11/cmake-build/CMakeFiles/libpykokkos-core.dir/Unity/unity_1_cxx.cxx:3:
| /tmp/pykokkos-base/external/pybind11/include/pybind11/detail/type_caster_base.h: In function ‘std::string pybind11::detail::error_string()’:
| /tmp/pykokkos-base/external/pybind11/include/pybind11/detail/type_caster_base.h:448:36: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
|   448 |                 "  " + handle(frame->f_code->co_filename).cast<std::string>() +
|       |                                

If you want to reproduce you can i.e., modify the pykokkos repo CI:

--- a/.github/workflows/main_ci.yml
+++ b/.github/workflows/main_ci.yml
@@ -13,7 +13,7 @@ jobs:
     strategy:
       matrix:
         platform: [ubuntu-latest]
-        python-version: ["3.8", "3.9", "3.10"]
+        python-version: ["3.11.0-rc.1"]
     runs-on: ${{ matrix.platform }}
     steps:
       - uses: actions/checkout@v3

and run act -j test_pykokkos

DEV: allow non-squash merges

Looks like the repo doesn't allow individual commits in a PR branch to be preserved on merging (see below). The debate around this stuff isn't super interesting, but I'd argue that leaving the decision about whether individual commits in a branch merit preservation is best left to the reviewer/merger, rather than having a hard-block at the repo level. Sometimes individual commits are complex and have well-written commit messages, and could be productively revisited later on, etc.

Also, pykokkos repo doesn't have this restriction, FWIW.