shefmarkh/celeritas

Celeritas

The Celeritas project implements HEP detector physics on GPU accelerator hardware with the ultimate goal of supporting the massive computational requirements of LHC-HL upgrade.

Installation and development

This project requires third-party libraries (TPLs) to build a fully-featured code. However, some of these requirements can be omitted to enable testing on personal machines with fewer available components.

• ROOT: for I/O
• an MPI implementation (such as OpenMPI): for shared memory parallelism
• VecGeom: on-device navigation of GDML or ROOT-defined detector geometry
• CUDA: on-device computation.

Development/testing requirements:

• CMake (build system)
• Git-LFS (large test files, binary documentation)
• LLVM/Clang (formatting enforcement)

Installing with Spack

Spack is an HPC-oriented package manager that includes numerous scientific packages, including those used in HEP. An included Spack "environment" (at scripts/dev/env/celeritas-{platform}.yaml) defines the required prerequisites for this project.

The script at scripts/dev/install-spack.sh provides a "one-button solution" to installing and activating the Spack prerequisites for building Celeritas. Alternatively, you can manually perform the following steps:

• Clone Spack following its getting started instructions
• Add CUDA to your $SPACK_ROOT/etc/spack/packages.yaml file
• Run spack env add celeritas scripts/dev/env/celeritas-linux.yaml (or replace linux with darwin if running on a mac); then spack -e celeritas concretize and spack -e celeritas install
• Run and add to your startup environment profile spack env activate celeritas
• Configure Celeritas by creating a build directory and running CMake (or ccmake for an interactive prompt for configuring options).

An example file for a packages.yaml that defines an externally installed CUDA on a system with an NVIDIA GPU that has architecture capability 3.5 is thus:

packages:
  cuda:
    paths:
      [email protected]: /usr/local/cuda-10.2
    buildable: False
  all:
    variants: cuda_arch=35

Configuring and building Celeritas

Example build scripts are available in scripts/build; an example CMake command looks like:

cmake  \
  -D CELERITAS_USE_CUDA=ON \
  -D CELERITAS_USE_MPI=OFF \
  -D CELERITAS_USE_VECGEOM=ON \
  -D CMAKE_BUILD_TYPE="RelWithDebInfo" \
  -D CMAKE_CXX_FLAGS="-Wall -Wextra -pedantic -Werror" \
  -D CMAKE_CUDA_FLAGS="-arch=sm_80" \
  ${SOURCE_DIR}

Commit hooks

Run scripts/dev/install-commit-hooks.sh to install a git post-commit hook that will amend each commit with clang-format updates if necessary.

More details

See the development wiki page for guidelines and best practices for code in the project.

The code design page outlines the basic physics design philosophy and classes.

Some codebase images and graphs are available on the celeritas-docs repo.