arcaneframework/framework

The Arcane Framework for HPC codes

Written by CEA/IFPEN and Contributors

(C) Copyright 2000-2024 CEA/IFPEN. All rights reserved.

All content is the property of the respective authors or their employers.

For more information regarding authorship of content, please consult the listed source code repository logs.

---

Development platform for unstructured 2D or 3D parallel computing codes.

Arcane is a development environment for parallel numerical calculation codes. It supports the architectural aspects of a calculation code, such as data structures for meshing and parallelism, as well as more environment-related aspects such as dataset configuration.

---

Table of Contents:

• Documentation
  • User documentation
  • Developer documentation
  • Alien documentation
• Changelog
• Key features
• Getting started
  • Compiling and/or installing Arcane
    • Docker images
    • Spack
• Examples of how to use the Arcane Framework
  • Arcane Benchs
  • Sharc
  • ArcaneFem
  • MaHyCo
• Rencontres Arcanes
• Screenshots

Documentation

The documentation is available online and is generated and stored in this GitHub repository.

User documentation

This documentation is intended for Arcane users.

• User documentation (French)
• User documentation (English) (Soon)

Developer documentation

This documentation is intended for Arcane developers.

• Developer documentation (French)
• Developer documentation (English) (Soon)

Alien documentation

This documentation is intended for Alien users and developers.

• Alien documentation

Changelog

The changelog is available in the documentation or in this repository.

Key features

Massively parallel

• Work on simple laptop or on a supercomputer (run case on more than 100k CPU core)
• Domain partitioning with message-passing (MPI, Shared Memory or hydrid MPI/Shared Memory)
• Unified Accelerator API (experimental)
  • CUDA
  • ROCM
  • SYCL (experimental)
  • oneTBB (experimental)
• Basic support for explicit vectorization
• Automatic Load Balancing with cell migration with several mesh partitioner
  • ParMetis
  • PTScotch
  • Zoltan

I/O

• Supported input mesh file type:
  • VTK Legacy (2.0 and 4.2)
  • VTK VTU
  • Lima
  • MED
  • xmf
  • Gmsh (4.1)
• Post-processing with the following supported output file type:
  • VTKHDF (V1 and V2)
  • Ensight7Gold
• Time history curves
• Support for checkpoint/restarting for long simulation

Mesh entities

• Multiple mesh entities (items) are usable in Arcane:
  • Node (0D), Edge (1D), Face and Cells
  • Particle
  • DoF (Degree of freedom)
• Full Connectivities between items (cell to node, node to edge, ...)
• Support for user connectivities
• Easy-to-use Arcane mesh variables on any kind of item
  • double, Int32, Int64, ..
  • Scalar, 1D, 2D or multi-dim
  • and many more...
• Several kind of meshes are supported (1D, 2D and 3D)
  • unstructured
  • unstructured with adaptative refinement
  • cartesian (experimental)
  • cartesian with patch refinement (AMR) (experimental)
  • polyedral mesh (any number of edges/faces per cell)
• Meshes are fully dynamic (adding/removing cells)
• All the connectivities and variables are usable on accelerators

Multi-constituents support

• Two levels of constituents (environment and materials)
• Any cell variable may have values par constituant

Performance and Verification and Validation

• Bit-to-bit comparison for Arcane variables
• Between-synchronizations comparing
• Unit test system integration
• Automatic profiling of loops using accelerator API
• Automatic profiling with sampling using Papi library or signal using SIGPROF
• Automatic use of CUDA CUPTI library to track unified memory (USM) moves between host and device

Decoupled and extensible framework

• Handling of case options via a descriptor file based on XML (axl files)
• Automatic generation of documentation
• Notion of Service (plugin) with separated interface/implementation to extend functionalities
• Notion of independant Modules to enable modularity
• Limited usage of C++ templates to make extensibility easier

Extensions to other langages

• Most of the classes are available in C#
• Python wrapping (work in progress, available at end of 2024)

Algebraic manipulation

• Use of Alien library to handle linear systems
• Coupling with several linear solver libraries
  • Hypre
  • Trilinos
  • PETSc

Other functionalities

• Standalone mode to use only some functionalities of Arcane
  • mesh connectivities
  • accelerator API
• Handling of time loop with automatic support to go back to a previous iteration

Getting started

Compiling and/or installing Arcane

To compile the Arcane Framework, you need an x86 or ARM64 CPU, Linux OS or Windows OS and a C++ compiler with a version that supports C++20 or higher.

For all other dependencies and more information, check out the Compiling/Installing guide.

Click here if the documentation is not available

To prepare your computer :

• Ubuntu 24.04
• Ubuntu 22.04 and CMake instructions
• Ubuntu 20.04 and CMake instructions
• AlmaLinux/RedHat 9 and CMake instructions
• ArchLinux based

To build and install Arcane:

• Compiling/Installing

Docker images

Docker images with the Arcane framework installed are available in this GitHub repository. More information here.

Spack

Spack recipes for Arcane are available here.

Examples of how to use the Arcane Framework

An introductory chapter with the construction of a Hello world is available here.

Examples of applications using Arcane are available on GitHub. Here is a non-exhaustive list:

Arcane Benchs

A set of mini-applications to evaluate Arcane functionalities. These are a good basis for getting started with Arcane.

Sharc

Arcane-based application for solving different geosciences problems.

ArcaneFem

Very simple codes to test Finite Element Methods using Arcane.

MaHyCo

TODO

Rencontres Arcanes

The next Rencontres Arcane are scheduled for 2025.

Les Rencontres Arcane on April 17th 2023.

The previous presentations are stored here.

Screenshots

Transient elastodynamics with ArcaneFEM

More Screenshots from ArcaneFEM

Aerodynamics

Bilaplacian

Linear elasticity

Electrostatics

Solving Fourier equation

Solving Laplace equation

Solving Poisson equation

Water concentration in a porous material with Sharc

More Screenshots from Sharc

Evolution of water concentration over time in a porous material

framework's Issues

Improve MicroHydro bench

The script extras/benchs/bench_microhydro.py allows to execute the MicroHydro module in arcane_tests_exec.
We need to do the following tasks:

• Move the source code outside of 'src/arcan/tests'. This way we will be able to launch this test outside of Arcane build.
• Simulate more variables for load-balancing to have more realistic messages.
• Simulate imbalance in some part to test mesh partitioning algorithms.

Remove usage of ItemTypeMng::singleton()

An instance of ItemTypeMng should be tied to an instance of IMesh. Add a method IMesh::itemTypeMng() to get the associated ItemTypeMng.

Separate data allocation from metadata in Array classes.

In the class Arccore::AbstractArray, there is only one allocation which contains both data and metadata (size, capacity, ...). There metadata are in the class ArrayImplBase.

We need to separate these two parts if we want to be able to use specific memory for the data.

Add support for node duplication in AMR

Currently when we refine a cell we only create nodes if needed.
This issue is to implement a mode where we can duplicate nodes.

Duplication allows numerical schemes to have different values for nodes or faces of different levels.

For example, at the moment, if we have a cell like this at level 0

Level 0
0---------1                
|         |
|         |
|         |
3---------2

Without duplication

Level 1

0----4----1                
|    |    |
7----8----5
|    |    |
3----6----2

With duplication

Level 1

9----4----10                
|    |    |
7----8----5
|    |    |
12---6----11

Support for creating Faces and/or Edges during mesh allocation

Currently IPrimaryMesh::allocateCells() does not allow to specify Face or Edge. These items are implicy created when we allocate cells. It works well but it has one drawback: we need to add a pass to renumber uniqueId() for these items. With big meshes (100 million+ cells) that phase may take a long time.

At the moment big meshes are always cartesian meshes. It is easy for the CartesianMeshGenerator to generate uniqueId() for faces or edges. Using IMeshModifier::addFaces() or IMeshModifier::addEdges() for example it should be possible to create faces or edges at the same time than cells.

Add CUDA platforms and tests to CI

The tool circle-ci(https://app.circleci.com) is currently used for ubuntu 20.04 compilation and check. CI runners with CUDA are available. We may use them to do our GPU test. If there are not free, we need to have runners somewhere to do this task (maybe at CEA?)

Same instance of singleton services when using shared memory for sub-domains

When we launch a binary with option -A,S=..., it seems there is only one instance shared by all sub-domains for a singleton service. The correct behavior is to have one instance per sub-domain.

FindHDF5 does not work properly with static libraries

According to https://alexreinking.com/blog/building-a-dual-shared-and-static-library-with-cmake.html, find_package should honor BUILD_SHARED_LIBS.

In the following test, https://github.com/arcaneframework/framework-ci/runs/4027731468?check_suite_focus=true , Arccon's findHDF5 finds a static library, and it does not work with Alien's dynamic library targets.

Other question: can Arccore work as static library ?

Remove unused parts of Arccon

It seems some parts of Arccon are no longer used. It may be useful to delete them.

I identified the following directories but there may be others:

• build-system/csharp
• build-system/dynamicloading
• build-system/languages
• build-system/bin
• some commands in build-system/commands

These parts are not used at CEA. @stdcm and @gratienj you may know if there are still used at IFPEN

Add support for non blocking collective operation in IMessagePassingMng interface

At the moment the non blocking collective operation are only available in Arcane::IParallelMng

Deploy CI using 'vcpkg' to this repository

Actions for CI are in framework-cirepository. We need to move these actions in this repository and run them when to validate pull requests.

Add more compilers and platforms for CI

At the moment, we have tests only for GCC 8, GCC 9 on Linux. At the minimum we need to add these configurations:

• GCC 11 with C++17 and C++20
• Clang 13 with C++17 and C++20

_clang_format is not valid

Clion inspection tool check _clang_format syntax against a schema validation and several statements do not match requirements.

Might be dependent of clang-format version used.

Improve MPI implementation of serialisation to handle unreliable machines

The current MPI implementation of serialize message does the following this:

• send the message in one MPI call if its size is small (by defaut 5000 ko)
• send the message in two MPI calls if it's not the case. The first message contains the total size and the second message is the full message.

Some MPI implementation may have (temporary?) problems when there are too many or too big messages.

To solve this problem, we can try several fix:

• send the message with multiple packets whose size is fixed
• do not send the full message if the corresponding receive message is not posted.

Evaluate circleci for CI

For open source and public github repo, CircleCI provide virtual machine with up to 16 CPU, aarch64 and NVIDIA GPU.
The goal is to develop a CI workflow for this tool.

Missing relative path for headers in doxygen documentation

In the generated documentation, the relative path is missing. For example in the following documentation:

The path should be arcane/geometric/BarycentricGeomShapeComputer.h

Handle DLL copy for tests on Windows

At the moment on windows the needed DLLs for executables are copied in the executable directory only for libraries managed by vcpkg. We need to modifiy the build to copy all the DLLs. CMake version 3.21 has a new variable TARGET_RUNTIME_DLLS to know which DLLs are needed for a target.
See: https://cmake.org/cmake/help/latest/manual/cmake-generator-expressions.7.html#genex:TARGET_RUNTIME_DLLS

Add CI for Windows 2022

[arccore] Cleanup array view classes and update to C++17

Refactor ArrayView, ConstArrayView and Span:

• Use common implementation for functions like subArrayInterval() or dumpArray().
• Use constexpr and noexcept if possible
• Add constructor or converter from std::array
• Add tests for subViewInterval()

Support for unstructured mesh without creation of Face items

With the new implementation of connectivities, it is possible to make Faces creation optional. It should be similar to the way Edges are handled.

Add support for VTK Legacy Mesh format 4.2

At the moment only the version 2.0 of this format is supported

Documentation:
https://kitware.github.io/vtk-examples/site/VTKFileFormats/
https://vtk.org/doc/nightly/html/IOLegacyInformationFormat.html

Add namespace for Arcane target in CMake config files

Prefix Arcane targets with Arcane:: in CMake config files.

We need to keep current target names for compatibility: use arcane_core or Arcane::arcane_core.

Migration of IGraph to GraphOnDoF (Graph implementation based on DoF families and ItemFamilyNetwork

Remove direct creation of Array and Array2

There are still parts of code where we directly build instance of Array or Array2:

• in Array::clone()
• in Array2::clone()
• in CaseOptionMultiSimpleT because we derive from ArrayView<Array<T>>.
• in constructors of Array2 which are public.
• change the return type of the generated get* functions in axl header.

These methods has to be removed.

Remove support for Glib2 version older than 2.32

Arcane no longer support Linux distributions using versions of Glib2 older than 2.32.
Removing this support allows to remove deprecated symbols concerning thread usage in Glib2 (thread, mutex, cond, ...)

Add support to N-dimensions loops in the TBB backend

At the moment, the TBB backend use multi-threading only for the outermost loop.
The goal is to add support for 2D and 3D, like the collapse(2) or collapse(3) arguments of the OpenMP parallel for.
Demand from @DavidDureau

Display memory usage for computed ItemGroup

Add a way to compute and display memory usage for ItemGroup which are dynamically allocated (for exemple allCells().own() or allCells().nodes()).

Artifacts for new versions

Spack recipes for Arcane, Arccon, Arccore and Axlstar (https://https://github.com/arcaneframework/spack_recipes) require that released version tarballs are properly available.

That is not currently the case.
#51 was an attempt to automatically do this, but it seems the feature is already present but has not been used yet. So, please, when upgrading any version files, also upload artifacts, and if possible update spack recipes.

Not knowing new releases of Arccon and Arccore is also annoying for Alien, that should embed the last released versions of Arccon and Arccore.

Add coverity checks to CI

Specific mesh partitioner for Cartesian Mesh

A specific partitioner is needed to make sure mesh parts are also a cartesian mesh.

Add support to compiler cache tool in the CI to speed up compilation

Link to sccache tool: https://github.com/mozilla/sccache.
This tool works on windows with the Microsoft Compiler MSVC (ccache does not support this case).
There is a fork of ccache which works on windows: https://github.com/cristianadam/ccache.git with branch msvc

Add a mode in MeshMaterialBackup to reduce memory usage

To reduce memory usage, the following development may be done:

• store values on disk (or in a temporary database)
• compress values in memory

Arcane does not compile with glibc >= 2.34 (Ubuntu 21.10)

__malloc_hooks have been deprecated for a long time and are now not explicitly available in malloc.h.

framework/arcane/src/arcane/utils/MemoryInfo.cc

Line 128 in 45953e9

__malloc_hook = old_malloc_hook;

From glibc 2.34 changelog:

• The deprecated memory allocation hooks __malloc_hook, __realloc_hook,
  __memalign_hook and __free_hook are now removed from the API. Compatibility
  symbols are present to support legacy programs but new applications can no
  longer link to these symbols. These hooks no longer have any effect on glibc
  functionality. The malloc debugging DSO libc_malloc_debug.so currently
  supports hooks and can be preloaded to get this functionality back for older
  programs. However this is a transitional measure and may be removed in a
  future release of the GNU C Library. Users may port away from these hooks by
  writing and preloading their own malloc interposition library.

Command line option 'MessagePassingService' is not read when MPI is enabled

IItemFamily::localConnectivityInfos() and IItemFamily::globalConnectivityInfos() have invalid values when using new connectivities

The values for these methods IItemFamily::localConnectivityInfos() and IItemFamily::globalConnectivityInfos() were calculated using legacy connectivity information. Now that legacy connectivity are gone, most of these values are not recomputed. The values for IItemConnectivityInfo::maxNodePerItem() seems to be valid.

Improve support for mesh services

To help users switch to mesh services instead of legacy mesh handling, we need to :

• make a documentation of the new usage
• add an implementation of IMeshBuilder for Lima reader
• add an implementation of IMeshBuilder for GMSH reader
• add a converter from legacy mesh building to new usage
• handle meshes partitioned with an external tool (files with pattern CPU*)

Add LLNL/Umpire backend for IMemoryRessourceMng

LLNL/Umpire (https://github.com/LLNL/Umpire) provides interesting features and optimizations : NUMA management, Memory pool, ...

It may be interesting to test and benchmark these features.

Add a mesh partitioner to map an unstructured mesh on a cartesian grid

Add helper class to generate mesh

This class will manage parts which are common to several mesh builder. For example:

• handle node coordinates
• simplify adding cell

This class will also add the possibility to directly create edges and faces if the mesh builder can specify theirs uniqueId().

May use #91

[arcane] Upgrade to CMake version 3.21

For windows build, we need recent versions of CMake:

• 3.20 to allow CMake configure_file() command to use source permission
• 3.21 to have TARGET_RUNTIME_DLLS generator expression needed to copy used DLLs in the same directory than Arcane libraries are installed

This task will be done in two phases:

• Upgrade to 3.18 on Unix and 3.21 on Windows (#40). This is needed because some Linux platforms used by Arcane do not yet have version 3.21 of CMake.
• Upgrade to 3.21 everywhere

Upgrade 'vcpkg' packages for CI

Move classes relative to cartesian mesh in an 'arcane_cartesianmesh' component

These classes are currently in arcane/ceapart/src/arcane/cea

Add support to run tests in parallel

To optimize ressources, we should be able to run multiple sequential tests at the same time using option -j from cmake.
Need to do the following tasks:

• ensure each test runs in a separate directory.

Add workflow in CI to check file encoding and header

We need to check every C++ file has:

• the correct encoding (UTF-8)
• the correct BOM
• the correct header with licence and emacs variables (-- tab-width: 2; indent-tabs-mode: nil; coding: utf-8-with-signature --)

Add more statistics during load balancing

To help debugging some cases, it may be useful to have some statistics during load balancing. For example:

• Memory usage
• Number of items and variables to transfer

Remove memory allocation for Item->Node connectivity with legacy connectivities

Currently ItemSharedInfo always allocate memory for nodes based on ItemTypeInfo::nbLocalNode(). These connectivities are no longer used so we can remove them. But doing that with the current version of Arcane implies incompatibilities with checkpoints:

• The last version of Arcane add version information in checkpoint (done in 20b603c)
• When everybody will use that version, remove node connectivity allocation.

Upgrade to '.Net 6'

Net6 is the new LTS version of .Net. The current LTS version is .Net 3.1 and its End Of Life is december 2022.

Add MSH reader in library 'arcane_std'

At the moment the MSH reader is in the library 'arcane_ios' which is not in the default library set loaded by Arcane.
We want to move the reader and the writer in arcane/std directory.

[arcane] SEGV when updating materials variables if the global variable is not used.

This Segmentation Violation occurs when IVariable::isUsed() is false for the global variable associated to a IMeshMaterialVariable.

Add Array implementation with stack allocation

For temporary and small array, it may be useful to allocate them on the stack instead of using the heap.

Add a StackArray class which used Array implementation with a specific allocator to use the stack if the size of the array is less than a specified size.

Need #45