nek5000/nekrs

our next generation fast and scalable CFD code

                 __    ____  _____
   ____   ___   / /__ / __ \/ ___/
  / __ \ / _ \ / //_// /_/ /\__ \ 
 / / / //  __// ,<  / _, _/___/ / 
/_/ /_/ \___//_/|_|/_/ |_|/____/  
COPYRIGHT (c) 2019-2023 UCHICAGO ARGONNE, LLC

nekRS is a fast and scaleable computational fluid dynamics (CFD) solver targeting HPC applications. The code started as an early fork of libParanumal in 2019.

Capabilities:

• Incompressible and low Mach-number Navier-Stokes + scalar transport
• High-order curvilinear conformal spectral elements in space
• Variable time step 2nd/3rd order semi-implicit time integration
• MPI + OCCA (backends: CUDA, HIP, OPENCL, SERIAL/C++)
• LES and RANS turbulence models
• Arbitrary-Lagrangian-Eulerian moving mesh
• Lagrangian phase model
• Overlapping overset grids
• Conjugate fluid-solid heat transfer
• Various boundary conditions
• VisIt & Paraview support for data analysis and visualization
• Legacy interface to Nek5000

Build Instructions

Requirements:

• Linux, Mac OS X (Microsoft WSL and Windows is not supported)
• C++17/C99 compatible compiler
• GNU/Intel/NVHPC Fortran compiler
• MPI-3.1 or later
• CMake version 3.18 or later

Download the latest release available under

https://github.com/Nek5000/nekRS/archive/refs/tags/v23.0.tar.gz 

or clone our GitHub repository:

https://github.com/Nek5000/nekRS.git

The master branch always points to the latest stable release while next provides an early preview of the next upcoming release (do not use in a production environment).

To build and install the code run:

CC=mpicc CXX=mpic++ FC=mpif77 ./nrsconfig [-DCMAKE_INSTALL_PREFIX=$HOME/.local/nekrs]

Build settings can be customized through CMake options passed to nrsconfig. Please remove the previous build and installation directory in case of an update.

Setting the Enviroment

Assuming you run bash and your install directory is $HOME/.local/nekrs, add the following line to your $HOME/.bash_profile:

export NEKRS_HOME=$HOME/.local/nekrs
export PATH=$NEKRS_HOME/bin:$PATH

then type source $HOME/.bash_profile in the current terminal window.

Run the Code

We try hard not to break userland but the code is evolving quickly so things might change from one version to another without being backward compatible. Please consult RELEASE.md before using the code.

cd $NEKRS_HOME/examples/turbPipePeriodic
mpirun -np 2 nekrs --setup turbPipe.par

For convenience we provide various launch scripts in the bin directory.

Documentation

For documentation, see our readthedocs page. For now it's just a dummy. We hope to improve it soon.

Discussion Group

Please visit GitHub Discussions. Here we help, find solutions, share ideas, and follow discussions.

Contributing

Our project is hosted on GitHub. To learn how to contribute, see CONTRIBUTING.md.

Reporting Bugs

All bugs are reported and tracked through Issues. If you are having trouble installing the code or getting your case to run properly, you should first vist our discussion group.

License

nekRS is released under the BSD 3-clause license (see LICENSE file). All new contributions must be made under the BSD 3-clause license.

Citing nekRS

NekRS, a GPU-Accelerated Spectral Element Navier-Stokes Solver

Acknowledgment

This research was supported by the Exascale Computing Project (17-SC-20-SC), a joint project of the U.S. Department of Energy's Office of Science and National Nuclear Security Administration, responsible for delivering a capable exascale ecosystem, including software, applications, and hardware technology, to support the nation's exascale computing imperative.

nekrs's Issues

Add stressFormulation

Adjust dt based on CFL + initial guess

• add dt = targetCFL(+max=...) to par
• add targetCFL key to par

For reference:
https://github.com/Nek5000/Nek5000/blob/master/core/subs1.f#L178

Use env-vars instead for paths instead of pre-processor defines

Run GPU CI

set boundary conditions in par file

Add asynchronous coarse grid solve option to parAlmond

Add timers + runtime statistics

Make mesh->boundaryInfo local

par KEYS are not validated

Switch to extended BDF/EXT featuring a larger stability area

Add temperature mesh struct (CHT)

Add support for HIP

Add linAlg Class

This can include libP's linAlg (available in staging) and our own operations.

Support up to 2^31 elements

Nek5000/Nek5000#504

Build coarse matrix (N=1) without creating the mesh explicitly

Avoid using system()

Currently our jit-compilation relies on system(). This can be troublesome (see e.g. #166) and lead to an undefined behaviour.

Updates:
d39e9d3: This commit ensures that we don't call system() provided that the user runs nrspre first.

1. Cross compilation
   Our current --build-only will not work in a cross compiling context using a different front end (compiler, CPU-arch, GPU etc). In this case one has precompile on the compute node provided this is supported.

2. OCCA relies on system() to jit-compile kernels. Using nrspre will precompile all required kernels. The actual will then load the kernels from cache. Another option (not available yet) would be to switch a true runtime compilation (e.g. NVRTC for Cuda). However this might be not supported for all backends. Moreover it doesn't allow caching.

Name change trigger rebuild of ins kernels

Add variable coeff elliptic solver

Add low-Mach support

Add to Spack

Add Valgrind CI test

Add scalar (temperature) transport

• merge CDS PR into our libP next branch

• update to latest BC treatment
• add tolerance and diffusitivty to cds struct
• add pointer to cds struct to ins struct

• Integrate CDS solver into nekRS

• add our own insSetup() and clean it up
• add CDS to nekRS build system
• pass number of scalars to mkSIZE
• set nek's ifheat and npscal
• read nek's boundaryID into mesh->EtoB
• read in insSetup nek's cbc (for field 1 and 2) into ins->EtoB and cds->EtoB
• add scalar support to nek_copyTo() and nek_copyFrom ()
• call cdsSetup in nekrs.cpp:main
• call cdsSolveStep in timerstepper:runPlan4

• BC Treatment

• Introduce ins->EtoB and cds->EtoB
• Support Dirichlet + non-zero Neumann (cdsDirichletCondition/cdsNeumannCondition)
• Add sM and sP to bcData struct
• velocity value shall be accessible in cdsDirichletCondition and cdsNeumannCondition
• scalar value shall be accessible in insDirichletCondition and insNeumannCondition

• Support user specified source term

• add scalarSource function pointer to udf sturct
• call function pointer in timerstepper:runPlan4

• Extend par reader

• read tolerance and diffusivity and set correspondig internal libP parameter

• Add RBC example

Add RBC example

Add HYPRE GPU option

Merge libP's staging branch

Add communication interface to platform

• MPI
• gslib's sarray_transfer

Eliminate all global + o(P) storage

There is some libP legacy during the setup phase:

Non-scaleable communication:

• non-scaleable MPI_Allgather(...)
• non-scaleable MPI_AllToAll(ellipticBuildContinuousHex3D, meshParallelConnect)

Non-scaleable data:

• mesh->boundaryInfo holds (face vertex ids, boundaryID) for the whole mesh

Check for soname field when linking to a shared lib

Add block elliptic solver

Use single memCpy + gs for velocity instead of every component

coarse grid solve is diverging

We have observed a few cases where HYPRE's BoomerAMG is diverging.

Update to new libP version (staging branch)

• switch to new bcData struct for all elements types
• disable TOMBO support for tris, tets
• remove makefile changes (except if they are HYPRE related)
• don't build HYPRE by default
• merge our libP changes into libP's experimental branch
• port ins to staging (Noel)
• update nekRS to staging

Reduce memory footprint

Rename boundary device functions

To support 2D and 3D at the same time:
insVelocityDirichletConditions3D -> insVelocityDirichletConditions

Switch to our own ins solver

• TOMBO + HEX only
• including okl

Add torder=3

Pick up wrong occa if user has a conflicting LD_LIBRARY_PATH

Encapsulate libP

Encapsulate into a namespace and load needed function dynamically at runtime.

Current dependencies:

• mesh struct
• setupAide struct
• elliptic struct

Add velocity vector solve

Incorrect ordering of C & Fortran stdout

Currently I cannot think of a way to sync them.

Add support for variable properties

Read default settings from config file

Add residual projection

Make it work with openCL

Add SP MG preconditioner version

Add additive multilevel Schwarz (AMS) preconditioner

Add mesh->EToB to bcData struct

Add infrastructure to support different kernel versions

Improve serial (CPU) performance

Compute node data online instead of using a file