gridtools/gtc

A stencil DSL toolchain written in Python.

The GridTools framework is a set of libraries and utilities to develop performance portable applications in the area of weather and climate. To achieve the goal of performance portability, the user-code is written in a generic form which is then optimized for a given architecture at compile-time. The core of GridTools is the stencil composition module which implements a DSL embedded in C++ for stencils and stencil-like patterns. Further, GridTools provides modules for halo exchanges, boundary conditions, data management and bindings to C and Fortran.

GridTools is successfully used to accelerate the dynamical core of the COSMO model with improved performance on CUDA-GPUs compared to the current official version, demonstrating production quality and feature-completeness of the library for models on lat-lon grids. The GridTools-based dynamical core is shipped with COSMO v5.7 and later, see release notes COSMO v5.7.

Although GridTools was developed for weather and climate applications it might be applicable for other domains with a focus on stencil-like computations.

A detailed introduction can be found in the documentation.

Installation instructions

git clone https://github.com/GridTools/gridtools.git
cd gridtools
mkdir -p build && cd build
cmake ..
make -j8
make test

For choosing the compiler, use the standard CMake techniques, e.g. setting the environment variables

CXX=`which g++` # full path to the C++ compiler
CC=`which gcc` # full path to theC compiler
FC=`which gfortran` # full path to theFortran compiler
CUDACXX=`which nvcc` # full path to NVCC
CUDAHOSTCXX=`which g++` # full path to the C++ compiler to be used as CUDA host compiler

Requirements

• C++17 compiler (see also list of tested compilers)
• Boost headers (1.73 or later)
• CMake (3.18.1 or later)
• CUDA Toolkit (11.0 or later, optional)
• MPI (optional, CUDA-aware MPI for the GPU communication module gcl_gpu)

Supported compilers

The GridTools libraries are currently nightly tested with the following compilers on CSCS supercomputers.

Compiler	Backend	Tested on	Comments
Cray clang version 12.0.3	all backends	Piz Daint	P100 GPU
Cray clang version 10.0.2 + NVCC 11.2	all backends	Piz Daint	P100 GPU
Cray clang version 12.0.3	all backends	Piz Daint	with -std=c++20
GNU 11.2.0 + NVCC 11.0	all backends	Piz Daint	P100 GPU
GNU 11.2.0 + NVCC 11.2	all backends	Dom	P100 GPU
GNU 8.3.0 + NVCC 11.2	all backends	Tsa	V100 GPU

Known issues

• Some tests are failing with ROCm3.8.0 (Clang 11).
• CUDA 11.0.x has a severe issue, see #1522. Under certain conditions, GridTools code will not compile for this version of CUDA. CUDA 11.1.x and later should not be affected by this issue.
• Cray Clang version 11.0.0 has a problem with the gridtools::tuple conversion constructor, see #1615.

Partly supported (expected to work, but not tested regularly)

Compiler	Backend	Date	Comments
Intel 19.1.1.217	all backends	2021-09-30	with cmake . -DCMAKE_CXX_FLAGS=-qnextgen
NVHPC 23.3	all backends	2023-04-20	only compilation is tested regularly in CI

Contributing

Contributions to the GridTools framework are welcome. Please open an issue for any bugs that you encounter or provide a fix or enhancement as a PR. External contributions to GridTools require us a signed copy of a copyright release form to ETH Zurich. We will contact you on the PR.

gtc's Issues

Make simple MLIR example for parsing experiments

use e.g. the "loop" dialect from MLIR as an example

Add customization of visit_ methods lookup order in eve visitors

Add a class toggle to enable or disable the use of node superclasses when looking up for suitable visit_ methods

Infer location type of local var as 2 passes

Re-enable mypy in pre-commit

In #49 I disabled mypy again as there was an error I didn't know how to fix.

Cartesian: SIR, GTGenIR, KSwapIR

• Extent SIR for structured grids
• Implement a GTGenIR (an IR close to GridTools code)
• Implement an IR which allows optimization which are not compatible with GridTools

Change GTC repository layout

Change the layout to have one top-level folder per sub-project.

Example:

REPO/
    gtc/
        src/
            gtc/
        docs/
        tests/
        setup.cfg
        setup.py
        pyproject.toml
        ...

    eve/
        src/
            eve/
        docs/
        tests/
        setup.cfg
        setup.py
        pyproject.toml
        ...

    gt_frontend/
        src/
        docs/
        tests/
        setup.cfg
        setup.py
        pyproject.toml
        ...

    cpputil/  # or gridtools-mesh or any other name
        cmake/
        include/
        tests/
        CMakeLists.txt
        ...

Make frozen nodes hashable

SIR: transformation infer location type

Use SID bounds in max neighbors

• propagate static information
• provide a hook to provide static max neighbors in case of atlas where this info is runtim

Update code generator and test it with no mesh concept

Since lookup tables are passed by the user, and they cannot be generally generated, it makes little sense to have a concept for the mesh, and we can just take the mesh as parameter, as we do for the fields. We could treat mesh/neighbor tables as "state" of the computation.

Namespace in visit_ methods

Currently there is no way to distinguish Nodes with the same name from a different dialect.

SIR: parallel model declarations

Add visitor utilities for simple tree traversals and filtering

• Add simple functions or classes to iterate over all the nodes of a tree in predefined orders
• Add filtering utilities based on iterators

Transform unstructured GTIR into a definition IR

Allow type variables in Nodes

It is sometimes useful to restrict types of fields of fields of a node.

Objectives

• Reduce the amount of verification boilerplate code inside the translators.
• Reduce the amount of state to be passed to child nodes inside the translators
• Elimination of redundant nodes

Pydantic supports this out of the box (see pydantic/pydantic#556 and pydantic/pydantic#595).

Open questions:

1. How to check for a field type to be a subtype of generic type? This will be a common question if we allow this, but Python does not allow subclass checks with generic, i.e. the following does not work:

from typing import Generic, TypeVar, List
# raises: TypeError: Subscripted generics cannot be used with class and instance checks
issubclass(List[int], List[T])

2. Consequences of changing the Node base class to GenericModel
3. Is this compatible with mypy?

Example 1

T = TypeVar('T')
class Constant(GT4PyAstNode, Generic[T]):
    value: T

class ExampleNode:
    some_int_constant: Constant[Int]

Example 2

T = TypeVar('T')

class Stencil(GTScriptAstNode, Generic[T]):
    iteration_spec: List[Union[IterationOrder, LocationSpecification, Interval]]
    body: List[T]

class Computation(GTScriptAstNode):
    name: str
    arguments: List[Argument]
    stencils: List[Union[Stencil[Stencil[Statement]], Stencil[Statement]]]

GTIR->NIR->UGPU->CUDA code

Finish lowering from GTIR to cuda code generation.

Create Pass infrastructure to handle node contexts

• Add node contexts
• Add symbol tables and scopes

Prototype frontend for unstructured

Write an absolute minimal frontend to parse GTScript into GTIR

Enums are not typesafe

In my unstructured sir -> naive example I can assign sir.LocationType to a field which requires a naive.LocationType.

Add FVM dependencies to gitpod

Mesh traits and concept helpers

• is_connectivity

• maybe has_connectivities (has_iterationspace)

• type for max_neighbors etc

• move to mesh.hpp and define LocationType tags (namespace topology)

new_infrastructure: Nodes with same field are not distinguished

If I have

class FieldAccess(Expr):
    name: Str

class VarAccess(Expr):
    name: Str

the visitor will not distinguish the 2 nodes. It will enter visit_FieldAccess even for VarAccess. This can be solved by type annotating the visit method's node argument.

However, no such workaround exists for the templates in the TemplatedGenerator. Adding an extra field in one of the 2 classes works, e.g.

class VarAccess(Expr):
    name: Str
    dummy: Optional[str]

Mypy checks with enums

I create this issue to reference from gt4py comments.

@enum.unique
class MyEnum(eve.IntEnum):
    member: 0

reveal(MyEnum.member) # <- int but should be MyEnum

Run this in outside of the eve package (maybe problem also inside?).

@egparedes and I couldn't figure out what's going on, in a short debugging session, but we stopped as it should go away in the next refactoring.

Validate model from assignment

Currently we only validate a Node on construction, but not if a field is assigned.

For validators this can be enabled in pydantic's config by setting

validate_assignment = True

However it seems this doesn't apply to root_validators. Bug in pydantic?

Codegen: iteration index in reduction with sparse field is wrong

Change in behavior in "New infrastructure"

In the new_infrastructure branch there is a change in behavior in the code generator.

For a field like
a_field: Optional[List[SomeNode]], if optional is None, the code generator will now expand to N o n e instead of skipping over it. In the template I used ''.join(a_field)

Pass-Manager

Serialization

Caching "derived information"

Just as a reminder to discuss it at some point...

Dawn has the concept of derived information which is some extra information computed from the IR and re-used by several transformations. (Things that are expensive to recompute.) Would be interesting to discuss if we could implement such a thing with automatic invalidation.

One example could be dependency graphs.
You would give_me_dependency(some_expression) and it will retrieve it from cache or compute it and store in cache.

This would be only a compiler performance optimization which we shouldn't do now.