ga-developers/ga-benchmark

A benchmark for Geometric Algebra libraries, library generators, and code optimizers.

ga-benchmark

This repository is a joint effort to define standards and methodologies for benchmarking Geometric Algebra libraries, library generators, and code optimizers.

The goal of this project is to help physicists, chemists, engineers, and computer scientists to choose the Geometric Algebra solution that best suits their practical needs, as well as to push further the improvement of the compared solutions and to motivate the development of new tools.

The current version compares seven C++ solutions:

• Gaalet (https://sourceforge.net/projects/gaalet)
• Gaalop (http://www.gaalop.de)
• Garamon (https://github.com/vincentnozick/garamon)
• GATL (https://github.com/laffernandes/gatl)
• GluCat (https://github.com/penguian/glucat)
• TbGAL (https://github.com/Prograf-UFF/tbgal)
• Versor (http://versor.mat.ucsb.edu)

But it is easy to include new ones!

The solutions are compared considering four models of geometry:

• Conformal (2D and 3D)
• Euclidean (2D, 3D, 4D, and 5D)
• Homogeneous (2D, 3D, and 4D)
• Minkowski (2D and 3D)

ten unary operations:

• Clifford conjugation
• Dualization
• Grade Involution
• Inversion
• Normalization
• Reversion
• Squared reverse norm
• Unary minus
• Unary plus
• Undualization

twelve binary operations:

• Addition
• Commutator product
• Dot product
• Geometric product
• Hestenes inner product
• Inverse geometric product
• Left contraction
• Outer product
• Regressive product
• Right contraction
• Scalar product
• Subtraction

and one algorithm:

• Inverse kinematics assuming the conformal model of 3D Euclidean space

The project documentation is structured as follows:

1. Requirements
2. Building and Running
3. Ploting Results
4. Latest Results
5. Further Knowledge
   • Gaalet
   • Gaalop
   • Garamon
   • GATL
   • GluCat
   • TbGAL
   • Versor
   • How ga-benchmark Recognizes a Solution
6. License

1. Requirements

Make sure that you have all the following tools, libraries, library generators, and code optimizers installed and working before attempting to use ga-benchmark.

Required tools:

• Your favorite C++17 compiler
• CMake
• Python 3 interpreter
• Jupyter Notebook

Required C++ libraries:

• Google Benchmark
• Threads (pthread library on Linux, shlwapi library on Windows, and kstat library on Solaris)

Required Python modules:

• argparse
• ipywidgets
• json
• scipy
• shutil
• tqdm
• typing

Expected Geometric Algebra libraries, library generators, and code optimizers:

• Gaalet: Geometric Algebra ALgorithms Expression Templates
• Gaalop: Geometic Algebra Algorithms Optimizer
• Garamon: Geometric Algebra Recursive and Adaptive Monster
• GATL: Geometric Algebra Template Library
• GluCat: Clifford Algebra Templates
• TbGAL: Tensor-based Geometric Algebra Library
• Versor

The Further Knowledge section describes how to install the expected libraries, library generators, and code optimizers, and how to make ga-benchmark recognize the existence of a solution to be compared, including custom ones not listed above. Notice that Gaalop and Garamon must be instrumented.

2. Building and Running

Use the git clone command to download the project:

git clone https://github.com/ga-developers/ga-benchmark.git ga-benchmark
cd ga-benchmark

The basic steps for configuring and building ga-benchmark look like this:

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
cmake --build . --parallel 8

Call the test target to run the benchmark:

cmake --build . --target test

Building and running ga-benchmark may take several hours. Please be patient and keep the computer allocated exclusively for the execution of the tests.

3. Ploting Results

After running ga-benchmark, use the Jupyter Notebook ga-benchmark/gabm_results.ipynb to see tables and charts that compare solutions. Those tables and charts are produced by functions defined in the ga-benchmark/gabm.py file.

Alternatively, you can generate the tables and charts from the command line. Assuming that the current folder is ga-benchmark/build and you want to save resulting charts and tables to the ga-benchmark/results folder, call:

cd ..
python -m gabm -i ./build -o ./results

If everything is alright, then results are in the ga-benchmark/results folder. This is the folder structure expected by the repository https://github.com/ga-developers/ga-benchmark-runs, since it also includes a copy of the input JSON files.

4. Latest Results

The latest results are available here: https://github.com/ga-developers/ga-benchmark-runs. Please, help us to keep it updated.

5. Further Knowledge

If your system does not include the expected libraries, library generators, and code optimizers listed above, then you must install them following the instructions provided by the developers.

The commands below summarize the installation process of each of the expected solutions. However, one should note that the process may have changed with the release of new versions.

Here, we assume that ga-benchmark is the current folder and Linux operating system. The installation procedures will place the expected solutions inside the ga-benchmark/libs folder.

Gaalet

svn checkout https://svn.code.sf.net/p/gaalet/code/trunk libs/Gaalet/repository
mkdir -p libs/Gaalet/install/include/gaalet
cp libs/Gaalet/repository/include/cpp0x/* libs/Gaalet/install/include/gaalet

Gaalop

sudo apt install xmaxima
sudo apt install default-jre
mkdir -p libs/Gaalop/download
wget -O libs/Gaalop/download/GaalopPrecompiler.tar.gz "http://www.gaalop.de/wp-content/uploads/x86-64/GaalopPrecompiler-2.0.1447-Linux.tar.gz"
tar xf libs/Gaalop/download/GaalopPrecompiler.tar.gz -C libs/Gaalop/download
mkdir -p libs/Gaalop/install/usr/local/bin/GaalopPrecompiler
mv libs/Gaalop/download/GaalopPrecompiler-2.0.1447-Linux/* libs/Gaalop/install/usr/local/bin/GaalopPrecompiler
rmdir libs/Gaalop/download/GaalopPrecompiler-2.0.1447-Linux

See Gaalop's download page for a list of prebuilt binaries of the precompiler for Linux and Windows.

Garamon

sudo apt install libeigen3-dev
git clone https://github.com/vincentnozick/garamon.git libs/Garamon/repository
mkdir libs/Garamon/repository/build
mkdir libs/Garamon/install
cd libs/Garamon/repository/build
cmake ..
make
for conf in ../../../../source/Garamon/algebras/*.conf
do
  ./garamon_generator $conf
  filename=$(basename -- "$conf")
  cd output/garamon_"${filename%.*}"
  mkdir build
  cd build
  cmake -DCMAKE_BUILD_TYPE=Release ..
  make
  make DESTDIR=../../../../../install install
  cd ../../..
done
cd ../../../..

GATL

git clone https://github.com/laffernandes/gatl.git libs/GATL/repository
mkdir libs/GATL/repository/cpp/build
mkdir libs/GATL/install
cd libs/GATL/repository/cpp/build
cmake -DCMAKE_INSTALL_PREFIX="$(cd ../../../install; pwd -P)" ..
cd ../../../../..

GluCat

sudo apt install libeigen3-dev
sudo apt install libboost-all-dev
git clone https://github.com/penguian/glucat.git libs/GluCat/repository
cd libs/GluCat/repository
make -f admin/Makefile.common cvs
./configure --disable-pyclical
make
make DESTDIR=$(realpath ../install) install
cd ../../..

TbGAL

sudo apt install libeigen3-dev
git clone https://github.com/Prograf-UFF/TbGAL.git libs/TbGAL/repository
mkdir libs/TbGAL/repository/build
mkdir libs/TbGAL/install
cd libs/TbGAL/repository/build
cmake -DCMAKE_INSTALL_PREFIX="$(cd ../../install; pwd -P)" ..
cd ../../../..

Versor

git clone https://github.com/wolftype/versor.git libs/Versor/repository
mkdir libs/Versor/repository/build
mkdir libs/Versor/install
cd libs/Versor/repository/build
cmake -DCMAKE_BUILD_TYPE=Release ..
make
make DESTDIR=../../install install
cd ../../../..

How ga-benchmark Recognizes a Solution

A given libraries, library generators, and code optimizers will be recognized by ga-benchmark if and only if the ga-benchmark/source folder includes a subfolder with the benchmark code of the given solution. The ga-benchmark/source/README.md file presents detailed instructions about how to prepare such subfolder.

6. License

This software is licensed under the GNU General Public License v3.0. See the LICENSE file for details.

ga-benchmark's Issues

GluCat 0.8.4 released

I have recently released GluCat 0.8.4: https://github.com/penguian/glucat/tags https://sourceforge.net/projects/glucat/files/
The main change is that PyClical is now based on Python 3.X.

How do I integrate it into the benchmark?

Convert color to hex

In plot.py
Convert color to hex to work around the case where the data to be plotted consists of 3 or 4 points. See
https://stackoverflow.com/questions/48872731/issue-with-matplotlib-scatterplot-and-color-maps and similar.

Allow GluCat to transform between matrix_multi and framed_multi

For GluCat to transform between matrix_multi<> and framed_multi<>, the indices LO and HI should be approximately equal in absolute value, and should be large enough to allow . For the benchmarks, I would suggest that for both
source/GluCatFramedMulti/SpecializedAlgebra.hpp
and
source/GluCatMatrixMulti/SpecializedAlgebra.hpp
the following definitions should be used.

#if (GABENCHMARK_D_DIMENSIONS < 4)
#define GABENCHMARK_GLUCAT_POSITIVE_GENERATORS (4)
#define GABENCHMARK_GLUCAT_NEGATIVE_GENERATORS (-4)
#else
#define GABENCHMARK_GLUCAT_POSITIVE_GENERATORS (8)
#define GABENCHMARK_GLUCAT_NEGATIVE_GENERATORS (-8)
#endif

This should be enough to accommodate all models to be tested.

For details, see https://github.com/penguian/glucat/blob/master/glucat/framed_multi_imp.h
https://github.com/penguian/glucat/blob/master/glucat/matrix_multi_imp.h
and https://github.com/penguian/glucat/blob/master/glucat/index_set_imp.h

Use indices properly with results of meshgrid

In plot.py,
for the results of meshgrid work correctly, the use of x and y must be switched around, and x_min and y_min must be properly taken into account. Unquote the print statements to check. See
https://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html

$ git diff -w plot.py
diff --git a/plot.py b/plot.py
index 9200446..ff0bafd 100644
--- a/plot.py
+++ b/plot.py
@@ -150,10 +150,16 @@ def _plot_data(data: dict, all_libraries: set, folder: str, verbose: bool) -> No
                         if group == 'Product':
                             x_min, x_max, y_min, y_max, z_min, z_max = _get_axes_bounds_for_products(operations)
                             X, Y = np.meshgrid(np.arange(x_min, x_max + 1), np.arange(y_min, y_max + 1))
+                            """
+                            print('x_min == {:>3}, x_max == {:>3}, 1+x_max-x_min == {:>3}, X.shape[1] == {}'.format(
+                                x_min, x_max, 1+x_max-x_min, X.shape[1]))
+                            print('y_min == {:>3}, y_max == {:>3}, 1+y_max-y_min == {:>3}, X.shape[0] == {}'.format(
+                                y_min, y_max, 1+y_max-y_min, X.shape[0]))
+                            """
                             X_, Y_ = X.ravel() - 0.5, Y.ravel() - 0.5
                             for operation, libraries in operations.items():
                                 relative_path = os.path.join(context_key, metric, model, str(d), group, operation)
-                                message('  Ploting results to "%s"... ', os.path.join('OUTPUT_FOLDER', relative_path))
+                                message('  Plotting results to "%s"... ', os.path.join('OUTPUT_FOLDER', relative_path))
                                 os.makedirs(os.path.join(folder, relative_path), exist_ok=True)
                                 best_library = np.full(X.shape, '', np.object)
                                 best_color = np.full(X.shape, -1, np.float32)
@@ -161,10 +167,9 @@ def _plot_data(data: dict, all_libraries: set, folder: str, verbose: bool) -> No
                                 for library, values in libraries.items():
                                     color_ind = all_libraries.index(library) / (len(all_libraries) - 1)
                                     color = cmap(color_ind)
-
                                     Z = np.full(X.shape, float('inf'), np.float32)
                                     for xy, z in values.items():
-                                        Z[xy[0] - x_min, xy[1] - y_min] = z
+                                        Z[xy[1] - y_min, xy[0] - x_min] = z
 
                                     replace = Z < best_Z
                                     best_library[replace] = library
@@ -196,7 +201,7 @@ def _plot_data(data: dict, all_libraries: set, folder: str, verbose: bool) -> No
                                 ax.set_yticks(np.arange(y_min, y_max + 1))
                                 ax.imshow(best_color, interpolation='none', origin='lower', cmap=cmap)
                                 for x, y in zip(X.flatten(), Y.flatten()):
-                                    ax.text(y, x, '%s\n%1.5f ms' % (best_library[x, y], best_Z[x, y]), va='center', ha='center')
+                                    ax.text(y, x, '%s\n%1.5f ms' % (best_library[y-y_min, x-x_min], best_Z[y-y_min, x-x_min]), va='center', ha='center')
                                 fig.tight_layout()
                                 fig.savefig(os.path.join(folder, relative_path, 'Result.pdf'))
                                 # plt.show()