symengine/symengine-bench

Benchmarking tools for SymEngine

SymEngine Benchmarks

Here is how to build and run the benchmarks:

docker build -t symengine/ubuntu_base ./ubuntu_base
docker build -t symengine/conda_base ./conda_base
docker build -t symengine/symengine --build-arg COMMIT=master ./symengine
docker run -it -p 8888:8888 symengine/symengine jupyter notebook --ip='*' --no-browser

Then open the Plots.ipynb notebook and execute it. It will produce graphs of the benchmarks.

Compare two commits

We can compare two commits as follows. The commit 398a3f39 should be faster than fdf132fc.

COMMIT=398a3f39
docker build -t symengine/symengine:$COMMIT --build-arg COMMIT=$COMMIT ./symengine
docker run -t -v `pwd`:/opt/ symengine/symengine:$COMMIT bash run_copy.sh
cp data.json data-$COMMIT.json
COMMIT=fdf132fc
docker build -t symengine/symengine:$COMMIT --build-arg COMMIT=$COMMIT ./symengine
docker run -t -v `pwd`:/opt/ symengine/symengine:$COMMIT bash run_copy.sh
cp data.json data-$COMMIT.json

And locally execute the Plot-two.ipynb notebook:

jupyter notebook Plot-two.ipynb

Which will plot the two data-*.json files.

symengine-bench's Issues

Compare two symengine branches

I think one of the most useful things would be to compare two symengine branches, and plot the results into the same graph (see also #4). That way one can run this and quickly see if a given PR slowed things down.

Compare two different installations of symengine

The issue #5 is for comparing two symengine branches, but we also need to compare the same symengine commit, but two different ways to install it, e.g.:

• with and without TCMALLOC
• with and without FLINT
• gcc 5.3 vs 7.0
• different docker base images, e.g. ubuntu 16.04 vs. alpine linux (the results can differ, due to gcc options about stack protection and different libc libraries)
  ...

So that we can determine how to get the fastest results for the given symengine commit.

Use TCMALLOC

And anything else that would speed things up. See also #6.

How to design the benchmarking

We definitely need the functionality #5 and #6. I am thinking that perhaps there should be a script, that runs benchmarks for the given symengine commit and a certain way to install it. And then there should be a script that builds docker images A and B with two different configuration (either different symengine commits to do #5 or different way to build it to do #6). Then one can use these two images A and B to generate the comparison graphs.

1. So it seems the lower layer just needs to build a certain symengine version a certain way, run benchmarks and save the result to data.json.
2. The upper layer then runs the lower layer 1. for two different configurations and plots both in the same graph.

For two different symengine versions this can be automated. But for two different docker configurations, this might not be practical to automate, so rather the user would simply save the data.json, then change some docker configuration, rerun, save the new data.json, and then plot both in the same graph.

Hook into asv

Provide a hook, so that one can use asv (https://github.com/spacetelescope/asv) to drive this. asv can only do a subset of symengine-bench (i.e. only time vs commits plots, and with the same build type), but it would still be useful to be able to use it.

Plot the results

Currently Travis just prints the data.json file with the benchmark results (e.g., https://travis-ci.org/symengine/symengine-bench/builds/268544309#L2418), but we should run a Jupyter notebook to create nice pdf files with the graphs and save them somewhere.