opencog/benchmark

Benchmarking the AtomSpace, the pattern matcher and other OpenCog systems

Benchmark

Collection of tools and data to benchmark the AtomSpace, the pattern matcher and other OpenCog processes.

Organization

Each distinct benchmark has it's own instructions for building and running it. See individual directories for details.

The most interesting benchmarks are probably the query-links and query-loop benchmarks (because they are "real-world" benchmarks) and the atomspace benchmark (because it has a very long measurement history, despite being deeply flawed).

• assembly -- examination of assembly code of some simple programs.

• atomspace -- A synthetic benchmark for measuring core AtomSpace functions, such as rates for creating Nodes and Links, rates for deleting them, and rates for setting and getting TruthValues and IncomingSets.

• micro -- Micro-benchmarks for misc items. These benchmarks do not work correctly! They're just plain broken! They report incorrect results! DO NOT USE! See the README in the subdirectory for details about what's wrong.

• python -- Benchmarks for the python bindings to various AtomSpace subsystems.

• query-trite -- A synthetic benchmark containing some trivial pattern matcher queries.

• query-links -- Several real-world benchmarks for performing complex pattern matchs on a real-world natural-language dataset. One models an activity one might need in a parser, that of joining together connectors. It creates 576 BindLink's with four GlobNodes and a Variable, each. The Globs are used to search for connectable connectors, similar to what parser would need to do. That is, this is a search over "jigsaw puzzle pieces", looking for pairs of jigsaw pieces that can connect.

  A second benchmark in this directory computes dot products of vectors in six different ways, each using an increasingly complex query pattern. Uses the same base dataset.

• query-loop -- A real-world benchmark for performing a complex pattern match on a real-world genome dataset. Uses EvaluationLink in a simple/straight-forward way to represent the edges and vertexes of a large graph (for gene/protein interactions), and then hunts for triangles and pentagons in that graph.

• cogserver -- Benchmark of network I/O to the cogserver. Highlights issues created by using netcat instead of using native socket I/O.

Using perf

It can be useful to use perf. Sometimes. Maybe. Here's the cheat-sheet:

To record 15 seconds worth:

sudo perf record -F 199 -p pid --call-graph fp -- sleep 15

CAUTION: need to run perf as root, and it writes crap into the /root/.debug folder! This eats disk!

View reports:

perf report
perf report -n
perf report --stdio -n -g -G

benchmark's Issues

measure getHandle() performance

A key activity for the atomspace is getting Nodes by name, and Links, by outgoing set. Neither of these are measured.

Fix Python benchmarks

Python benchmarks are broken. At the moment all of benchmarks are guarded by #if HAVE_CYTHONX preprocessor condition:

benchmark/atomspace/atomspace/AtomSpaceBenchmark.cc

Line 467 in 9974909

#if HAVE_CYTHONX

But code cannot be compiled when HAVE_CYTHONX is defined.

benchmark 'typical' EvaluationLinks

The current atomspace benchmark only measures node and link creation, and not 'typical' data. resulting in real-life performance that is one or two orders of magnitude worse than what's measured. See for example: opencog/test-datasets#7

The benchmark needs to be updated to measure more 'typical' data.

Move make_benchmark_graphs.py scripts from atomspace to benchmark repo

File make_benchmark_graphs.py is part of opencog/benchmark/atomspace/atomspace benchmarkin.
Todo:

• move make_benchmark_graphs.py into benchmark repo
• fix atomspace/atomspace/README.md using new file location

Use standard C++ benchmarking library for atomspace microbenchmarks

To follow up the @linas comment: here and to not reinvent the wheel I would propose using some ready C++ benchmarking library.

Requirements to such library:

• ability to keep run results
• ability to compare run results
• showing info on system which is used to perform test
• anti-optimization tricks

Google Benchmark (https://github.com/google/benchmark) seems to be a good candidate. Unfortunately there are no ready to install packages for benchmark libraries so it will be additional manual step in building procedure.

Other well known libraries:

• Celero (https://github.com/DigitalInBlue/Celero)
• Nonius (https://github.com/libnonius/nonius)
• Hayai (https://github.com/nickbruun/hayai)

Some review and comparision can be found here (full articles are here and here).

Measure `getHandle()` performance

One key activity is getting Nodes by name, and Links by outgoing set. This is not measured currently, it should be.

Measure ScopeLink, GetLink, BindLink, DualLink performance

Scope links are probably the most expensive link types to construct and insert in the AtomSpace. First calculating the hash value during construction is more expensive than for non scope links. Scope links like PatternLink, BindLink, etc, are even more expensive to construct so. Then inserting them to the AtomSpace also requires more computation like testing for alpha-equivalence.

query_benchmark does support scope links but mostly for querying, not construction and insertion...

All that to say that scope links really should be supported by atomspace_bm (if that doesn't get replaced by something else soon).