Python vs. C++ backend

[emxaz]

Python vs. C++ backend

[nickovic]

The actual monitoring algorithms for temporal logic operators are implemented in both C++ (rtamt/cpplib/stl/rtamt_stl_library) and Python (rtamt/operation/stl). Both implementations provide the same functionality. Either one or the other is dynamically loaded based on the is_pure_python flag in STLSpecification object (the default value is true, i.e. the pure Python version). We use Boost.Python to create Python wrappers around C++ objects (see (rtamt/cpplib/stl/rtamt_stl_library_wrapper).

The PyPi package currently contains both the source distribution and the binary wheel, i.e. C++ library is pre-compiled on an Ubuntu 16.04 distribution. Our first tests show no visible performance gains in the C++ implementation compared to its Python counterpart, but we need to study it more closely. In principle, we plan to keep both implementation and to have a PyPi package that automatically builds the C++ binaries from the source.

I suggest using the Python-only implementation for now, as we do not have yet visible evidence about the advantage of using the Python/C++ version.

[nickovic]

I hope it answers your questions, do not hesitate to ask for further clarifications.

[emxaz]

Thanks, that about covers it for now. I'm interested to hear more regarding the performance tests you ran -- were the specifications simple enough that the overhead of Python didn't make a measurable difference?

[nickovic]

I reran the performance tests, and now I am indeed able to measure the difference between the pure Python and the Python/C++ implementation - I must have done something wrong in my first evaluation.

There are one major source of complexity in STL monitors - the size of the temporal operator intervals (defining the number of samples that you need to process). I used the time library and recorded the time before and after making an update with time.time()). I used a simple STL formula (always[0:K] (a+b <= 2)) and varied K, and indeed there is a considerable difference between the pure Python and the Python/C++.

The difference starts getting noticeable for K >= 5000

K Python Python/C++
5k 0.009 0.0014
50k 0.2 0.008
500k 40 0.07
etc.

[nickovic]

I reran the performance tests, and now I am indeed able to measure the difference between the pure Python and the Python/C++ implementation - I must have done something wrong in my first evaluation.

There are one major source of complexity in STL monitors - the size of the temporal operator intervals (defining the number of samples that you need to process). I used the time library and recorded the time before and after making an update with time.time()). I used a simple STL formula (always[0:K] (a+b <= 2)) and varied K, and indeed there is a considerable difference between the pure Python and the Python/C++.

The difference starts getting noticeable for K >= 5000

K Python Python/C++
5k 0.009 0.0014
50k 0.2 0.008
500k 40 0.07
etc.