dead2/stabilizer

Stabilizer: Rigorous Performance Evaluation

Stabilizer

*Statistically Rigorous Performance Evaluation

This repo is maintained by Hans Kristian Rosbach aka Dead2 (stabilizer àt circlestorm dót org)

About this fork

This is a fork of the long-unmaintained Stabilizer by Charlie Curtsinger and Emery D. Berger, consider visiting the original repo.

This version supports LLVM version 12, although there seem to be some crashes with SZ_STACK or SZ_CODE enabled. SZ_HEAP and SZ_LINK seem to work fine however.

Changes in this fork compared to the original:

• Partial LLVM 12 compatibility inherited from other 3rd party forks of Stabilizer.
• Completely rewritten compiler wrapper, much better compatibility with actual clang behavior, and much less likely to require major buildsystem changes.
• Enabling/disabling features is now done by env variables instead of parameters.
• Dropped support for GCC/Gfortran since DragonEgg has not been ported to newer LLVM.
• Removed scripts and configs for running SPEC CPU2006.

Help is wanted for testing and fixing the remaining crashes. Despite the crashes, this is still useful for enabling heap randomizations.

Currently LLVM version 12 is supported and tested, but it likely also supports several older versions (maybe without crashes too) and possibly newer versions. Please provide test feedback, so I can update this text.

I am not an LLVM expert, but it seemed that one of the biggest problems this project had was that it absolutely required a huge amount of changes to most buildsystems and would not pass even the most basic compiler tests in CMake or GNU Autoconf. This likely limited the audience that was able to test and use Stabilizer, resulting in few people interested in fixing bugs.

About Stabilizer

Stabilizer is a compiler transformation and runtime library for dynamic memory layout randomization. Programs built with Stabilizer run with randomly-placed functions, stack frames, and/or heap objects. Functions and stack frames are moved repeatedly during execution. A random memory layout eliminates the effect of layout on performance, and repeated randomization leads to normally-distributed execution times. This makes it straightforward to use standard statistical tests for performance evaluation.

A more detailed description of Stabilizer is available in the Paper.

Stabilizer Requirements

Stabilizer requires LLVM version 12, see above comment about other versions. Stabilizer runs on OSX and Linux, and supports x86, x86_64, and PowerPC.

The build system assumes LLVM include files will be accessible through your default include path.

szcc, the compiler wrapper, is written in Python and requires Python3

Building Stabilizer

$ git clone https://github.com/Dead2/stabilizer.git
$ git submodule update --init --recursive
$ make

By default, Stabilizer is built with debug output enabled. Run make clean release to build the release version with asserts and debug output disabled.

Using Stabilizer

Stabilizer includes the szcc and szcc++ compiler wrapper, which builds programs using the Stabilizer compiler transformations. szcc passes on common clang flags, and is compatible with C and C++ inputs.

To manually compile a program in foo.c with Stabilizer, run:

export SZ_CODE=1 SZ_HEAP=1 SZ_STACK=1 SZ_LINK=1
$ szcc foo.c -o foo

The exported env flags enable the various randomizations, and may be used in any combination.

• SZ_CODE Move functions repeatedly during execution.
• SZ_HEAP Randomize stack location.
• SZ_STACK Move stack repeatedly during execution.
• SZ_LINK Randomly reorder linking (only once, during compilation)
• SZ_VERBOSE Turns on verbose debugging output from szcc during compile.
• SZ_CLEAN Turns on removal of temp files produced by szcc during compile.

Letting szcc impersonate Clang

szcc can also be run by using clang and clang++ symlinks if the stabilizer dir is added to PATH. You can also specify compilers manually.

Path:

export PATH=/path/to/stabilizer/:$PATH
./configure

Manually with CMake:

cmake . -DCMAKE_C_COMPILER=/path/to/stabilizer/clang -DCMAKE_CXX_COMPILER=/path/to/stabilizer/clang++

Manually with GNU Autoconf:

CC=/path/to/stabilizer/clang CXX=/path/to/stabilizer/clang++ ./configure

The resulting executable is linked against libstabilizer.so (or .dylib on OSX). Place this library somewhere in your system's dynamic library search path or add the Stabilizer base directory to your LD_LIBRARY_PATH or DYLD_LIBRARY_PATH environment variable to allow running the compiled programs.

Credits

Original code by Charlie Curtsinger and Emery D. Berger

Copyright (C) 2013 University of Massachusetts Amherst

License

Stabilizer is distributed under the GNU GPLv2 license. Contact [email protected] if you are interested in licensing Stabilizer for commercial use.

stabilizer's Issues

TODO list, Help wanted

This is a list of suggested improvements that anyone can contribute to, I will try to keep it updated as we add/remove items.
Please try not to clutter this thread with replies related to the below. If need be you can open a new issue dedicated to discussing one of the points below.

Wanted improvements:

• Fix for SZ_STACK related crashes.
• Fix for SZ_CODE related crashes.
• Port to ARM/Aarch64.
• Port to S390.
• Port to Windows.
• More and better comments, especially useful in llvm-pass code that may be hard to follow and debug for beginners.
• Investigate whether pass/IntrinsicLibcalls.h needs additional Intrinsics added due to new ones having been added in later llvm versions. [1]
• Improved tests [2].
• Github Actions CI testing, preferably also running the tests on all supported platforms and architectures.

1: There is very little info about why these intrinsics are on the list and others are not, if you know the details then please create a PR with
improved comments to document this feature.

2: The current tests are not able to detect the Stack and Code crashes, and it is hard to determine whether the current tests actually passed or failed due to an abundance of hard-to-understand output.
An option to improve test understanding would be to change the project from hand-written makefiles to CMake and then utilize CTest.

Question: Can the randomization code be separated out and called within a library

It's great that you've decided to maintain this tool ❤️

I am interested in doing all the memory layout randomization from within an LLVM compiled application of my own. I write in Julia (a programming language compiled using LLVM), and we're discussing how we can make our benchmarking tools handle memory layout issues better. My idea is to call a function that does the randomization, between benchmark evaluations.

Is the randomization functionality nicely encapsulated within a single C++ function? If so, which one? If not, is it possible to make it so? Perhaps only some of the randomization is possible this way? Perhaps it isn't possible to shuffle code about while it's running?

Many thanks 🙏

Crash on first line of code

Tested using:
Clang 12.0.1 (Fedora 12.0.1-1.fc34)
Built zlib-ng from develop branch and ran the resulting example (basic self-tests) binary.

export SZ_CODE=1 SZ_LINK=0 SZ_HEAP=0 SZ_STACK=0 SZ_CLEAN=1
cmake -DCMAKE_VERBOSE_MAKEFILE=ON -DWITH_BENCHMARKS=OFF -DWITH_GTEST=OFF -DWITH_OPTIM=OFF -DWITH_NEW_STRATEGIES=OFF -DCMAKE_C_COMPILER=/home/hansr/github/stabilizer/szcc -DCMAKE_C_FLAGS="-Wall -Wpedantic -g3 -ggdb -O0" .`

gdb ./example

Program received signal SIGSEGV, Segmentation fault.
main (argc=1, argv=0x7fffffffde58) at /home/hansr/github/zlib/zlib-ng/test/example.c:960
960         if (zVersion()[0] != myVersion[0]) {
=> 0x00000000004040e4 <main+20>:        ff 15 56 03 70 15       call   *0x15700356(%rip)        # 0x15b04440

This is the first line of actual code in main(), after variable definitions.

Did a little experiment to see whether adding a line above that with __asm__("nop"); would do anything at all.

#0  main (argc=1, argv=0x7fffffffde58) at /home/hansr/github/zlib/zlib-ng/test/example.c:962
962         if (zVersion()[0] != myVersion[0]) {
=> 0x00000000004040e5 <main+21>:        ff 15 55 00 3a a0       call   *-0x5fc5ffab(%rip)        # 0xffffffffa07a4140

Not much changed, but the call address seems to be very random, unless there is a bit pattern I am not seeing.

Seems to me like something is still quite wrong somewhere in the Stabilizer plugin pass when it comes to patching the call addresses.

SZ_STACK crash with debuginfo compiled in

Reproduceable using Clang 14.0.5 and zlib-ng develop snapshot (probably the same with any version of zlib-ng really).

export SZ_CODE=0 SZ_LINK=0 SZ_HEAP=0 SZ_STACK=1 SZ_CLEAN=1
cmake -DCMAKE_VERBOSE_MAKEFILE=ON -DWITH_BENCHMARKS=OFF -DCMAKE_C_COMPILER=/home/hansr/github/stabilizer/szcc -DCMAKE_CXX_COMPILER=/home/hansr/github/stabilizer/szcc++ -DCMAKE_C_FLAGS="-g" .

Can be reproduced about 80% of the time by running this:
ctest . -j 10 --exclude-regex "benchmark.*" --progress --output-on-failure

There are two tests that typically fail, sometimes only one of them and sometimes neither.

[  FAILED  ] 2 tests, listed below:
[  FAILED  ] deflate.concurrency
[  FAILED  ] deflate_quick.bi_valid

Not much details available:

[ RUN      ] deflate.concurrency
/home/hansr/github/zlib/zlib-ng/test/test_deflate_concurrency.cc:146: Failure
Expected equality of these values:
  0
  err
    Which is: -3
invalid stored block lengths
[  FAILED  ] deflate.concurrency (476 ms)
[----------] 11 tests from deflate (481 ms total)

[----------] 2 tests from deflate_quick
[ RUN      ] deflate_quick.bi_valid
/home/hansr/github/zlib/zlib-ng/test/test_deflate_quick_bi_valid.cc:76: Failure
Expected equality of these values:
  err
    Which is: 0
  1
/home/hansr/github/zlib/zlib-ng/test/test_deflate_quick_bi_valid.cc:79: Failure
Expected equality of these values:
  err
    Which is: -3
  0
[  FAILED  ] deflate_quick.bi_valid (0 ms)

Interesting that only these two tests repeatedly fails when we compile with -g, but none of the other 61 tests fail.
minigzip also fails, but less often, around 5-10% chance it seems like, at least on quite big files.

Note: SZ_CODE also fails 53/63 tests if enabled, but that seems to be an unrelated bug. I need to look at how to narrow that down to a cause.

crush on attempt to call `__tls_get_addr`

I have no MRE, but example binary from zlib-ng crushes because of that.

It turns out in some cases access to a TLS variable can generate an IP relative call on the backend, which of course breaks if the function was relocated at runtime. Since it happens on the compiler backend I don't know how to distinguish cases when stabilizer should keep access to a TLS variable intact and when to replace it.

As always we can replace all accesses to TLS variables with a call to a non-relocatable helper function.

SZ_CODE Function relocation crash info

This thread is for tracking information related to the crash on application-startup when SZ_CODE is enabled.

If anyone wants to have a go at fixing this bug, please contribute any additional information that can be useful in fixing this bug.

I suspect that this bug is somewhere in pass/Stabilizer.cpp, where the related llvm pass integration is.

Would Stabilizer actually help on modern hardware?

Hi, I saw this fork from the original stabilizer repo and is very interested in it. I wanted to know how much a difference does this make for modern hardware with much larger cache size, more associative ways and better hardware prefetcher. Putting some figures in the readme can help others know that whether this is still important nowadays and attract others to contribute.

Tests libquntum and bzip crash in release build

$ cmake -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=clang++-14 -DCMAKE_CXX_FLAGS="-isystem /usr/include/llvm-14 -isystem /usr/include/llvm-c-14"
...
$ cmake --build build
...
$ ctest --test-dir build --output-on-failure
Internal ctest changing into directory: /usr/local/src/stabilizer/build
Test project /usr/local/src/stabilizer/build
    Start 1: hello
1/3 Test #1: hello ............................   Passed    0.01 sec
    Start 2: libquantum
2/3 Test #2: libquantum .......................Subprocess aborted***Exception:   0.04 sec
 [/usr/local/src/stabilizer/runtime/libstabilizer.cpp:219]  ABORT: Fault at 0x7fbfc2c6da37, accessing address 0x564abc104020
/usr/local/src/stabilizer/build/runtime/libstabilizer.so(_Z5panicv+0x21) [0x7fbfc2c6d581] [at 0x7fbfc2c6d581]
/usr/local/src/stabilizer/build/runtime/libstabilizer.so(_Z7onFaultiP9siginfo_tPv+0x5d) [0x7fbfc2c6f27d] [at 0x7fbfc2c6f27d]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fbfc2858520] [at 0x7fbfc2858520]
/usr/local/src/stabilizer/build/runtime/libstabilizer.so(_ZN8Function6copyToEPv+0x107) [0x7fbfc2c6da37] [at 0x7fbfc2c6da37]
/usr/local/src/stabilizer/build/runtime/libstabilizer.so(_ZN16FunctionLocationC2EP8Function+0x74) [0x7fbfc2c6dde4] [at 0x7fbfc2c6dde4]
/usr/local/src/stabilizer/build/runtime/libstabilizer.so(_ZN8Function8relocateEv+0x24) [0x7fbfc2c6daf4] [at 0x7fbfc2c6daf4]
/usr/local/src/stabilizer/build/runtime/libstabilizer.so(_Z6onTrapiP9siginfo_tPv+0x80) [0x7fbfc2c6ef00] [at 0x7fbfc2c6ef00]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fbfc2858520] [at 0x7fbfc2858520]
/usr/local/src/stabilizer/build/tests-prefix/src/tests-build/libquantum/libquantum(stabilizer_main+0) [0x564abc0fa990] [at 0x564abc0fa990]
/usr/local/src/stabilizer/build/runtime/libstabilizer.so(main+0xf8) [0x7fbfc2c6ee38] [at 0x7fbfc2c6ee38]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fbfc283fd90] [at 0x7fbfc283fd90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fbfc283fe40] [at 0x7fbfc283fe40]
/usr/local/src/stabilizer/build/tests-prefix/src/tests-build/libquantum/libquantum(+0x60a5) [0x564abc0ec0a5] [at 0x564abc0ec0a5]

    Start 3: bzip2
3/3 Test #3: bzip2 ............................Subprocess aborted***Exception:   4.55 sec
 [/usr/local/src/stabilizer/runtime/libstabilizer.cpp:219]  ABORT: Fault at 0x4b6fabee, accessing address (nil)
/usr/local/src/stabilizer/build/runtime/libstabilizer.so(_Z5panicv+0x21) [0x7f8ecb744581] [at 0x7f8ecb744581]
/usr/local/src/stabilizer/build/runtime/libstabilizer.so(_Z7onFaultiP9siginfo_tPv+0x5d) [0x7f8ecb74627d] [at 0x7f8ecb74627d]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8ecb32c520] [at 0x7f8ecb32c520]
/usr/local/src/stabilizer/build/tests-prefix/src/tests-build/bzip2/bzip2(+0x42fe) [0x5606cc6902fe] [at 0x4b6fabee]


33% tests passed, 2 tests failed out of 3

Total Test time (real) =   4.62 sec

The following tests FAILED:
          2 - libquantum (Subprocess aborted)
          3 - bzip2 (Subprocess aborted)
Errors while running CTest

rename LLVMStabilizer.so

I checked ldconfig sources and I see no way to register LLVMStabilizer.so as a system library. In my opinion it is desirable to do so and get rid of the absolute path in opt -load command.

Right now the absolute path defined here: https://github.com/magras/stabilizer/blob/115c2f78288df717411e1fff641d1d5fb8edcdfb/szcc#L272

This is another step towards packaging stabilizer.

C++ exceptions

Code randomization breaks exception handling:

int main() {
    try { throw 42; }
    catch (int) {}
}

$ SZ_CODE=1 SZ_LOWER=1 szcc++ exception.cpp

...
LandingPadInst not the first non-PHI instruction in the block.
  %7 = landingpad { i8*, i32 }
          catch i8* %.indirect1
in function stabilizer_main
LLVM ERROR: Broken function found, compilation aborted!
...

distutils package is deprecated

/usr/local/bin/szcc:10: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives

szcc doesn't work from /usr/bin

I don't think that it is a good idea to check existence of clang at the same path. It makes impossible to install szcc to a system dir.

stabilizer/szcc

Lines 36 to 37 in 115c2f7

	if tmp and os.path.split(tmp)[0] == os.path.dirname(__file__):
	tmp = None

Remove duplicate loads from relocation table

Stabilizer's relocation table consists of constants, so it should be possible to reuse values loaded from it. And in an ideal world duplicate calls to pure functions (readnone) should be merged too. This should reduce the impact of stabilizer on performance.

SZ_STACK Stack relocation crash info

This thread is for tracking information related to the random crashes occuring when SZ_STACK is enabled.
If anyone wants to have a go at fixing this bug, please contribute any additional information that can be useful in fixing this bug.

I suspect that this bug is somewhere in pass/Stabilizer.cpp, where the related llvm pass integration is.