ssrg-vt/hetersec

Source code for the HeterSec project.

HeterSec - Software diversification using ISA heterogeneity

---

Overview

HeterSec is a framework that enables application software diversification using ISA heterogeneity. It runs diversified processes on top of off-the-shelf commodity machines of different ISAs (e.g., x86-64, AArch64). HeterSec hides the complex differences between ISAs including that between instructions, memory layout, registers, and ABIs, among others, and makes it easier to build and launch ISA-diversified application instances. To demonstrate HeterSec's objectives, the project has developed prototypes of two techniques that use ISA heterogeneity for software diversification as proofs-of-concept: multi-ISA-based moving target defense (MTD) and multi-ISA-based multi-version execution (MVX).

The work is published at RAID 2020 (raid20.pdf).

---

Requirement

We can run HeterSec on real x86_64 and arm64 machines connected using ConnectX-4 InfiniBand. However, it's easier to set up the environment with QEMU VMs. Here we demonstrate how to run HeterSec on QEMU VMs.

i) Install dependency packages:

$ sudo apt-get update
$ sudo apt-get install build-essential libssl-dev libncursesw5-dev git curl bc bridge-utils
$ sudo apt-get install qemu-system-x86 qemu-system-arm gcc-aarch64-linux-gnu

ii) Download QEMU images:

• HeterSec_VM_images.tar.gz (610MB downloaded; 4GB x86.img, 2GB arm.img after extract the tarball). https://drive.google.com/file/d/1IPHnREc_J2CSbe186u0pvh6XY7Pd4Yiu/view?usp=sharing

The username/password for the VMs: popcorn/popcorn.

iii) Configure host network

Use the init_tap_network.sh script to set up the tap0/tap1 interfaces and enable IP forwarding on the host machine (eth0 is your ethernet NIC name):

$ ./init_tap_network.sh eth0

More detail can be found here to set up the host network.

---

Setup

Build the kernels

Clone the souce code:

$ git clone --recursive https://github.com/ssrg-vt/HeterSec.git

Make a copy of the kernel source:

$ cd HeterSec/src
$ cp -r hetersec-kernel hetersec-kernel-arm64

Build the x86_64 kernel and arm64 kernel respectively:

$ cp hetersec-kernel/kernel/popcorn/configs/config-hs-x86_64-qemu hetersec-kernel/.config
$ make -C hetersec-kernel -j8
$ cp hetersec-kernel-arm64/kernel/popcorn/configs/config-hs-arm64-qemu hetersec-kernel-arm64/.config
$ ARCH="arm64" make -C hetersec-kernel-arm64 -j8

Boot the VMs

Boot the VMs with newly built kernels (the following scripts are used in the tap network setting; if you use the bridge, please follow the QEMU boot script here and here):

$ sudo qemu-system-x86_64 \
    -enable-kvm -cpu host -smp 2 -m 4096 -no-reboot -nographic \
    -drive id=root,media=disk,file=x86.img \
    -net nic,macaddr=00:da:bc:de:00:13 -net tap,ifname=tap0,script=no \
    -kernel hetersec-kernel/arch/x86/boot/bzImage \
    -append "root=/dev/sda1 console=ttyS0" \
    -pidfile vm0.pid 2>&1 | tee vm0.log

$ sudo qemu-system-aarch64 \
    -machine virt -cpu cortex-a57 -m 4096 -nographic \
    -drive id=root,if=none,media=disk,file=arm.img \
    -device virtio-blk-device,drive=root \
    -netdev type=tap,id=net0,ifname=tap1 \
    -device virtio-net-device,netdev=net0,mac=00:da:bc:de:02:11 \
    -kernel hetersec-kernel-arm64/arch/arm64/boot/Image \
    -append "root=/dev/vda console=ttyAMA0" \
    -pidfile vm1.pid 2>&1 | tee vm1.log

Kernel message layer:

Build the message layer kernel modules and copy them to the corresponding VMs:

$ make -C hetersec-kernel/msg_layer
$ ARCH="arm64" make -C hetersec-kernel-arm64/msg_layer/
$ scp hetersec-kernel/msg_layer/msg_socket.ko popcorn@[x86-VM-IP]:~
$ scp hetersec-kernel-arm64/msg_layer/msg_socket.ko popcorn@[arm-VM-IP]:~

Setup the nodes information (in /etc/popcorn/nodes) and install the msg_socket.ko on each VM:

[x86 VM] ~ $ cat /etc/popcorn/nodes
10.2.0.2
10.2.1.2
[arm VM] ~ $ cat /etc/popcorn/nodes
10.2.0.2
10.2.1.2

Install the kernel module on each VM:

[x86 VM] ~ $ sudo insmod msg_socket.ko
[arm VM] ~ $ sudo insmod msg_socket.ko

Now you are ready to run the applications.

Test cases

Build MTD application binaries

Please see detailed instructions here.

Alternatively, we have also pre-built the test cases for multi-ISA MTD. You can just copy the pre-built binaries (in test/pre-built) to your target machines (VMs). More detailed instruction can be found here.

To start the MTD process, we need to configure a list of potential migration points. More instructions can be found here.

Build MVX application binaries

1. Make sure the musl-libc (from the same source code) is installed on both VMs (assume it is in /usr/local/musl)
2. Copy the HeterSec/test/mvx to both VMs.
3. run make

Running test cases (demos)

• Randomly executing code on nodes with different ISA (multi-ISA MTD):

• Running code on two ISA-different nodes (multi-ISA MVX):

To start the MVX variants, you need to use a issue a HeterSec system call to make the process as MVX process. We prepared a shared library, you can run the variant from the ARM node by executing LD_PRELOAD=./loader.so <bin>

hetersec's Issues

basic test error

Hello. When I run mvx basic test on arm, I got this error on x86 kernel, It seems to be the filename or command error, but I don`t know how to fix it. Would you please help me, thanks!
I make this experiment on qume
popcorn@arm:~/mvx$ LD_PRELOAD=./loader.so ./basic

popcorn@x86:~/mvx$ [42121.379789] SYSC_open: filename open.c, ret 0
[42121.383889] SYSC_open: ret -2 (0xfffffffe)
[42121.387782] mvx_update_fd_vtab: Should not happen for fd id -2
[42121.407778] follower ret -9 != 18 from master
[42121.423786] [MVX Violation] kernel/popcorn/mvx_engine.c:328 mvx_follower_wait_exec
[42121.427780] mvx_update_fd_vtab: Should not happen for fd id -1
[42121.443900] follower ret -9 != 18 from master
[42121.455883] [MVX Violation] kernel/popcorn/mvx_engine.c:225 follower_writev
[42121.459780] follower ret -9 != 63 from master
[42121.463961] ------------[ cut here ]------------
[42121.464863] kernel BUG at fs/read_write.c:1007!
[42121.465599] invalid opcode: 0000 [#6] SMP
[42121.466357] Modules linked in: msg_socket
[42121.467057] CPU: 0 PID: 27729 Comm: open Tainted: G D 4.4.137-popcorn+ #1
[42121.467773] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1 04/01/2014
[42121.467773] task: ffff8800bb78c580 ti: ffff8800b9134000 task.ti: ffff8800b9134000
[42121.467773] RIP: 0010:[] [] SyS_writev+0x1c7/0x220
[42121.467773] RSP: 0018:ffff8800b9137e90 EFLAGS: 00010202
[42121.467773] RAX: 000000000000007f RBX: 0000000000000001 RCX: 0000000000000000
[42121.467773] RDX: 000000000000007f RSI: 0000000000000000 RDI: ffff8800b9137f1f
[42121.467773] RBP: ffff8800bac55780 R08: 0000000000000004 R09: ffff8800b9137ea0
[42121.467773] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000007f
[42121.467773] R13: 000000000000007f R14: 0000000000000000 R15: 0000000000000000
[42121.467773] FS: 00007ffff7ffe288(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
[42121.467773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[42121.467773] CR2: 0000000000000003 CR3: 00000000bb74a000 CR4: 0000000000360670
[42121.467773] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[42121.467773] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[42121.467773] Stack:
[42121.467773] 000000000000003f ffffffff813fc25e 0000000000000000 0000000000000000
[42121.467773] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[42121.467773] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[42121.467773] Call Trace:
[42121.467773] [] ? __schedule+0x1ce/0x730
[42121.467773] [] ? entry_SYSCALL_64_fastpath+0x1e/0x8e
[42121.467773] Code: c3 76 2a 48 c1 e0 04 4d 89 e5 48 8d 7c 24 10 48 01 e8 48 83 78 08 7f 4c 0f 46 68 08 48 8b 30 44 89 ea e8 2d a1 13 00 85 c0 74 c6 <0f> 0b 48 89 ef e8 cf 94 ff ff 48 8b
[42121.467773] RIP [] SyS_writev+0x1c7/0x220
[42121.467773] RSP
[42121.467777] ---[ end trace 08c469fbdb08ec3f ]---