firecracker-microvm/nsdi2020-data

Firecracker benchmarking code

This repository contains all the scripts, source code, and data used for our NSDI 2020 paper on "Firecracker: Lightweight Virtualization for Serverless Applications".

Build the pre-requisites

The ./prep directory contains scripts and other tools required to run the tests. Most tests uses minimal OS images build with linuxkit. It also contains a slightly modified version of firecracker and builds a new, statically linked binary for qemu.

The following should build everything:

cd ./prep
make

Binaries (and BIOS) are placed into ./bin and kernel and root filesystem images are placed into ./img.

A working docker and go installation is required for building, but not for running the tests and you should be able to copy the ./bin and ./img directories to a target system to run tests there without these pre-requisites.

Prepare the host

This assumes you are running on a Ubuntu system.

Install some host side utilities

sudo apt update
sudo apt install -y \
    iperf3 \
    jq \
    hwloc-nox
    numactl \

Running the tests

There are a bunch of scripts in ./scripts. They assume you are in the ./scripts directory when you run them.

cd ./scripts
./run-all.sh ../data/test

configures the host system and runs most of the tests. Test results are stored in the directory provided.

nsdi2020-data's Issues

QEMU MicroVM: why such bad results and why not in the Firecracker paper?

Dear Firecracker paper authors,

While looking at some other paper (rellermeyer/container_benchmarks#1), I noticed the unexplainably bad results of QEMU's MicroVM in your GitHub repo.

Here is the QEMU MicroVM command line: https://github.com/firecracker-microvm/nsdi2020-data/blob/cf1b30f674523ee44363f74e16bcb301ffb10598/scripts/util_start_qemu_uvm.sh

Here is the commit that added QEMU MicroVM data: b35147f

You also updated all the plots, some of them include QEMU MicroVM perf, especially this one: https://github.com/firecracker-microvm/nsdi2020-data/blob/master/data/m5d.metal/boot-serial-all.pdf

If I'm reading the plot correctly, and if I'm not confusing two shades of "light blue", then QEMU MicroVM exhibits extremely bad boot time (~600ms, about 2.5x worse than normal QEMU).

I also notice that your final paper https://www.usenix.org/system/files/nsdi20-paper-agache.pdf doesn't mention QEMU MicroVM at all (this may be just timing issues, since your GitHub updates are dated with 9. February and the NSDI conference was on 25. February).

So I have two questions:

• Do you have any explanations on QEMU MicroVM boot times? Did you maybe expand on this in some follow-up paper, video, blog, etc?
• How can you explain the fact that QEMU's MicroVM behaves worse than normal QEMU ( with normal machine type)?
• Why didn't you include any mentions of QEMU's MicroVM in the final paper?