firecracker-microvm/firecracker

Secure and fast microVMs for serverless computing.

Our mission is to enable secure, multi-tenant, minimal-overhead execution of container and function workloads.

Read more about the Firecracker Charter here.

What is Firecracker?

Firecracker is an open source virtualization technology that is purpose-built for creating and managing secure, multi-tenant container and function-based services that provide serverless operational models. Firecracker runs workloads in lightweight virtual machines, called microVMs, which combine the security and isolation properties provided by hardware virtualization technology with the speed and flexibility of containers.

Overview

The main component of Firecracker is a virtual machine monitor (VMM) that uses the Linux Kernel Virtual Machine (KVM) to create and run microVMs. Firecracker has a minimalist design. It excludes unnecessary devices and guest-facing functionality to reduce the memory footprint and attack surface area of each microVM. This improves security, decreases the startup time, and increases hardware utilization. Firecracker has also been integrated in container runtimes, for example Kata Containers and Weaveworks Ignite.

Firecracker was developed at Amazon Web Services to accelerate the speed and efficiency of services like AWS Lambda and AWS Fargate. Firecracker is open sourced under Apache version 2.0.

To read more about Firecracker, check out firecracker-microvm.io.

Getting Started

To get started with Firecracker, download the latest release binaries or build it from source.

You can build Firecracker on any Unix/Linux system that has Docker running (we use a development container) and bash installed, as follows:

git clone https://github.com/firecracker-microvm/firecracker
cd firecracker
tools/devtool build
toolchain="$(uname -m)-unknown-linux-musl"

The Firecracker binary will be placed at build/cargo_target/${toolchain}/debug/firecracker. For more information on building, testing, and running Firecracker, go to the quickstart guide.

The overall security of Firecracker microVMs, including the ability to meet the criteria for safe multi-tenant computing, depends on a well configured Linux host operating system. A configuration that we believe meets this bar is included in the production host setup document.

Contributing

Firecracker is already running production workloads within AWS, but it's still Day 1 on the journey guided by our mission. There's a lot more to build and we welcome all contributions.

To contribute to Firecracker, check out the development setup section in the getting started guide and then the Firecracker contribution guidelines.

Releases

New Firecracker versions are released via the GitHub repository releases page, typically every two or three months. A history of changes is recorded in our changelog.

The Firecracker release policy is detailed here.

Design

Firecracker's overall architecture is described in the design document.

Features & Capabilities

Firecracker consists of a single micro Virtual Machine Manager process that exposes an API endpoint to the host once started. The API is specified in OpenAPI format. Read more about it in the API docs.

The API endpoint can be used to:

• Configure the microvm by:
  • Setting the number of vCPUs (the default is 1).
  • Setting the memory size (the default is 128 MiB).
  • Configuring a CPU template.
• Add one or more network interfaces to the microVM.
• Add one or more read-write or read-only disks to the microVM, each represented by a file-backed block device.
• Trigger a block device re-scan while the guest is running. This enables the guest OS to pick up size changes to the block device's backing file.
• Change the backing file for a block device, before or after the guest boots.
• Configure rate limiters for virtio devices which can limit the bandwidth, operations per second, or both.
• Configure the logging and metric system.
• [BETA] Configure the data tree of the guest-facing metadata service. The service is only available to the guest if this resource is configured.
• Add a vsock socket to the microVM.
• Add a entropy device to the microVM.
• Start the microVM using a given kernel image, root file system, and boot arguments.
• [x86_64 only] Stop the microVM.

Built-in Capabilities:

• Demand fault paging and CPU oversubscription enabled by default.
• Advanced, thread-specific seccomp filters for enhanced security.
• Jailer process for starting Firecracker in production scenarios; applies a cgroup/namespace isolation barrier and then drops privileges.

Tested platforms

We test all combinations of:

Instance	Host OS & Kernel	Guest Rootfs	Guest Kernel
c5n.metal	al2 linux_4.14	ubuntu 22.04	linux_4.14
m5n.metal	al2 linux_5.10		linux_5.10
m6i.metal	al2023 linux_6.1
m6a.metal
m6g.metal
m7g.metal

Known issues and Limitations

• The pl031 RTC device on aarch64 does not support interrupts, so guest programs which use an RTC alarm (e.g. hwclock) will not work.

Performance

Firecracker's performance characteristics are listed as part of the specification documentation. All specifications are a part of our commitment to supporting container and function workloads in serverless operational models, and are therefore enforced via continuous integration testing.

Policy for Security Disclosures

The security of Firecracker is our top priority. If you suspect you have uncovered a vulnerability, contact us privately, as outlined in our security policy document; we will immediately prioritize your disclosure.

FAQ & Contact

Frequently asked questions are collected in our FAQ doc.

You can get in touch with the Firecracker community in the following ways:

• Security-related issues, see our security policy document.
• Chat with us on our Slack workspace Note: most of the maintainers are on a European time zone.
• Open a GitHub issue in this repository.
• Email the maintainers at [email protected].

When communicating within the Firecracker community, please mind our code of conduct.

firecracker's Issues

Restful API

Firecracker is controlled via a RESTful API.

• Figure out how to pass file descriptors atomically within the API message. #14
• Create API definition #4
• Implement the application program interface #15

Create Base Virtual Machine Manager Process

Outcome:

• create VM with KVM & assign memory;

• do interrupt emulation;

• boot a basic guest Kernel image + qboot;

• fake a serial console.

• #17

Research File Descriptors Passing

• Figure out how to pass file descriptors atomically within the API message.
• Do a short PoC on the mechanism.
• Investigate and PoC the mechanism using RUST.

VirtIO Network Virtualization

Implement VirtIO network virtualization, based off the crosVM implementation.

Done when the guest OS can see & use a network device.

Limit the size of a virtuqueue to its maximum possible

Currently the access modifier of the 'size' member of a virtqueue could lead to the maximum size becoming smaller than its actual size which would in turn invalidate the queue (effect: unlimited number of error messages).
Explicitly check that the size does not exceed max_size before setting it.
We should also put a limit on the number of messages displayed in case queue becomes invalid. See is_valid function from virtio/queue.rs.

High Level Comparison between Anthony's Demo and Crosvm

Unit Test Catch-Up

With the unit test coverage tooling complete:

• define coverage targets for all crates, and
• bring unit test coverage up to par in existing crates.
  • #80 VirtIO
  • #85 Legacy devices
  • #86 KVM
  • #87 net_util
  • #95 sys_util

VCPU Support

VMM: Enable Serial

I/O Rate Limiting

We will need network and storage rate limiting since we don't trust the guests (CPU is handled by c-groups).

• Change current PoC to use timer_fd as a source for refreshing the token buckets when limiter is at capacity (while not at capacity – when budget is still available – buckets are refreshed without the use of an external timer, it is done in the transaction processing flow). TimerFDs are a linux kernel feature and have a C interface so either use an external rust crate, or implement our own. Will try to find an external crate of high quality that also suits our needs.
• Design and refactor the currently very hacky PoC to end up with proper/clean Block IO rate limiting.
• Block IO rate limiting proper implementation might spill into this week as well.
• Implement Network rate limiting – once we get the current design and code right for Block IO rate-limiting, network implementation should be smooth.

Create a VM and Boot a Linux 4.9 kernel

Use uncompressed kernel image.
No device emulation.

Storage Virtualization

Implement VirtIO storage virtualization, based off the crosVM implementation.

Done when the guest OS can see & use a block storage device

Investigate VM Initialization and BIOS Emulation

Understand the required initializations for a vm to boot (kernel offset in guest memory, registry initialization, ..)

We need to better understand the BIOS emulation purpose and its boot time impact.
https://github.com/bonzini/qboot
We will afterwards decide if we need a bios to boot or not.

VMM: Enable Syslog

Enforce virtio's limits specification on virtqueues

As per the virtio 1.0 specification, the maximum queue size has to be a power of 2 and be less then 32768 (see chapter 2.4 Virtqueues). Insert a check when creating a new queue that makes sure the specification is followed.

Refactor crosvm's KVM Crate

Remove memory allocation and code not related to kvm kernel module interface from the crate.

Add API for live-update

Emulate ACPI

Emulate the "Advanced Configuration and Power Interface" to support power management features.

Investigate and Fix the Setup of the LAPIC Registers

Currently for setting the lapic state registers we use std::mem::transmute. Based on the documentation: 'transmute is incredibly unsafe. There are a vast number of ways to cause undefined behavior with this function. transmute should be the absolute last resort'. Moreover, when trying to set the APIC_LVT0 register inside a zeroed out array, rust-gdb does not show any change in the registers array.

Check the Number of Guest Physical Memory Slots Against the Maximum Allowed

Currently there is no mechanism for checking that the number of memory slots does not exceed the maximum allowed (from kvm api documentation, this can be done with KVM_CAP_NR_MEMSLOTS).

Create API Definition

• Identify API needs and design a model for interacting with firecracker's resources
• Ramp-up on swagger
• Describe the API using swagger (OpenAPI Specification)

Virtual Socket Implementation

Add a virtual socket implementation for host - guest communication. VirtIO/VSOCK is a first option for this.

VMM: Integrate i8042 From Crossvm

Optimize the size of the kernel config used for Firecracker with VIRTIO

The minimal Linux kernel config we have been using so far does not have the relevant VIRTIO options enabled. Moreover, simply adding these options does not lead to a successful boot for a simple filesystem image created with deboostrap.

The boot process completes successfully for a larger config file (such as the one created by make defconfig + VIRTIO options), but we would like to disable all unnecessary features (or as many of them as possible).

Add kernel support for kvmclock

The current fallback clocksource of the kernel is 'tsc' which counts the number of cycles since reset. As a consequence, 'busybox date' is incorrect (constant). By enabling kvm clock we should obtain a synchronized date with the host.

Investigate Bug in Setting up the FPU through the KVM Interface

Currently, as part of the x86 configuration prior to booting the kernel, the floating point registers gets set up by calling KVM_GET_FPU and KVM_SET_FPU. After trying to alter the mxcsr and fcw registers, the only one that gets set is the fcw. Only when the order of the mxcsr in the kvm structure is changed by bringing it closer to fcw, its value gets set.

Support Boot to Userspace in Less than 150ms

This is owned by the AL Distro team, but we may need to do work here. Current potential work items:

VMM: Add Support for Timers

Firecracker Secure Start & Jailing

When running a VM, Firecracker must be an unprivileged, contained process. If it's started as root, it should drop privileges and jail it's self as soon as possible.

Emulate RTC

CI Build-Time Tests

Via our CI system, ensure that PR/Merge actions are preceded by:

• Successful build.
• Unit testing.
• Rust FMT.

VMM: Fix Style Issues via Cargo FMT to Establish a Baseline

Emulate SMP

Emulate "Symmetric Multiprocessing" to support multiple CPUs.

Boot the Amazon Linux Kernel / Distro

To begin with, customers will expect to boot their current Amazon Linux images (or a modified version there-of).

VMM: Propagate Errors and Don't Panic

Boot Linux Kernel + Busybox

Correct the help message for the mem_size argument

Should be MiB instead of MB.

Move device control/epoll logic to a single thread

The crosvm device model uses one control thread per each VIRTIO device. We want to move this logic to a single thread, which handles all devices.

VM Initialization and Boot Protocol Implementation

[ ] #27 BIOS emulation
[ ] #28 Implement the boot protocol for linux kernel compressed images

Serialize/Deserialize the Machine Model (including all emulated devices)

Legacy Device Emulation (With Live Update)

We need a minimal device model, and emulate:

• Real-Time Clock
• Power-Management
• UART, for the serial console (for development purposes, should be turned off in the release)

Implement the Boot Protocol for Linux Kernel Compressed Images

Ensure support for Lambda's time/clock system calls

Time sys calls used by Lambda:

• clock_gettime(CLOCK_REALTIME)
• clock_gettime(CLOCK_MONOTONIC)
• gettimeofday()

VMM: Connect Stdin to Serial

Generate Unit Test Coverage Results for all Crates

Prints output to stdout.

Enforce Memory Limitations While Registering a mmio Device

Currently the device manager starts off with a memory base address which gets incremented with every mmio device registration. There is no limit enforcement on the space the device manager could use for that. Investigate and possibly fix if memory overlaps are possible.

API - Implement the Application Program Interface

VMM: Replace poll() with epoll()

The current polling mechanism is poll() which has the following disadvantages:

• copies the used fd set each time poll() is called
• cannot pass private data per fd

We should use epoll to overcome these problems.

Definition of done:

• Have the vmm use epoll() for the async loop.

Pause/Resume VMs using the API

KVM Sample in Rust