linecode/oomd

A userspace out-of-memory killer

oomd

oomd is userspace Out-Of-Memory (OOM) killer for linux systems.

Background

Out of memory killing has historically happened inside kernel space. On a memory overcommitted linux system, malloc(2) and friends will never fail. However, if an application dereferences the returned pointer and the system has run out of physical memory, the linux kernel is forced take extreme measures, up to and including killing processes. This is typically a slow and painful process because the kernel spends an unbounded amount of time swapping in and out pages and evicting the page cache. Furthermore, configuring policy is not very flexible while being somewhat complicated.

oomd aims to solve this problem in userspace. oomd leverages PSI and cgroupsv2 to monitor a system holistically. oomd then takes corrective action in userspace before an OOM occurs in kernel space. Corrective action is configured via a flexible plugin system, in which custom code can be written. By default, this involves killing offending processes. This enables an unparalleled level of flexibility where each workload can have custom protection rules. Furthermore, time spent churning pages in kernelspace is minimized. In practice at Facebook, we've regularly seen 30 minute host lockups go away entirely.

Building and installing

Note that oomd requires PSI to function. This kernel feature has not yet been upstreamed (as of 7/18/18).

oomd currently depends on meson, libfolly, and jsoncpp. The dependency on folly will soon be removed (as of 7/18/18).

$ git clone https://github.com/facebook/oomd
$ cd oomd
$ meson build && ninja -C build
$ cd build && sudo ninja install

Configuring oomd

oomd receives runtime configuration from two sources: environment vars and a config file. You typically do not need to change the default environment values. However, to finely tune oomd, you may want to.

Config file

Default location: /etc/oomd.json

Example config:

{
    "cgroups": [
        {
            "target": "system.slice",
            "kill_list": [
                {"chef.service": { "kill_pressure": "60", "max_usage": "100" } },
                {"sshd.service": { "max_usage": "inf" } }
            ],
            "oomdetector": "default",
            "oomkiller": "noop"
        },
        {
            "target": "workload.slice",
            "kill_list": [],
            "oomdetector": "default",
            "oomkiller": "default"
        }
    ],
    "version": "0.2.0"
}

This example config describes the following:

• oomd shall monitor two cgroups: 'system.slice' and 'workload.slice'
• system.slice
  • When an OOM is detected, first look at the kill list and attempt to kill chef.service first if it's generating >= 60% of total memory pressure OR it's using >= 100MB of memory
  • Never kill 'sshd.service'
  • Use the default oomdetector
  • Use the custom 'noop' killing plugin
• workload.slice
  • Do not specify kill list policy
  • Use the default oomdetector
  • Use the default oomkiller

Environment variables

OOMD_INTERVAL

• How often oomd polls the system
• Unit: seconds (integer)
• Default: 5

OOMD_VERBOSE_INTERVAL

• How often oomd logs verbose state to /dev/kmsg
• Unit: seconds (integer)
• Default: 300

OOMD_POST_KILL_DELAY

• How long oomd will sleep after performing a corrective action
• Unit: seconds (integer)
• Default: 15

OOMD_THRESHOLD

• How sensitive oomd will be to slow growing OOMs
• Unit: none (integer)
• Values: [0, 100], 0 being extremely sensitive and 100 being not sensitive
• Default: 60

OOMD_HIGH_THRESHOLD

• How sensitive oomd will be to fast growing OOMs
• Unit: none (integer)
• Values: [0, 100], 0 being extremely sensitive and 100 being not sensitive
• Default: 80

OOMD_HIGH_THRESHOLD_DURATION

• How quick oomd will be to declare a fast growing OOM
• Unit: seconds (integer)
• Default: 10

OOMD_LARGER_THAN

• When killing via size heuristic, only kill targets using greater than X% of total used memory
• Unit: percent (integer)
• Values: [0, 100]
• Default: 50

OOMD_GROWTH_ABOVE

• When killing via growth heuristic, only consider P(X) top growers
• Unit: percent (integer)
• Values: [0, 100]
• Default: 80

OOMD_MIN_SWAP_PCT

• When swap is enabled, perform corrective actions when X% of swap is remaining
• Unit: percent (int)
• Default: 15

Tests

oomd depends on gtest/gmock to run tests. Installing gtest/gmock from master is preferred.

If meson detects gtest/gmock is installed, meson will generate build rules for tests.

$ cd oomd
$ rm -rf build
$ meson build && ninja test -C build

License

oomd is GPL 2 licensed, as found in the LICENSE file.