rfjakob/earlyoom

earlyoom - Early OOM Daemon for Linux

earlyoom's Issues

Preferred processes

First, I wanted to thank you all for this great software. It's been quite helpful. In particular, Firefox usually eats too much memory and causes my computer to freeze. Earlyoom certainly helps with that.

However, when Firefox updated to v57 recently, it split out its tabs into sub-processes (similiar to Chrome). However, the main Firefox process often has the highest oom_score and is killed by earlyoom. For some reason, doing this orphans the sub-processes and causes a huge CPU spike---and usually a crash.

Everything works fine when a sub-processes is killed. So ideally, earlyoom would be able to selectively kill a tab's sub-process and not the entire Firefox process.

If you have any recommendations about how to accomplish this, I would really appreciate them. Otherwise, I found earlier comments by @joeytwiddle about implementing a "priority kill." So, I forked his code and modified it slightly. I've been testing it for a few days and it seems to do the job quite well. If you'd like---and if it's okay by @joeytwiddle ---I'd be happy to submit a PR.

Can compile but running fails with: Could not find "MemAvailable:"

The new version compiles fine, but when I run it I get:

$ ./earlyoom
earlyoom v0.7-1-g131aa9c
Could not find "MemAvailable:"

Any idea how I can fix this?

(gcc 4.6.3, libc 2.15 on Ubuntu 12.04)

Not killing processes

#78
IMO it was unexpected behavior.

What happens if start earlyoom -m 50?
Is it true that the swap is then not counted at all?

What about SIGTERM?

2018 will end soon, but the signal is still not supported.

Makefile getting version only works when checking out from git repo.

I was hoping someone could fix this line in the make file such that it works if you download a release: VERSION ?= $(shell git describe --tags --dirty)

The following shell code might be a good starting point, but since I don't know make I'm not sure if it is appropriate.
git describe --tags --dirty 2> /dev/null || basename $PWD | sed 's/earlyoom-//'

This assumes you have downloaded a release like so

VERSION=0.11
wget -q https://github.com/rfjakob/earlyoom/archive/v${VERSION}.tar.gz
tar -xzf v${VERSION}.tar.gz
cd earlyoom-${VERSION}

As you can see the version is included in the filename so I propose using the file name as the basis for this. Though, I could see having a VERSION file would be much simpler.

Feel free to ignore me and just delete this issue (serious).

earlyoom crash: Could not convert number: Numerical result out of range

I just noticed earlyoom did this, next time I'll try to provide more debug info:

Out of memory! avail: 138 MiB < min: 786 MiB
Killing process 4084 kworker/u17:3
Out of memory! avail: 155 MiB < min: 786 MiB
Killing process 4084 kworker/u17:3
Out of memory! avail: 157 MiB < min: 786 MiB
Killing process 4084 kworker/u17:3
Could not convert number: Numerical result out of range

But I think earlyoom should never kill kernel threads ;) and not quit on that conversion error.

Wrong victim name

I ran tail /dev/zero, and tail killed, but earlyoom 1.0 displays wrong name

mem avail: 0 MiB (0 %), swap free: 2425 MiB (41 %)
mem avail: 0 MiB (0 %), swap free: 1631 MiB (27 %)
mem avail: 0 MiB (0 %), swap free: 830 MiB (14 %)
Out of memory! avail: 0 MiB < min: 587 MiB
Killing process 1526 sudo
mem avail: 5057 MiB (86 %), swap free: 5564 MiB (94 %)
mem avail: 5196 MiB (88 %), swap free: 5566 MiB (94 %)

Add a hint on how to ignore swap usage

I used -s 100 to practically ignore the free swap memory. Maybe this is worth hinting at when using ./earlyoom -h or man.

What is a good default memory/swap limit?

At various times I have tried 5%, 10% and 15%. What have you guys found works for you?

I found on both my machines, I need to set the OOM killer to engage when I have 300 MB RAM and 300 MB swap remaining. This is the point when the swap starts thrashing.

(Although curiously on my machine with 8 GB RAM and 2 GB swap, if I don't run earlyoom, eventually I will end up with 600 MB RAM and 0 MB swap remaining, and at this point the system becomes unresponsive.)

I wonder if a fixed value of 300 MB RAM and 300 MB swap would satisfy most users. (I don't see why someone with 16 GB RAM would need to engage the OOM killer earlier than someone with 8 GB RAM.)

But obviously on a small machine (e.g. with only 1Gig RAM), the 300MB threshold would be far too high! So perhaps we should be using a percentage, with 300 as the upper limit...?

Interested to hear what values others are using.

/etc/earlyoom.conf instead of /etc/default/earlyoom

earlyoom.conf may be looking like follow:

# This is earlyoom config file

# comment
mem_min_sigkill = 6 %

# comment
swap_min_sigkill = 200 M

# for chromium, comment how it works
decrease_oom_score_adj = False

# comment, how it works
desktop_notifications = False

# comment, examples
prefer = 

# comment
avoid = ssh|Xorg

# comment
self_nice = 0

# comment
self_oom_score_adj = 0

# comment
debug = False

Advantages is it more convenient for users, IMHO.
Users may easy find all defaults and what means each parameter.

What is disadvantages?

Unexpected behavior: unknown version instead of 1.0

subj

Add a warning before a process is killed

How about a feature that would warn me if the free memory gets close to the minimum? This way I could first try stopping the process myself.

Priority kill

Basically I always run out of memory in the same scenario: I open Dota, I open a few Chromium tabs, and eventually when Dota starts the game I run out of memory. I want Chromium to be killed, not Dota.

So for me either banning a process from being killed, or prioritizing a process to get killed would do the trick. It would be nice if there was a configuration for that.

Sleep(10)

Closed too early. The discussion should be continued.

Firstly, sleep(10) not declared and users are not warned.

Secondly, sleep(10) is too long, sleep(3) is enough.

Not killing processes

I installed the package from AUR, enabled (and started) the service and manually started it with
sudo earlyoom -m 50, but it does not work.

I started a jupyter notebook and got the memory down to 2% when it stopped (because it finished what it was loading), but earlyoom never even touched it.

output.txt

What am I doing wrong?

Serious bug in get_entry_fatal

In 0.10, get_entry_fatal does not return val in the good case.

This code only works by accident with -O0.

[FR] - Get notification when killing a process

Hello,

I really like earlyoom, but sometimes it kills some of my processes without no notification. I have to wonder what happened. e.g. python program is just silently ended without no information about being killed or whatsoever.

It would be really awesome if we could have some event on which notification would be triggered, e.g. the most simple one like:

notify-send 'EarlyOOM' 'Process {$name}:{$PID} killed' --icon=dialog-information

right now it could be implemented as a workaround, when watching stdout of systemd unit. But some nicer solution would be great.

Not informative output if swap enabled

Out of memory! avail: 0 MiB < min: 587 MiB

One this line is not enaugh. Earlyoom not displays swap status info.

I want to see stdout like this:

mem avail: 0 MiB < min: 587 MiB
swap free: 500 MiB < min: 587 MiB
Preventing OOM: killing process...

Zram support

What about zram support and mem_used_total as a trigger?
Now earlyoom does not help when writing the data with high entropy in zram swap.

Error: Could not find a process to kill

Got some process to eat the whole of RAM. earlyoom appears to have crashed.

If it's relevant, I don't have swap.

$ sudo systemctl status earlyoom
● earlyoom.service
   Loaded: loaded (/etc/systemd/system/earlyoom.service; enabled)
   Active: failed (Result: exit-code) since Mon 2016-02-08 00:05:04 CET; 3 days ago
  Process: 699 ExecStart=/usr/local/bin/earlyoom (code=exited, status=9)
 Main PID: 699 (code=exited, status=9)

Feb 08 00:05:04 zoob earlyoom[699]: earlyoom v0.7-10-gb4c6bc0
Feb 08 00:05:04 zoob earlyoom[699]: mem total: 31600 MiB, min: 3160 MiB (10 %)
Feb 08 00:05:04 zoob earlyoom[699]: swap total: 0 MiB, min: 0 MiB (10 %)
Feb 08 00:05:04 zoob earlyoom[699]: Error: Could not find a process to kill
Feb 08 00:05:04 zoob systemd[1]: earlyoom.service: main process exited, code=exited, status=9/n/a
Feb 08 00:05:04 zoob systemd[1]: Unit earlyoom.service entered failed state.

Low memory is not OOM

Out of memory (OOM) is an often undesired state of computer operation where no additional memory can be allocated for use by programs or the operating system. Such a system will be unable to load any additional programs, and since many programs may load additional data into memory during execution, these will cease to function correctly. This usually occurs because all available memory, including disk swap space, has been allocated.

In normal operation, earlyoom should not occur at all OOM, since preventing OOM is the main task of earlyoom, not a reaction to OOM that has already occurred.

What about replace "Out of memory!" in earlyoom stdout with "Preventing OOM!"?

Installation of earlyoom.default

The default config installs to /etc/default/earlyoom.default

The service file expects it here:

EnvironmentFile=-/etc/default/earlyoom

This should fix it up:

--- Makefile.orig	2017-11-29 19:01:21.537246323 -0500
+++ Makefile	2017-11-29 19:01:25.327274351 -0500
@@ -36,8 +36,8 @@
 	sed "s|:TARGET:|$(PREFIX)$(BINDIR)|g;s|:SYSCONFDIR:|$(SYSCONFDIR)|g" $< > $@
 
 install-default: earlyoom.default
-	install -d $(DESTDIR)$(SYSCONFDIR)/default/
-	install -m 644 $< $(DESTDIR)$(SYSCONFDIR)/default/
+	install -d $(DESTDIR)$(SYSCONFDIR)/default
+	cp $< $(DESTDIR)$(SYSCONFDIR)/default/earlyoom
 
 install-bin: earlyoom
 	install -d $(DESTDIR)$(PREFIX)$(BINDIR)/

Thanks!

broken install

$ git clone https://github.com/rfjakob/earlyoom.git
Клонирование в «earlyoom»…
remote: Counting objects: 568, done.
remote: Compressing objects: 100% (85/85), done.
remote: Total 568 (delta 69), reused 101 (delta 43), pack-reused 439
Получение объектов: 100% (568/568), 136.47 KiB | 0 bytes/s, готово.
Определение изменений: 100% (326/326), готово.
$ cd earlyoom
$ make
cc -Wall -Wextra -DVERSION="v1.1-12-g647107d" -g -fstack-protector-all -std=gnu99 -o earlyoom meminfo.c kill.c main.c sanitize.c
pandoc MANPAGE.md -s -t man > earlyoom.1
/bin/sh: 1: pandoc: not found
Makefile:22: ошибка выполнения рецепта для цели «earlyoom.1»
make: *** [earlyoom.1] Error 127

Notification shows wrong process name

On Ubuntu 16.04, running as user (not as service) the notify-osd notification reported the correct PID but the wrong process name "gnome-system-monitor" when in fact killing whatsapp. The terminal output shows the correct PID and name.

Sleep 10 causes earlyoom not prevents OOM

I ran while true; do tail /dev/zero; done, and eventually Xorg has been killed because earlyoom slept.

support logging to separate file

would be nice to be able to do this when running in daemon mode.

Reduce badness of certain processes using regexp

I found earlyoom was too often killing the main firefox process, or the main chrome process, when I would really rather it had killed just one or two of the chrome tabs I had open, instead of killing a whole browser session!

My approach to this has been to reduce the badness of processes matching a certain regexp. I compare the regexp against the process cmdline, rather than just the process name, so that I can be kind to the main process, but not to tabs or extensions. (I divide badness by 32.)

My fork based on your older code is here: https://github.com/joeytwiddle/earlyoom/tree/logging_usleep

It has been working for me, and I found it very useful (so that I don't lose my whole browser when I have 100 tabs open). I hope you also like the idea, and will consider merging it in! (I could create a PR.)

Here is a more up-to-date version which is merged with your oom_adj code (which is kind of overkill!): https://github.com/joeytwiddle/earlyoom

However now I am interested in reducing the badness of recently-created or running processes. (Because I would prefer if earlyoom attacks chrome tabs I haven't used for a long time, rather than tabs I am using right now.) (I suspect recently started processes are much easier to detect in Linux than recently used... ;) ) That's now in #8

Misestimated free RAM

Earlyoom killed my app because I don't have enough free RAM:

$ sudo journalctl -b | tail
...
Jul 04 19:52:50 xylix earlyoom[699]: Out of memory! avail: 3157 MiB < min: 3160 MiB
Jul 04 19:52:50 xylix earlyoom[699]: Killing process 20118 (VirtualBox)
...

But I have more than enough (~19 GB) in disposable buffers/cache.

$ free -h
             total       used       free     shared    buffers     cached
Mem:           30G        27G       3.6G        13G       228M        15G
-/+ buffers/cache:        11G        19G
Swap:           0B         0B         0B

Decrease CPU usage

There is no sense to check memory 10 times per second if there is more than one gigabyte of available memory. You can optimize the frequency of memory checks to decrease the frequency of memory checks and decrease CPU usage. Now the sleep period is 0.1. You can find the time until the next check as follows:

t1 = MemAvailable / 4000000
t2 = SwapFree / 1000000
sleep_until_next_mem_check = t1 + t2

I implemented a similar algorithm in nohang, which is OOM preventer written in Python. If nohang checked memory 10 times per second, then the CPU usage would be too large. Now when using this algorithm for calculating the period between memory checks, the CPU usage is significant only at a low memory level. If there is a lot of memory available, then nohang can use the processor even less than earlyoom. Demo https://youtu.be/8vjeolxw7Uo

I think it's possible to improve earlyoom in the same way. In most cases it will reduce the CPU usage by an order of magnitude.

Notify-send as root

earlyoom -N 'sudo -u YOUR_USER DISPLAY=:0 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/YOUR_USER_ID/bus notify-send' is not needed

earlyoom -N 'sudo -u YOUR_USER DISPLAY=:0 notify-send'
is completely enough

Notify-send slowdown

...if & not in the end of the command.

Replace
"-i dialog-warning 'earlyoom' 'Killing process %d %s'", victim_pid, victim_name);
with
"-i dialog-warning 'earlyoom' 'Killing process %d %s' &", victim_pid, victim_name);
will fix the problem

Set when to kill OOM when using `systemctl` (systemd)

Hey, just installed OOM, and I started earlyoom by doing sudo systemctl enable earlyoom and sudo systemctl start earlyoom. Where do I set the -m option for earlyoom?

Inside earlyoom.service:

ExecStart=/usr/local/bin/earlyoom $EARLYOOM_ARGS

Not sure where $EARLYOOM_ARGS is

del -k

IMO -k option is deprecated and should be deleted at all.

How to free up caches/buffers?

earlyoom starts killing processes when just 10% of memory "available".

$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G         12G        660M        883M        2,7G        2,0G
Swap:            0B          0B          0B

How to force system to free up more buffer/caches under certain conditions so that it frees more memory to become available?

About 0.6MB RSS

please, fix earlyoom.1

Would it be useful to use rate of swapin as a threshold?

Hello,

In addition to the absolute amount of memory swapped out, would it make sense to use the quantity of swap-in or the amount of time spent waiting on swap-in (if available) as a trigger for kill?

My use case are servers that have overcommit off, which means they are very conservative on how much memory can be committed, and extent of concurrency of multiple processes winds up penalized. When one considers simple Python, Go, or even C programs can allocate hundreds of megabytes of virtual memory that they never use (cat does this for me, in fact), this is a meaningful problem.

Thus, the amount of swap I may wish to allocate be somewhat large.

However, these servers have soft-real-time performance requirements. It's not desirable to actually let memory be swapped in. It would be better to start killing things if swap-in pressure gets beyond the residual.

Thoughts?

Way to tell earlyoom not to kill some process (respect oom_score_adj)

Hello,

I have some processes which can consume a lot of memory, but I don't want them to be killed by earlyoom. And I would also like to be able to set some processes to be preferably killed. So it would be really great if earlyoom could be told to respect oom_score_adj. I know there is problem for you with Chrome setting oom_score_adj too high, but for me this is desired behavior (rather kill browser tab than some memory hungry - but important - process, computation, compilation, etc.). I know I can compile it with USE_KERNEL_OOM_KILLER, which would respect oom_score_adj, but I want to use userspace killing mechanism (I don't want to enable sysrq "f" trigger and I have also bad experience with kernel oom killer in general).

Ideally there should be some more fine grained setting, like list of processes (by name) for which earlyoom should ignore oom_score_adj (this would solve your problem with Chrome) and setting how much should be oom_score_adj taken into account when deciding what to kill. But for the start, even simple (preferably runtime) option to respect/don't respect oom_score_adj would be enough.

Cannot compile due to usage of long int

$ make
cc -Wextra -DVERSION=\"v0.10-4-g8ea46ee\" -g -o earlyoom main.c meminfo.c kill.c
main.c: In function ‘main’:
main.c:131:11: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘long int’ [-Wformat=]
    printf("mem avail: %lu MiB (%d %%), swap free: %lu MiB (%d %%)\n",
           ^
kill.c: In function ‘get_process_stats’:
kill.c:86:12: warning: use of assignment suppression and length modifier together in gnu_scanf format [-Wformat=]
  fscanf(f, "%*lu %lu", &(p.vm_rss));
            ^

Ubuntu 16.04

Never-Kill-RegEx

Last suggestion for today:

While testing the script (setting the minimum RAM to ca. 95%) I noticed that it would even kill my desktop environment in order to free up some space. I guess there are more "realistic" cases where an option Do not kill those processes, whatever happens, would be useful, though. So how about adding this? :)

Why -p option is not enabled by default?

subj

"make install" does not install man page

When installing earlyoom, the program's manual page (earlyoom.1) is not installed. The attached, updated Makefile adds targets for "install-man" and "uninstall-man" which can be called directly or by an uninstall process.
Makefile-earlyoom.txt

Default syntax

Hi,

I'm running Debian Stretch and my regex skills are poor.

I put the following in /etc/default/earlyoom
``EARLYOOM_ARGS="-m 15 -s 100 -r 5 --prefer '(firefox-esr|Web\sContent|gvfsd-mtp)'"

If I do ps ax | grep earlyoom, I see:

/usr/bin/earlyoom -m 15 -s 100 -r 5 --prefer (firefox-esr|WebsContent|gvfsd-mtp)

which looks like the \s I did to match a space is being misinterpeted.

I am trying to match any of the three piggies: firefox-esr, Web Content, gvfsd-mtp and to ignore the state of Swap and kill based on RAM usage.

Am I doing it correctly?

Thanks!
Mark

Configuration for service

Is there possibility to set custom percentages and run as service?

Avoid killing my active Chrome tab

I am a tab junkie. (130 tabs currently open in Chrome, and sometimes I run Firefox alongside it.)

I have started noticing a recurring issue than when my memory is nearly full and I open a heavy webpage, after a few seconds the tab I am working in gets killed!

The behaviour I would prefer is that the new tab would be left running, even though it is using a lot of memory, and an older tab/process would be killed.

(Example "heavy" pages are Google Maps, Facebook, and Google Drive spreadsheets.)

So I would like a process to have its badness reduced when it first starts, but this would gently slide back to the default over time.

I wonder if the start time of a processes is available from se.exec in /proc/*/sched_start... (Edit: No, I found what I needed in .../stat.)

Admittedly my proposed solution will NOT help if I was to switch to an old tab/app and cause it to start consuming a lot of memory. For that situation, it might be more appropriate track which process the user has been recently interacting with. But I fear that might not be easy to detect. A simpler alternative would be to reduce badness of active processes, hence favouring the destruction of processes that have been idle for some time.

Alias for -h

This is a really small one. When you do --help it should show the help page, and when there is an unrecognized option it should suggest -h.
Cheers <3

Use linux/sysinfo.h ?

Parsing /proc/meminfo 10 times a second, could you for an even lighter footprint benefit from using linux/sysinfo.h interface?

http://stackoverflow.com/questions/349889/how-do-you-determine-the-amount-of-linux-system-ram-in-c

How much memory does earlyoom use?

1.8 MiB (Debian 9 x86_64)
2.0 MiB (Fedora 28)
2.2 MiB (Debian 8 i386, https://i.imgur.com/MaRPWwE.png)
Not about 0.6 MiB.

Sanitize process name for GUI notification helper

Is it not enough to drop out only ', ", backslash and backtick? Why do you drop out all Unicode?

Unexpected behavior when add swap after start earlyoom

sudo ./earlyoom
[sudo] пароль для user:
earlyoom (unknown version)
mem total: 5875 MiB, min: 587 MiB (10 %)
swap total: 0 MiB, min: 0 MiB (10 %)
mem avail: 4912 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4910 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4910 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4910 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4906 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4898 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4901 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4902 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4902 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4905 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4905 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4905 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4905 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4905 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4878 MiB (83 %), swap free: 5875 MiB (100 %)
mem avail: 4878 MiB (83 %), swap free: 5875 MiB (100 %)
mem avail: 4878 MiB (83 %), swap free: 5875 MiB (100 %)
mem avail: 4879 MiB (83 %), swap free: 5875 MiB (100 %)
mem avail: 4879 MiB (83 %), swap free: 5875 MiB (100 %)
...
mem avail: 0 MiB (0 %), swap free: 1034 MiB (17 %)
mem avail: 0 MiB (0 %), swap free: 927 MiB (15 %)
mem avail: 0 MiB (0 %), swap free: 824 MiB (14 %)
mem avail: 2 MiB (0 %), swap free: 698 MiB (11 %)
mem avail: 0 MiB (0 %), swap free: 604 MiB (10 %)
mem avail: 0 MiB (0 %), swap free: 501 MiB (8 %)
mem avail: 0 MiB (0 %), swap free: 396 MiB (6 %)
mem avail: 0 MiB (0 %), swap free: 273 MiB (4 %)
mem avail: 0 MiB (0 %), swap free: 177 MiB (3 %)
mem avail: 0 MiB (0 %), swap free: 84 MiB (1 %)
Out of memory! avail: 0 MiB < min: 587 MiB
Killing process 2456 python3
mem avail: 1606 MiB (27 %), swap free: 472 MiB (8 %)
mem avail: 5184 MiB (88 %), swap free: 5572 MiB (94 %)
mem avail: 5177 MiB (88 %), swap free: 5573 MiB (94 %)
mem avail: 5173 MiB (88 %), swap free: 5574 MiB (94 %)
mem avail: 5173 MiB (88 %), swap free: 5574 MiB (94 %)

I expect that earlyoom kill victim if avail = 10%, not 0 MiB.

Options for users who want to use swap too

My use case differs slightly from yours. I have a swap partition and I want to use it.

I want earlyoom to trigger when both the swap and the memory are nearly full.

I have been using this adapted version of earlyoom for a couple of weeks now, and it has served me well: https://github.com/joeytwiddle/earlyoom/tree/wait_for_swap_to_fill

However it is not compatible with your use case!

I wonder if we should make the effort to combine both these requirements together, so that one package can serve both users with your requirements, and users with my requirements.