rfjakob / earlyoom Goto Github PK
View Code? Open in Web Editor NEWearlyoom - Early OOM Daemon for Linux
License: MIT License
earlyoom - Early OOM Daemon for Linux
License: MIT License
First, I wanted to thank you all for this great software. It's been quite helpful. In particular, Firefox usually eats too much memory and causes my computer to freeze. Earlyoom certainly helps with that.
However, when Firefox updated to v57 recently, it split out its tabs into sub-processes (similiar to Chrome). However, the main Firefox process often has the highest oom_score and is killed by earlyoom. For some reason, doing this orphans the sub-processes and causes a huge CPU spike---and usually a crash.
Everything works fine when a sub-processes is killed. So ideally, earlyoom would be able to selectively kill a tab's sub-process and not the entire Firefox process.
If you have any recommendations about how to accomplish this, I would really appreciate them. Otherwise, I found earlier comments by @joeytwiddle about implementing a "priority kill." So, I forked his code and modified it slightly. I've been testing it for a few days and it seems to do the job quite well. If you'd like---and if it's okay by @joeytwiddle ---I'd be happy to submit a PR.
The new version compiles fine, but when I run it I get:
$ ./earlyoom
earlyoom v0.7-1-g131aa9c
Could not find "MemAvailable:"
Any idea how I can fix this?
(gcc 4.6.3, libc 2.15 on Ubuntu 12.04)
#78
IMO it was unexpected behavior.
What happens if start earlyoom -m 50?
Is it true that the swap is then not counted at all?
2018 will end soon, but the signal is still not supported.
I was hoping someone could fix this line in the make file such that it works if you download a release: VERSION ?= $(shell git describe --tags --dirty)
The following shell code might be a good starting point, but since I don't know make I'm not sure if it is appropriate.
git describe --tags --dirty 2> /dev/null || basename $PWD | sed 's/earlyoom-//'
This assumes you have downloaded a release like so
VERSION=0.11
wget -q https://github.com/rfjakob/earlyoom/archive/v${VERSION}.tar.gz
tar -xzf v${VERSION}.tar.gz
cd earlyoom-${VERSION}
As you can see the version is included in the filename so I propose using the file name as the basis for this. Though, I could see having a VERSION file would be much simpler.
Feel free to ignore me and just delete this issue (serious).
I just noticed earlyoom did this, next time I'll try to provide more debug info:
Out of memory! avail: 138 MiB < min: 786 MiB
Killing process 4084 kworker/u17:3
Out of memory! avail: 155 MiB < min: 786 MiB
Killing process 4084 kworker/u17:3
Out of memory! avail: 157 MiB < min: 786 MiB
Killing process 4084 kworker/u17:3
Could not convert number: Numerical result out of range
But I think earlyoom should never kill kernel threads ;) and not quit on that conversion error.
I ran tail /dev/zero, and tail killed, but earlyoom 1.0 displays wrong name
mem avail: 0 MiB (0 %), swap free: 2425 MiB (41 %)
mem avail: 0 MiB (0 %), swap free: 1631 MiB (27 %)
mem avail: 0 MiB (0 %), swap free: 830 MiB (14 %)
Out of memory! avail: 0 MiB < min: 587 MiB
Killing process 1526 sudo
mem avail: 5057 MiB (86 %), swap free: 5564 MiB (94 %)
mem avail: 5196 MiB (88 %), swap free: 5566 MiB (94 %)
I used -s 100
to practically ignore the free swap memory. Maybe this is worth hinting at when using ./earlyoom -h
or man
.
At various times I have tried 5%, 10% and 15%. What have you guys found works for you?
I found on both my machines, I need to set the OOM killer to engage when I have 300 MB RAM and 300 MB swap remaining. This is the point when the swap starts thrashing.
(Although curiously on my machine with 8 GB RAM and 2 GB swap, if I don't run earlyoom, eventually I will end up with 600 MB RAM and 0 MB swap remaining, and at this point the system becomes unresponsive.)
I wonder if a fixed value of 300 MB RAM and 300 MB swap would satisfy most users. (I don't see why someone with 16 GB RAM would need to engage the OOM killer earlier than someone with 8 GB RAM.)
But obviously on a small machine (e.g. with only 1Gig RAM), the 300MB threshold would be far too high! So perhaps we should be using a percentage, with 300 as the upper limit...?
Interested to hear what values others are using.
earlyoom.conf may be looking like follow:
# This is earlyoom config file
# comment
mem_min_sigkill = 6 %
# comment
swap_min_sigkill = 200 M
# for chromium, comment how it works
decrease_oom_score_adj = False
# comment, how it works
desktop_notifications = False
# comment, examples
prefer =
# comment
avoid = ssh|Xorg
# comment
self_nice = 0
# comment
self_oom_score_adj = 0
# comment
debug = False
Advantages is it more convenient for users, IMHO.
Users may easy find all defaults and what means each parameter.
What is disadvantages?
subj
How about a feature that would warn me if the free memory gets close to the minimum? This way I could first try stopping the process myself.
Basically I always run out of memory in the same scenario: I open Dota, I open a few Chromium tabs, and eventually when Dota starts the game I run out of memory. I want Chromium to be killed, not Dota.
So for me either banning a process from being killed, or prioritizing a process to get killed would do the trick. It would be nice if there was a configuration for that.
Closed too early. The discussion should be continued.
Firstly, sleep(10) not declared and users are not warned.
Secondly, sleep(10) is too long, sleep(3) is enough.
I installed the package from AUR, enabled (and started) the service and manually started it with
sudo earlyoom -m 50
, but it does not work.
I started a jupyter notebook and got the memory down to 2% when it stopped (because it finished what it was loading), but earlyoom never even touched it.
What am I doing wrong?
In 0.10, get_entry_fatal does not return val in the good case.
This code only works by accident with -O0
.
Hello,
I really like earlyoom
, but sometimes it kills some of my processes without no notification. I have to wonder what happened. e.g. python
program is just silently ended without no information about being killed or whatsoever.
It would be really awesome if we could have some event on which notification would be triggered, e.g. the most simple one like:
notify-send 'EarlyOOM' 'Process {$name}:{$PID} killed' --icon=dialog-information
right now it could be implemented as a workaround, when watching stdout of systemd unit. But some nicer solution would be great.
Out of memory! avail: 0 MiB < min: 587 MiB
One this line is not enaugh. Earlyoom not displays swap status info.
I want to see stdout like this:
mem avail: 0 MiB < min: 587 MiB
swap free: 500 MiB < min: 587 MiB
Preventing OOM: killing process...
What about zram support and mem_used_total as a trigger?
Now earlyoom does not help when writing the data with high entropy in zram swap.
Got some process to eat the whole of RAM. earlyoom appears to have crashed.
If it's relevant, I don't have swap.
$ sudo systemctl status earlyoom
● earlyoom.service
Loaded: loaded (/etc/systemd/system/earlyoom.service; enabled)
Active: failed (Result: exit-code) since Mon 2016-02-08 00:05:04 CET; 3 days ago
Process: 699 ExecStart=/usr/local/bin/earlyoom (code=exited, status=9)
Main PID: 699 (code=exited, status=9)
Feb 08 00:05:04 zoob earlyoom[699]: earlyoom v0.7-10-gb4c6bc0
Feb 08 00:05:04 zoob earlyoom[699]: mem total: 31600 MiB, min: 3160 MiB (10 %)
Feb 08 00:05:04 zoob earlyoom[699]: swap total: 0 MiB, min: 0 MiB (10 %)
Feb 08 00:05:04 zoob earlyoom[699]: Error: Could not find a process to kill
Feb 08 00:05:04 zoob systemd[1]: earlyoom.service: main process exited, code=exited, status=9/n/a
Feb 08 00:05:04 zoob systemd[1]: Unit earlyoom.service entered failed state.
Out of memory (OOM) is an often undesired state of computer operation where no additional memory can be allocated for use by programs or the operating system. Such a system will be unable to load any additional programs, and since many programs may load additional data into memory during execution, these will cease to function correctly. This usually occurs because all available memory, including disk swap space, has been allocated.
In normal operation, earlyoom should not occur at all OOM, since preventing OOM is the main task of earlyoom, not a reaction to OOM that has already occurred.
What about replace "Out of memory!" in earlyoom stdout with "Preventing OOM!"?
The default config installs to /etc/default/earlyoom.default
The service file expects it here:
EnvironmentFile=-/etc/default/earlyoom
This should fix it up:
--- Makefile.orig 2017-11-29 19:01:21.537246323 -0500
+++ Makefile 2017-11-29 19:01:25.327274351 -0500
@@ -36,8 +36,8 @@
sed "s|:TARGET:|$(PREFIX)$(BINDIR)|g;s|:SYSCONFDIR:|$(SYSCONFDIR)|g" $< > $@
install-default: earlyoom.default
- install -d $(DESTDIR)$(SYSCONFDIR)/default/
- install -m 644 $< $(DESTDIR)$(SYSCONFDIR)/default/
+ install -d $(DESTDIR)$(SYSCONFDIR)/default
+ cp $< $(DESTDIR)$(SYSCONFDIR)/default/earlyoom
install-bin: earlyoom
install -d $(DESTDIR)$(PREFIX)$(BINDIR)/
Thanks!
$ git clone https://github.com/rfjakob/earlyoom.git
Клонирование в «earlyoom»…
remote: Counting objects: 568, done.
remote: Compressing objects: 100% (85/85), done.
remote: Total 568 (delta 69), reused 101 (delta 43), pack-reused 439
Получение объектов: 100% (568/568), 136.47 KiB | 0 bytes/s, готово.
Определение изменений: 100% (326/326), готово.
$ cd earlyoom
$ make
cc -Wall -Wextra -DVERSION="v1.1-12-g647107d" -g -fstack-protector-all -std=gnu99 -o earlyoom meminfo.c kill.c main.c sanitize.c
pandoc MANPAGE.md -s -t man > earlyoom.1
/bin/sh: 1: pandoc: not found
Makefile:22: ошибка выполнения рецепта для цели «earlyoom.1»
make: *** [earlyoom.1] Error 127
On Ubuntu 16.04, running as user (not as service) the notify-osd notification reported the correct PID but the wrong process name "gnome-system-monitor" when in fact killing whatsapp. The terminal output shows the correct PID and name.
I ran while true; do tail /dev/zero; done
, and eventually Xorg has been killed because earlyoom slept.
would be nice to be able to do this when running in daemon mode.
I found earlyoom was too often killing the main firefox process, or the main chrome process, when I would really rather it had killed just one or two of the chrome tabs I had open, instead of killing a whole browser session!
My approach to this has been to reduce the badness of processes matching a certain regexp. I compare the regexp against the process cmdline, rather than just the process name, so that I can be kind to the main process, but not to tabs or extensions. (I divide badness by 32.)
My fork based on your older code is here: https://github.com/joeytwiddle/earlyoom/tree/logging_usleep
It has been working for me, and I found it very useful (so that I don't lose my whole browser when I have 100 tabs open). I hope you also like the idea, and will consider merging it in! (I could create a PR.)
Here is a more up-to-date version which is merged with your oom_adj
code (which is kind of overkill!): https://github.com/joeytwiddle/earlyoom
However now I am interested in reducing the badness of recently-created or running processes. (Because I would prefer if earlyoom attacks chrome tabs I haven't used for a long time, rather than tabs I am using right now.) (I suspect recently started processes are much easier to detect in Linux than recently used... ;) ) That's now in #8
Earlyoom killed my app because I don't have enough free RAM:
$ sudo journalctl -b | tail
...
Jul 04 19:52:50 xylix earlyoom[699]: Out of memory! avail: 3157 MiB < min: 3160 MiB
Jul 04 19:52:50 xylix earlyoom[699]: Killing process 20118 (VirtualBox)
...
But I have more than enough (~19 GB) in disposable buffers/cache.
$ free -h
total used free shared buffers cached
Mem: 30G 27G 3.6G 13G 228M 15G
-/+ buffers/cache: 11G 19G
Swap: 0B 0B 0B
There is no sense to check memory 10 times per second if there is more than one gigabyte of available memory. You can optimize the frequency of memory checks to decrease the frequency of memory checks and decrease CPU usage. Now the sleep period is 0.1. You can find the time until the next check as follows:
t1 = MemAvailable / 4000000
t2 = SwapFree / 1000000
sleep_until_next_mem_check = t1 + t2
I implemented a similar algorithm in nohang, which is OOM preventer written in Python. If nohang checked memory 10 times per second, then the CPU usage would be too large. Now when using this algorithm for calculating the period between memory checks, the CPU usage is significant only at a low memory level. If there is a lot of memory available, then nohang can use the processor even less than earlyoom. Demo https://youtu.be/8vjeolxw7Uo
I think it's possible to improve earlyoom in the same way. In most cases it will reduce the CPU usage by an order of magnitude.
earlyoom -N 'sudo -u YOUR_USER DISPLAY=:0 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/YOUR_USER_ID/bus notify-send'
is not needed
earlyoom -N 'sudo -u YOUR_USER DISPLAY=:0 notify-send'
is completely enough
...if &
not in the end of the command.
Replace
"-i dialog-warning 'earlyoom' 'Killing process %d %s'", victim_pid, victim_name);
with
"-i dialog-warning 'earlyoom' 'Killing process %d %s' &", victim_pid, victim_name);
will fix the problem
Hey, just installed OOM, and I started earlyoom by doing sudo systemctl enable earlyoom
and sudo systemctl start earlyoom
. Where do I set the -m
option for earlyoom
?
Inside earlyoom.service
:
ExecStart=/usr/local/bin/earlyoom $EARLYOOM_ARGS
Not sure where $EARLYOOM_ARGS
is
IMO -k option is deprecated and should be deleted at all.
earlyoom starts killing processes when just 10% of memory "available".
$ free -h
total used free shared buff/cache available
Mem: 15G 12G 660M 883M 2,7G 2,0G
Swap: 0B 0B 0B
How to force system to free up more buffer/caches under certain conditions so that it frees more memory to become available?
please, fix earlyoom.1
Hello,
In addition to the absolute amount of memory swapped out, would it make sense to use the quantity of swap-in or the amount of time spent waiting on swap-in (if available) as a trigger for kill?
My use case are servers that have overcommit off, which means they are very conservative on how much memory can be committed, and extent of concurrency of multiple processes winds up penalized. When one considers simple Python, Go, or even C programs can allocate hundreds of megabytes of virtual memory that they never use (cat
does this for me, in fact), this is a meaningful problem.
Thus, the amount of swap I may wish to allocate be somewhat large.
However, these servers have soft-real-time performance requirements. It's not desirable to actually let memory be swapped in. It would be better to start killing things if swap-in pressure gets beyond the residual.
Thoughts?
Hello,
I have some processes which can consume a lot of memory, but I don't want them to be killed by earlyoom. And I would also like to be able to set some processes to be preferably killed. So it would be really great if earlyoom could be told to respect oom_score_adj. I know there is problem for you with Chrome setting oom_score_adj too high, but for me this is desired behavior (rather kill browser tab than some memory hungry - but important - process, computation, compilation, etc.). I know I can compile it with USE_KERNEL_OOM_KILLER, which would respect oom_score_adj, but I want to use userspace killing mechanism (I don't want to enable sysrq "f" trigger and I have also bad experience with kernel oom killer in general).
Ideally there should be some more fine grained setting, like list of processes (by name) for which earlyoom should ignore oom_score_adj (this would solve your problem with Chrome) and setting how much should be oom_score_adj taken into account when deciding what to kill. But for the start, even simple (preferably runtime) option to respect/don't respect oom_score_adj would be enough.
$ make
cc -Wextra -DVERSION=\"v0.10-4-g8ea46ee\" -g -o earlyoom main.c meminfo.c kill.c
main.c: In function ‘main’:
main.c:131:11: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘long int’ [-Wformat=]
printf("mem avail: %lu MiB (%d %%), swap free: %lu MiB (%d %%)\n",
^
kill.c: In function ‘get_process_stats’:
kill.c:86:12: warning: use of assignment suppression and length modifier together in gnu_scanf format [-Wformat=]
fscanf(f, "%*lu %lu", &(p.vm_rss));
^
Ubuntu 16.04
Last suggestion for today:
While testing the script (setting the minimum RAM to ca. 95%) I noticed that it would even kill my desktop environment in order to free up some space. I guess there are more "realistic" cases where an option Do not kill those processes, whatever happens, would be useful, though. So how about adding this? :)
subj
When installing earlyoom, the program's manual page (earlyoom.1) is not installed. The attached, updated Makefile adds targets for "install-man" and "uninstall-man" which can be called directly or by an uninstall process.
Makefile-earlyoom.txt
Hi,
I'm running Debian Stretch and my regex skills are poor.
I put the following in /etc/default/earlyoom
``EARLYOOM_ARGS="-m 15 -s 100 -r 5 --prefer '(firefox-esr|Web\sContent|gvfsd-mtp)'"
If I do ps ax | grep earlyoom, I see:
/usr/bin/earlyoom -m 15 -s 100 -r 5 --prefer (firefox-esr|WebsContent|gvfsd-mtp)
which looks like the \s I did to match a space is being misinterpeted.
I am trying to match any of the three piggies: firefox-esr, Web Content, gvfsd-mtp and to ignore the state of Swap and kill based on RAM usage.
Am I doing it correctly?
Thanks!
Mark
Is there possibility to set custom percentages and run as service?
I am a tab junkie. (130 tabs currently open in Chrome, and sometimes I run Firefox alongside it.)
I have started noticing a recurring issue than when my memory is nearly full and I open a heavy webpage, after a few seconds the tab I am working in gets killed!
The behaviour I would prefer is that the new tab would be left running, even though it is using a lot of memory, and an older tab/process would be killed.
(Example "heavy" pages are Google Maps, Facebook, and Google Drive spreadsheets.)
So I would like a process to have its badness reduced when it first starts, but this would gently slide back to the default over time.
I wonder if the start time of a processes is available from se.exec
in /proc/*/sched_start
... (Edit: No, I found what I needed in .../stat
.)
Admittedly my proposed solution will NOT help if I was to switch to an old tab/app and cause it to start consuming a lot of memory. For that situation, it might be more appropriate track which process the user has been recently interacting with. But I fear that might not be easy to detect. A simpler alternative would be to reduce badness of active processes, hence favouring the destruction of processes that have been idle for some time.
This is a really small one. When you do --help it should show the help page, and when there is an unrecognized option it should suggest -h.
Cheers <3
Parsing /proc/meminfo 10 times a second, could you for an even lighter footprint benefit from using linux/sysinfo.h interface?
http://stackoverflow.com/questions/349889/how-do-you-determine-the-amount-of-linux-system-ram-in-c
1.8 MiB (Debian 9 x86_64)
2.0 MiB (Fedora 28)
2.2 MiB (Debian 8 i386, https://i.imgur.com/MaRPWwE.png)
Not about 0.6 MiB.
Is it not enough to drop out only ', ", backslash and backtick? Why do you drop out all Unicode?
sudo ./earlyoom
[sudo] пароль для user:
earlyoom (unknown version)
mem total: 5875 MiB, min: 587 MiB (10 %)
swap total: 0 MiB, min: 0 MiB (10 %)
mem avail: 4912 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4910 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4910 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4910 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4906 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4898 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4901 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4902 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4902 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4905 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4905 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4905 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4905 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4905 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4878 MiB (83 %), swap free: 5875 MiB (100 %)
mem avail: 4878 MiB (83 %), swap free: 5875 MiB (100 %)
mem avail: 4878 MiB (83 %), swap free: 5875 MiB (100 %)
mem avail: 4879 MiB (83 %), swap free: 5875 MiB (100 %)
mem avail: 4879 MiB (83 %), swap free: 5875 MiB (100 %)
...
mem avail: 0 MiB (0 %), swap free: 1034 MiB (17 %)
mem avail: 0 MiB (0 %), swap free: 927 MiB (15 %)
mem avail: 0 MiB (0 %), swap free: 824 MiB (14 %)
mem avail: 2 MiB (0 %), swap free: 698 MiB (11 %)
mem avail: 0 MiB (0 %), swap free: 604 MiB (10 %)
mem avail: 0 MiB (0 %), swap free: 501 MiB (8 %)
mem avail: 0 MiB (0 %), swap free: 396 MiB (6 %)
mem avail: 0 MiB (0 %), swap free: 273 MiB (4 %)
mem avail: 0 MiB (0 %), swap free: 177 MiB (3 %)
mem avail: 0 MiB (0 %), swap free: 84 MiB (1 %)
Out of memory! avail: 0 MiB < min: 587 MiB
Killing process 2456 python3
mem avail: 1606 MiB (27 %), swap free: 472 MiB (8 %)
mem avail: 5184 MiB (88 %), swap free: 5572 MiB (94 %)
mem avail: 5177 MiB (88 %), swap free: 5573 MiB (94 %)
mem avail: 5173 MiB (88 %), swap free: 5574 MiB (94 %)
mem avail: 5173 MiB (88 %), swap free: 5574 MiB (94 %)
I expect that earlyoom kill victim if avail = 10%, not 0 MiB.
My use case differs slightly from yours. I have a swap partition and I want to use it.
I want earlyoom to trigger when both the swap and the memory are nearly full.
I have been using this adapted version of earlyoom for a couple of weeks now, and it has served me well: https://github.com/joeytwiddle/earlyoom/tree/wait_for_swap_to_fill
However it is not compatible with your use case!
I wonder if we should make the effort to combine both these requirements together, so that one package can serve both users with your requirements, and users with my requirements.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.