[Problem]: iobroker gets killed by oom-kill constantly

[robig]

[Problem]: iobroker gets killed by oom-kill constantly

[Apollon77]

Can you provide more details? like ... how many RAM you give the container? How many it uses it? How many states and objects you have? How many instanes? Maybe you could turn on history logging for system.host.xyz states, especially memRss and such to maybe see whats happening there

[robig]

Here's more log:

Feb 29 18:02:43 io4 kernel: [777999.994702] Exception stack(0xc56d1fa8 to 0xc56d1ff0)
Feb 29 18:02:43 io4 kernel: [777999.994709] 1fa0:                   001b237c affcce46 0000001f affcce46 001b237c 00000000
Feb 29 18:02:43 io4 kernel: [777999.994716] 1fc0: 001b237c affcce46 0000001f 00000004 00000000 04295850 04295718 bef17914
Feb 29 18:02:43 io4 kernel: [777999.994722] 1fe0: 00000004 bef17880 b6d1e361 b6c97616
Feb 29 18:02:43 io4 kernel: [777999.994729] Mem-Info:
Feb 29 18:02:43 io4 kernel: [777999.994745] active_anon:49279 inactive_anon:765591 isolated_anon:0
Feb 29 18:02:43 io4 kernel: [777999.994745]  active_file:195169 inactive_file:517262 isolated_file:0
Feb 29 18:02:43 io4 kernel: [777999.994745]  unevictable:4 dirty:25 writeback:43
Feb 29 18:02:43 io4 kernel: [777999.994745]  slab_reclaimable:14094 slab_unreclaimable:48895
Feb 29 18:02:43 io4 kernel: [777999.994745]  mapped:114488 shmem:107313 pagetables:8152 bounce:0
Feb 29 18:02:43 io4 kernel: [777999.994745]  free:413772 free_pcp:119 free_cma:80193
Feb 29 18:02:43 io4 kernel: [777999.994758] Node 0 active_anon:197116kB inactive_anon:3062364kB active_file:780676kB inactive_file:2069048kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:457952kB dirty:100kB writeback:172kB shmem:429252kB writeback_tmp:0kB kernel_stack:6744kB all_unreclaimable? no
Feb 29 18:02:43 io4 kernel: [777999.994773] DMA free:344968kB min:28672kB low:32768kB high:36864kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:1792kB inactive_file:1856kB unevictable:0kB writepending:272kB present:786432kB managed:629492kB mlocked:0kB pagetables:976kB bounce:0kB free_pcp:44kB local_pcp:0kB free_cma:320772kB
Feb 29 18:02:43 io4 kernel: [777999.994780] lowmem_reserve[]: 0 0 7284 7284
Feb 29 18:02:43 io4 kernel: [777999.994827] HighMem free:1310120kB min:512kB low:49044kB high:97576kB reserved_highatomic:0KB active_anon:197116kB inactive_anon:3062168kB active_file:778940kB inactive_file:2067192kB unevictable:16kB writepending:0kB present:7458816kB managed:7458816kB mlocked:16kB pagetables:31632kB bounce:0kB free_pcp:420kB local_pcp:0kB free_cma:0kB
Feb 29 18:02:43 io4 kernel: [777999.994834] lowmem_reserve[]: 0 0 0 0
Feb 29 18:02:43 io4 kernel: [777999.994875] DMA: 2004*4kB (UEC) 1761*8kB (UEC) 302*16kB (UEC) 91*32kB (EC) 85*64kB (C) 88*128kB (C) 81*256kB (C) 69*512kB (C) 65*1024kB (C) 38*2048kB (C) 24*4096kB (C) = 345304kB
Feb 29 18:02:43 io4 kernel: [777999.995057] HighMem: 230*4kB (UM) 25*8kB (UM) 8*16kB (UM) 4*32kB (UM) 7*64kB (UM) 4*128kB (U) 3*256kB (U) 1*512kB (U) 1*1024kB (U) 92*2048kB (M) 273*4096kB (M) = 1311264kB
Feb 29 18:02:43 io4 kernel: [777999.995222] 820819 total pagecache pages
Feb 29 18:02:43 io4 kernel: [777999.995231] 1075 pages in swap cache
Feb 29 18:02:43 io4 kernel: [777999.995240] Swap cache stats: add 12657, delete 11582, find 9/11
Feb 29 18:02:43 io4 kernel: [777999.995247] Free swap  = 86780kB
Feb 29 18:02:43 io4 kernel: [777999.995255] Total swap = 102396kB
Feb 29 18:02:43 io4 kernel: [777999.995263] 2061312 pages RAM
Feb 29 18:02:43 io4 kernel: [777999.995270] 1864704 pages HighMem/MovableOnly
Feb 29 18:02:43 io4 kernel: [777999.995277] 39235 pages reserved
Feb 29 18:02:43 io4 kernel: [777999.995285] 81920 pages cma reserved
...
Feb 29 18:02:43 io4 kernel: [777999.996521] Out of memory: Killed process 6458 (iobroker.js-con) total-vm:608740kB, anon-rss:439232kB, file-rss:40316kB, shmem-rss:0kB, UID:1000 pgtables:2456kB oom_score_adj:0

I'am using docker compose to start Iobroker:

version: '2'

services:
  iobroker:
    container_name: "iobroker"
    image: "buanet/iobroker:latest"
    hostname: "iobroker"
    restart: always
    privileged: true
    mem_limit: 4G
    mem_reservation: 3G
    cpus: 2
    #oom_score_adj: -500
    oom_kill_disable: true
    network_mode: "host"
    volumes:
      - "./data:/opt/iobroker"
      - "./backups:/opt/iobroker/backups/"
      - "./etc/resolv.conf:/etc/resolv.conf"
    #environment:
    #  - AVAHI=true
    security_opt:
      - "seccomp:unconfined"
    dns:
      - 192.168.22.1
      - 8.8.8.8
    extra_hosts:
      - "influxdb:127.0.0.1"

You can see me trying to avoid the killing by setting oom_kill_disable and such.
I do really wonder, If there was something else going on on the system? Why did the reboot solve the issue, but a restart of docker container did not?
Did it got killed by the host system, so the oom settings obove could not help?

Everything is back to normal for now, after rebooting the Raspi.

This is not the first time this happens. So I expect more trouble.
My current solution is, to disable some intensive Script and also disable the latest adapter I added.
But I invested on a 8GB model for a reason, so why can I only use 2GB of it?

I also suspect It to be related to some DNS failures, that are logged at the same time.

Thanks for your help.

[Apollon77]

As said maybe start enabling history adapter for some "memrss" states to see what really happens and where the meomory is growing ... linux is weird because it do not needingly kill the process that caused OOM but maybe others ... sooooo ... more details needed. Same for logs.

[robig]

OK, since reporting that. I've enabled history for all memrss data. Thanks for that hint, I did not know this feature!

Today, the killing just happened again (last time was about 2 weeks ago). Here's the history graph:

I did not do any changes on the system today. Suddenly, the curve is flat with no change (I'am using fill(previous) in grafana).

In the docker logs I can only see one line:

2024-03-22T09:05:55.837195629Z /opt/scripts/iobroker_startup.sh: line 593:   324 Killed                  gosu iobroker node node_modules/iobroker.js-controller/controller.js

From my opinion, it gets killed once we reach something over 2GB for a amount of time.
Afterwards Iam not able to startup iobroker again. I have to reboot the Raspi. :(
(You can see the restart on the spike, the 2nd arrow is pointing to)

[Apollon77]

Hm .. so the question is where the 2GB "limit" comes from.

In fact you habve 25 instances listed in your chart ... wjth running normally instances nortmally need 50-80 MB "minimum" because of nodejs ..

The js-controller (so the "system.host.*" values are missing because they also also costiing ram. Can you please also show this?

There is also one adapter that seem to need more RAM over time ... is that switchbot-ble? all other seem kind of stable in their ram usage ...