Comments (6)
Can you provide more details? like ... how many RAM you give the container? How many it uses it? How many states and objects you have? How many instanes? Maybe you could turn on history logging for system.host.xyz states, especially memRss and such to maybe see whats happening there
from iobroker.docker.
Here's more log:
Feb 29 18:02:43 io4 kernel: [777999.994702] Exception stack(0xc56d1fa8 to 0xc56d1ff0)
Feb 29 18:02:43 io4 kernel: [777999.994709] 1fa0: 001b237c affcce46 0000001f affcce46 001b237c 00000000
Feb 29 18:02:43 io4 kernel: [777999.994716] 1fc0: 001b237c affcce46 0000001f 00000004 00000000 04295850 04295718 bef17914
Feb 29 18:02:43 io4 kernel: [777999.994722] 1fe0: 00000004 bef17880 b6d1e361 b6c97616
Feb 29 18:02:43 io4 kernel: [777999.994729] Mem-Info:
Feb 29 18:02:43 io4 kernel: [777999.994745] active_anon:49279 inactive_anon:765591 isolated_anon:0
Feb 29 18:02:43 io4 kernel: [777999.994745] active_file:195169 inactive_file:517262 isolated_file:0
Feb 29 18:02:43 io4 kernel: [777999.994745] unevictable:4 dirty:25 writeback:43
Feb 29 18:02:43 io4 kernel: [777999.994745] slab_reclaimable:14094 slab_unreclaimable:48895
Feb 29 18:02:43 io4 kernel: [777999.994745] mapped:114488 shmem:107313 pagetables:8152 bounce:0
Feb 29 18:02:43 io4 kernel: [777999.994745] free:413772 free_pcp:119 free_cma:80193
Feb 29 18:02:43 io4 kernel: [777999.994758] Node 0 active_anon:197116kB inactive_anon:3062364kB active_file:780676kB inactive_file:2069048kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:457952kB dirty:100kB writeback:172kB shmem:429252kB writeback_tmp:0kB kernel_stack:6744kB all_unreclaimable? no
Feb 29 18:02:43 io4 kernel: [777999.994773] DMA free:344968kB min:28672kB low:32768kB high:36864kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:1792kB inactive_file:1856kB unevictable:0kB writepending:272kB present:786432kB managed:629492kB mlocked:0kB pagetables:976kB bounce:0kB free_pcp:44kB local_pcp:0kB free_cma:320772kB
Feb 29 18:02:43 io4 kernel: [777999.994780] lowmem_reserve[]: 0 0 7284 7284
Feb 29 18:02:43 io4 kernel: [777999.994827] HighMem free:1310120kB min:512kB low:49044kB high:97576kB reserved_highatomic:0KB active_anon:197116kB inactive_anon:3062168kB active_file:778940kB inactive_file:2067192kB unevictable:16kB writepending:0kB present:7458816kB managed:7458816kB mlocked:16kB pagetables:31632kB bounce:0kB free_pcp:420kB local_pcp:0kB free_cma:0kB
Feb 29 18:02:43 io4 kernel: [777999.994834] lowmem_reserve[]: 0 0 0 0
Feb 29 18:02:43 io4 kernel: [777999.994875] DMA: 2004*4kB (UEC) 1761*8kB (UEC) 302*16kB (UEC) 91*32kB (EC) 85*64kB (C) 88*128kB (C) 81*256kB (C) 69*512kB (C) 65*1024kB (C) 38*2048kB (C) 24*4096kB (C) = 345304kB
Feb 29 18:02:43 io4 kernel: [777999.995057] HighMem: 230*4kB (UM) 25*8kB (UM) 8*16kB (UM) 4*32kB (UM) 7*64kB (UM) 4*128kB (U) 3*256kB (U) 1*512kB (U) 1*1024kB (U) 92*2048kB (M) 273*4096kB (M) = 1311264kB
Feb 29 18:02:43 io4 kernel: [777999.995222] 820819 total pagecache pages
Feb 29 18:02:43 io4 kernel: [777999.995231] 1075 pages in swap cache
Feb 29 18:02:43 io4 kernel: [777999.995240] Swap cache stats: add 12657, delete 11582, find 9/11
Feb 29 18:02:43 io4 kernel: [777999.995247] Free swap = 86780kB
Feb 29 18:02:43 io4 kernel: [777999.995255] Total swap = 102396kB
Feb 29 18:02:43 io4 kernel: [777999.995263] 2061312 pages RAM
Feb 29 18:02:43 io4 kernel: [777999.995270] 1864704 pages HighMem/MovableOnly
Feb 29 18:02:43 io4 kernel: [777999.995277] 39235 pages reserved
Feb 29 18:02:43 io4 kernel: [777999.995285] 81920 pages cma reserved
...
Feb 29 18:02:43 io4 kernel: [777999.996521] Out of memory: Killed process 6458 (iobroker.js-con) total-vm:608740kB, anon-rss:439232kB, file-rss:40316kB, shmem-rss:0kB, UID:1000 pgtables:2456kB oom_score_adj:0
I'am using docker compose to start Iobroker:
version: '2'
services:
iobroker:
container_name: "iobroker"
image: "buanet/iobroker:latest"
hostname: "iobroker"
restart: always
privileged: true
mem_limit: 4G
mem_reservation: 3G
cpus: 2
#oom_score_adj: -500
oom_kill_disable: true
network_mode: "host"
volumes:
- "./data:/opt/iobroker"
- "./backups:/opt/iobroker/backups/"
- "./etc/resolv.conf:/etc/resolv.conf"
#environment:
# - AVAHI=true
security_opt:
- "seccomp:unconfined"
dns:
- 192.168.22.1
- 8.8.8.8
extra_hosts:
- "influxdb:127.0.0.1"
You can see me trying to avoid the killing by setting oom_kill_disable and such.
I do really wonder, If there was something else going on on the system? Why did the reboot solve the issue, but a restart of docker container did not?
Did it got killed by the host system, so the oom settings obove could not help?
Everything is back to normal for now, after rebooting the Raspi.
This is not the first time this happens. So I expect more trouble.
My current solution is, to disable some intensive Script and also disable the latest adapter I added.
But I invested on a 8GB model for a reason, so why can I only use 2GB of it?
I also suspect It to be related to some DNS failures, that are logged at the same time.
Thanks for your help.
from iobroker.docker.
As said maybe start enabling history adapter for some "memrss" states to see what really happens and where the meomory is growing ... linux is weird because it do not needingly kill the process that caused OOM but maybe others ... sooooo ... more details needed. Same for logs.
from iobroker.docker.
OK, since reporting that. I've enabled history for all memrss data. Thanks for that hint, I did not know this feature!
Today, the killing just happened again (last time was about 2 weeks ago). Here's the history graph:
I did not do any changes on the system today. Suddenly, the curve is flat with no change (I'am using fill(previous) in grafana).
In the docker logs I can only see one line:
2024-03-22T09:05:55.837195629Z /opt/scripts/iobroker_startup.sh: line 593: 324 Killed gosu iobroker node node_modules/iobroker.js-controller/controller.js
From my opinion, it gets killed once we reach something over 2GB for a amount of time.
Afterwards Iam not able to startup iobroker again. I have to reboot the Raspi. :(
(You can see the restart on the spike, the 2nd arrow is pointing to)
from iobroker.docker.
Hm .. so the question is where the 2GB "limit" comes from.
In fact you habve 25 instances listed in your chart ... wjth running normally instances nortmally need 50-80 MB "minimum" because of nodejs ..
The js-controller (so the "system.host.*" values are missing because they also also costiing ram. Can you please also show this?
There is also one adapter that seem to need more RAM over time ... is that switchbot-ble? all other seem kind of stable in their ram usage ...
from iobroker.docker.
Related Issues (20)
- [Feature Request]: Podman support (rootless by default) in installation script. HOT 6
- [Feature Request]: Working together on a helm chart HOT 2
- [Problem]: cannot pull most of the tags for ghcr.io/buanet/iobroker HOT 1
- [Feature Request]: Add set -e to healthcheck.sh HOT 1
- [Feature Request]: migrate to PUID/PGID since it is the defacto standard HOT 2
- [Problem]: userscripts are not owned by the SETGID/SETUID user HOT 2
- [Problem]: iobroker_startup.sh should respect traps HOT 3
- [Feature Request]: Disable keep-alive feature for containers, when not in maintenance mode HOT 6
- [Feature Request]: Make the list of available backups sorted
- [Problem]: arm/v7 support HOT 10
- [Problem]: Small typo in docs HOT 1
- [Problem]: Container on QNAP NAS: Backitup can't access mount point HOT 2
- [Problem]: Running Iobroker as a podman rootless container HOT 1
- Backup of influxdb running in different docker is not working HOT 8
- [Problem]: Update js-controller HOT 2
- Failed to load definitions for node: Error: HOT 1
- The selected advertiser, \"avahi\", isn't available on this platform HOT 2
- ERROR: Process exited with code 25 HOT 4
- [Problem]: Use bundled npm version instead of latest / don't upgrade npm manually HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from iobroker.docker.