Crash and restart when generating thumbnails for images on RPi

[katonagl]

Crash and restart when generating thumbnails for images on RPi

[neo1973]

Which RPi model with how much memory?

[katonagl]

Sorry for missing that.

RPi3b with 1GB of memory, out of which 768MB is available to processes. During the debug session the memory usage is around 400MB when generation thumbnails with topping around 550MB for short periods. Checked it with the top command. There was some memory as cache and always min 50MB of pure free memory.
This is same when staying at the beginning of the folder (no scroll down) and no restart and when after scrolling down it restarts.
I have an RPi4 with 8GB, currently not used for Kodi, but it also had this bug (For some time I gave up using Kodi to view pictures).

[neo1973]

At least on the RPi3b I can almost guarantee it's a resource issue, the same happens on my RPi3b when I do as such:

dmesg

[242213.285695] lowmem_reserve[]: 0 0 180 180
[242213.293518] DMA: 1277*4kB (MEC) 538*8kB (MEC) 117*16kB (UEC) 14*32kB (UC) 3*64kB (C) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 11924kB
[242213.310545] 899 total pagecache pages
[242213.318115] 0 pages in swap cache
[242213.325334] Free swap  = 186804kB
[242213.332511] Total swap = 941052kB
[242213.339611] 242688 pages RAM
[242213.346294] 46080 pages HighMem/MovableOnly
[242213.354267] 7357 pages reserved
[242213.361176] 131072 pages cma reserved
[242213.493305] kworker/0:1 invoked oom-killer: gfp_mask=0xc2cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP|__GFP_NOMEMALLOC), order=0, oom_score_adj=0
[242213.509797] CPU: 0 PID: 789 Comm: kworker/0:1 Tainted: G        WC         6.6.30-4-rpi #1
[242213.521985] Hardware name: BCM2835
[242213.529273] Workqueue: events usbnet_deferred_kevent
[242213.538163]  unwind_backtrace from show_stack+0x18/0x1c
[242213.547317]  show_stack from dump_stack_lvl+0x50/0x68
[242213.556208]  dump_stack_lvl from dump_header+0x54/0x1fc
[242213.565267]  dump_header from oom_kill_process+0x23c/0x248
[242213.574369]  oom_kill_process from out_of_memory+0x100/0x344
[242213.583600]  out_of_memory from __alloc_pages+0xa74/0xf38
[242213.592383]  __alloc_pages from new_slab+0x388/0x440
[242213.600784]  new_slab from ___slab_alloc+0x3ec/0x930
[242213.609084]  ___slab_alloc from __kmem_cache_alloc_node+0x194/0x48c
[242213.618673]  __kmem_cache_alloc_node from __kmalloc_node_track_caller+0x50/0x178
[242213.629435]  __kmalloc_node_track_caller from kmalloc_reserve+0x68/0xfc
[242213.639449]  kmalloc_reserve from __alloc_skb+0xb8/0x164
[242213.648160]  __alloc_skb from __netdev_alloc_skb+0x44/0x190
[242213.657045]  __netdev_alloc_skb from rx_submit+0xec/0x278
[242213.665859]  rx_submit from usbnet_deferred_kevent+0x90/0x334
[242213.674953]  usbnet_deferred_kevent from process_one_work+0x160/0x378
[242213.684879]  process_one_work from worker_thread+0x298/0x4f8
[242213.694015]  worker_thread from kthread+0xe4/0x100
[242213.702225]  kthread from ret_from_fork+0x14/0x38
[242213.710301] Exception stack(0xf0a29fb0 to 0xf0a29ff8)
[242213.718715] 9fa0:                                     00000000 00000000 00000000 00000000
[242213.730288] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[242213.741862] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[242213.751983] Mem-Info:
[242213.757641] active_anon:0 inactive_anon:65 isolated_anon:0
                 active_file:612 inactive_file:284 isolated_file:0
                 unevictable:0 dirty:10 writeback:0
                 slab_reclaimable:2066 slab_unreclaimable:9484
                 mapped:3 shmem:0 pagetables:1048
                 sec_pagetables:0 bounce:0
                 kernel_misc_reclaimable:0
                 free:3064 free_pcp:602 free_cma:2685
[242213.822429] Node 0 active_anon:0kB inactive_anon:168kB active_file:2348kB inactive_file:1236kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:12kB dirty:40kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:1752kB pagetables:4192kB sec_pagetables:0kB all_unreclaimable? no
[242213.855312] DMA free:11924kB boost:0kB min:3456kB low:4320kB high:5184kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:2080kB inactive_file:1504kB unevictable:0kB writepending:40kB present:786432kB managed:757004kB mlocked:0kB bounce:0kB free_pcp:2408kB local_pcp:832kB free_cma:10740kB
[242213.890307] lowmem_reserve[]: 0 0 180 180
[242213.898147] DMA: 1277*4kB (MEC) 538*8kB (MEC) 117*16kB (UEC) 14*32kB (UC) 3*64kB (C) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 11924kB
[242213.915110] 899 total pagecache pages
[242213.922627] 0 pages in swap cache
[242213.929848] Free swap  = 186804kB
[242213.937068] Total swap = 941052kB
[242213.944242] 242688 pages RAM
[242213.951005] 46080 pages HighMem/MovableOnly
[242213.959050] 7357 pages reserved
[242213.966026] 131072 pages cma reserved
[242213.973469] Unreclaimable slab info:
[242213.980893] Name                      Used          Total
[242213.990101] zs_handle               1328KB       1332KB
[242213.999095] RAWv6                     63KB         63KB
[242214.007878] tw_sock_TCPv6              3KB          3KB
[242214.016681] request_sock_TCPv6          3KB          3KB
[242214.025282] TCPv6                    127KB        127KB
[242214.033844] bio-240                    4KB          4KB
[242214.042250] bio-288                    7KB          7KB
[242214.050640] bio-304                    7KB          7KB
[242214.058960] bio-336                    7KB          7KB
[242214.067325] bio-96                    44KB         44KB
[242214.075635] bio-208                    4KB          4KB
[242214.083974] mqueue_inode_cache         31KB         31KB
[242214.092355] nfs_commit_data            8KB          8KB
[242214.100680] nfs_read_data            191KB        265KB
[242214.108975] rpc_buffers              208KB        288KB
[242214.117250] ip4-frags                  3KB          3KB
[242214.125418] UDP                       63KB         63KB
[242214.133678] tw_sock_TCP               15KB         15KB
[242214.141850] request_sock_TCP          15KB         15KB
[242214.150019] TCP                      123KB        123KB
[242214.158081] cachefiles_object_jar         32KB         52KB
[242214.166570] bio-192                    8KB          8KB
[242214.174620] ep_head                   16KB         16KB
[242214.182746] request_queue             31KB         31KB
[242214.190715] bio-144                  144KB        169KB
[242214.198800] biovec-max               732KB        870KB
[242214.206783] biovec-128               252KB        252KB
[242214.214816] biovec-64                126KB        126KB
[242214.222769] audit_buffer              34KB         36KB
[242214.230734] skbuff_small_head        157KB        220KB
[242214.238782] skbuff_head_cache        106KB        106KB
[242214.246762] file_lock_ctx           5289KB       5300KB
[242214.254715] fsnotify_mark_connector         15KB         15KB
[242214.263175] taskstats                 30KB         30KB
[242214.271102] proc_dir_entry           272KB        272KB
[242214.279000] pde_opener                15KB         15KB
[242214.286882] sigqueue                  15KB         15KB
[242214.294771] shmem_inode_cache       1037KB       1086KB
[242214.302643] kernfs_iattrs_cache         39KB         39KB
[242214.310633] kernfs_node_cache       1921KB       1929KB
[242214.318479] names_cache              128KB        128KB
[242214.326312] net_namespace            122KB        122KB
[242214.334118] key_jar                   35KB         35KB
[242214.341875] uts_namespace             30KB         30KB
[242214.349633] nsproxy                   15KB         15KB
[242214.357335] vm_area_struct           407KB        454KB
[242214.365099] signal_cache             324KB        330KB
[242214.372800] sighand_cache            338KB        378KB
[242214.380518] task_struct             1054KB       1110KB
[242214.388255] anon_vma_chain           115KB        136KB
[242214.395977] anon_vma                 135KB        135KB
[242214.403576] trace_event_file          95KB         95KB
[242214.411018] maple_node               167KB        236KB
[242214.418349] task_group               102KB        102KB
[242214.425519] mm_struct                 63KB         63KB
[242214.432692] vmap_area                253KB        382KB
[242214.439671] kmalloc-cg-8k             96KB         96KB
[242214.446665] kmalloc-cg-4k            204KB        224KB
[242214.453644] kmalloc-cg-2k            192KB        192KB
[242214.460633] kmalloc-cg-1k            134KB        176KB
[242214.467616] kmalloc-cg-512            72KB         72KB
[242214.474569] kmalloc-cg-256            32KB         32KB
[242214.481557] kmalloc-cg-192           573KB        913KB
[242214.488523] kmalloc-cg-128           103KB        112KB
[242214.495549] kmalloc-cg-64             84KB        100KB
[242214.502500] kmalloc-8k               416KB        416KB
[242214.509385] kmalloc-4k              1308KB       1344KB
[242214.516283] kmalloc-2k               544KB        544KB
[242214.523119] kmalloc-1k               732KB        768KB
[242214.529988] kmalloc-512              832KB        880KB
[242214.536886] kmalloc-256              148KB        148KB
[242214.543760] kmalloc-192              156KB        157KB
[242214.550604] kmalloc-128              324KB        368KB
[242214.557543] kmalloc-64              2422KB       2560KB
[242214.564415] kmem_cache_node           20KB         20KB
[242214.571286] kmem_cache                28KB         28KB
[242214.578162] Tasks state (memory values in pages):
[242214.584538] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[242214.594930] [    245]     0   245    14024       17    98304      256          -250 systemd-journal
[242214.605761] [    280]     0   280     7410       32    61440      320         -1000 systemd-udevd
[242214.616464] [    291]   980   291     4045        0    61440      224             0 systemd-network
[242214.627371] [    408]     0   408    12776       94    69632      160             0 rngd
[242214.637291] [    410]   977   410     4726       32    69632      416             0 systemd-resolve
[242214.648199] [    411]   976   411     6138       32    61440      160             0 systemd-timesyn
[242214.659142] [    417]    81   417     2151       65    45056      128          -900 dbus-broker-lau
[242214.670064] [    420]    81   420      846       96    32768      128          -900 dbus-broker
[242214.680636] [    423]   102   423    15690      128    81920      192             0 polkitd
[242214.690850] [    425]     0   425     2341       64    45056      160         -1000 sshd
[242214.700811] [    426]     0   426     3794        0    57344      192             0 systemd-logind
[242214.711657] [    428]     0   428    12615       96    77824      160             0 upowerd
[242214.721897] [    429] 63137   429   157237        0   143360     2777             0 snowflake-proxy
[242214.732870] [    433]     0   433     1438       64    36864       32             0 agetty
[242214.743014] [  32184]   973 32184      963       96    28672       32             0 kodi-standalone
[242214.754106] [  32189]   973 32189     4318       96    61440      256           100 systemd
[242214.764442] [  32191]   973 32191     5444       64    69632      288           100 (sd-pam)
[242214.774893] [  32198]   973 32198     5465       91    65536      256             0 (sd-pam)
[242214.785320] [   2185]   973  2185      963       32    32768       32             0 kodi
[242214.795434] [   2190]   973  2190   378011       13  2551808   178914             0 kodi.bin
[242214.805898] [  31256]     0 31256     3356      128    53248      192             0 sshd
[242214.815988] [  31263]  1000 31263     4319      160    57344      224           100 systemd
[242214.826283] [  31265]  1000 31265     5444      115    69632      256           100 (sd-pam)
[242214.836756] [  31276]  1000 31276     3468      133    53248      352             0 sshd
[242214.846893] [  31277]  1000 31277     1120       96    28672      160             0 bash
[242214.857138] [  31288]   972 31288     2855        0    53248      192             0 avahi-daemon
[242214.868005] [  31289]   972 31289     2856      127    45056       64             0 avahi-daemon
[242214.878863] [    730]  1000   730     2587       64    40960       96             0 su
[242214.888858] [    733]     0   733     1106       64    28672      192             0 bash
[242214.899025] [    762]     0   762     2422        0    40960      576             0 htop
[242214.909154] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=kodi.bin,pid=2190,uid=973
[242214.924259] Out of memory: Killed process 2190 (kodi.bin) total-vm:1512044kB, anon-rss:0kB, file-rss:52kB, shmem-rss:0kB, UID:973 pgtables:2492kB oom_score_adj:0

There was some memory as cache and always min 50MB of pure free memory.

That's only half the truth, free memory doesn't necessarily mean usable due to memory fragmentation.

I have an RPi4 with 8GB, currently not used for Kodi, but it also had this bug (For some time I gave up using Kodi to view pictures).

Would need a log for this but could be a different issue.

[katonagl]

Thanks. I have checked the logs and I see the following

[47002.106102] [drm:vc4_bo_create [vc4]] ERROR Failed to allocate from GEM DMA helper:
[47002.106448] vc4-drm soc:gpu: [drm] V3D: 205752kb BOs (1486)
[47002.106473] vc4-drm soc:gpu: [drm] V3D shader: 116kb BOs (29)
[47002.106489] vc4-drm soc:gpu: [drm] dumb: 4052kb BOs (1)
[47002.106504] vc4-drm soc:gpu: [drm] binner: 16384kb BOs (1)
[47002.106522] vc4-drm soc:gpu: [drm] RCL: 16kb BOs (2)
[47002.106537] vc4-drm soc:gpu: [drm] BCL: 68kb BOs (2)
[47002.106552] vc4-drm soc:gpu: [drm] total purged BO: 42568kb BOs (215)

I guess this is indeed lack of memory. Am I right? In this case an option to pre-generate thumbnails when idle would be useful for devices with less resources.

I will look for the RPi4 but it needs some time.

[neo1973]

I guess this is indeed lack of memory. Am I right?

Yes, looks like it.

In this case an option to pre-generate thumbnails when idle would be useful for devices with less resources.

That or what has been discussed here: #22135 (comment)
But so far nobody had followed up on that.

I will look for the RPi4 but it needs some time.

Sure 🙂 I know of at least one crash we fixed for Kodi 21 regarding picture viewing, so there is a realistic chance it will work.