FluentBit restarts sporadically with "[engine] caught signal (SIGSEGV)" on ARM64 Nodes in Kubernetes Cluster (AWS EKS)

[cloud-thought]

FluentBit restarts sporadically with "[engine] caught signal (SIGSEGV)" on ARM64 Nodes in Kubernetes Cluster (AWS EKS)

[patrick-stephens]

Does it happen with 3.0?

[cloud-thought]

Hey @patrick-stephens, have not tried version 3.0. We are using this Helm Chart https://artifacthub.io/packages/helm/fluent/fluent-bit

[patrick-stephens]

I would try in case it's already resolved, in which case it's a case of figuring out what to backport or if it still affects it. I can't easily test ARM clusters I'm afraid so relies on user info.

[cloud-thought]

Alright will give it a try. Nonetheless, maybe other users have similar issues and would like to report. I will post an update as soon as I have enough data.

[cloud-thought]

Ok, didn't have to wait for that long to encounter the same issue:

Defaulted container "fluent-bit" out of: fluent-bit, reloader
Fluent Bit v3.0.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________  
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/ 

[2024/03/25 15:06:58] [engine] caught signal (SIGSEGV)
#0  0xaaaae6aec54b      in  cb_engine_sched_timer() at src/flb_engine.c:175
#1  0xaaaae6af09a7      in  flb_sched_event_handler() at src/flb_scheduler.c:428
#2  0xaaaae6aed42f      in  flb_engine_start() at src/flb_engine.c:1025
#3  0xaaaae6acd9ff      in  flb_lib_worker() at src/flb_lib.c:674
#4  0xffffaf669647      in  ???() at ???:0
#5  0xffffaefce01b      in  ???() at ???:0
#6  0xffffffffffffffff  in  ???() at ???:0

This time it happened on nearly all ARM64 nodes we have running in our cluster.