[error] [input:kubernetes_events:kubernetes_events.0] upstream connection initialization error

[ashishkumar256]

[error] [input:kubernetes_events:kubernetes_events.0] upstream connection initialization error

[air3ijai]

Works fine on Kubernetes 1.29.1

Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.1

[2024/03/25 12:01:03] [ info] [input:kubernetes_events:kubernetes_events.0] API server: https://kubernetes.default.svc:443
[2024/03/25 12:01:03] [ info] [input:kubernetes_events:kubernetes_events.0] thread instance initialized
[2024/03/25 12:01:03] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2024/03/25 12:01:03] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2024/03/25 12:01:03] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2024/03/25 12:01:03] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2024/03/25 12:01:03] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2024/03/25 12:01:03] [ info] [output:es:es.0] worker #0 started
[2024/03/25 12:01:03] [ info] [output:es:es.0] worker #1 started
[2024/03/25 12:01:03] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/03/25 12:01:03] [ info] [sp] stream processor started

details

fluent-bit.tf

# Fluent Bit
resource "helm_release" "fluent-bit" {
  name       = "fluent-bit"
  repository = "https://fluent.github.io/helm-charts"
  chart      = "fluent-bit"
  version    = "0.44.0"
  namespace  = logs
  timeout    = 300
  values     = [file("${path.module}/values/fluent-bit.yaml")]
}

fluent-bit.yaml

# Chart - https://github.com/fluent/helm-charts/tree/main/charts/fluent-bit

# Main
kind: Deployment
replicaCount: 1

image:
  repository: cr.fluentbit.io/fluent/fluent-bit
  tag: 3.0.0-debug

testFramework:
  enabled: false

# Permissions
rbac:
  create: true
  nodeAccess: true
  eventsAccess: true

# Configuration
config:
  service: |
    [SERVICE]
        flush         1
        daemon        Off
        LogLevel      info
        Config_Watch  On
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

  inputs: |
    [INPUT]
        Name      kubernetes_events
        Tag       kube_events
        Kube_url  https://kubernetes.default.svc

  outputs: |
    [OUTPUT]
        Name                es
        Match               kube_events
        Host                elastic
        Port                9200
        Compress            gzip
        Type                _doc
        Suppress_Type_Name  On
        Logstash_Format     On
        Logstash_Prefix     kube-events
        tls                 On
        tls.verify          Off

    [OUTPUT]
        name    stdout
        match   kube_events
        Format  json

Can it be related to your setup?

• #7920

[patrick-stephens]

I do not see any debug level logs in your original output either @ashishkumar256 which I'd expect with the log level set that way.

How are you deploying all this? The OSS helm chart?
What's the full pod spec and make sure the cluster role is being properly configured for the SA?
Are you sure you have the right TLS and K8S API server address? Can you confirm it works with the -debug image?

[ashishkumar256]

Thanks @patrick-stephens, above shared configurations seems to be working. I was using my own written helm charts that were having all required configurations & permissions. I'll compare it, get it fixed.