[nri-kube-events] attempting to write warnings to RO filesystem

[paul91]

[nri-kube-events] attempting to write warnings to RO filesystem

[rbabyuk]

I hit the same issue, it's weird as /tmp is mounted and 'emptydir' and 'readonly' flag is not set, meaning 'readonly' is false by default.

kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/app $ ls -la
total 38988
drwxr-xr-x    1 root     root          4096 Dec  4 09:51 .
drwxr-xr-x    1 root     root          4096 Dec  4 09:51 ..
drwxrwxrwx    3 root     root          4096 Dec  4 08:01 config
-rwxr-xr-x    1 root     root      39909837 Jul 20 14:32 nri-kube-events
/app $ ls -la  /
total 68
drwxr-xr-x    1 root     root          4096 Dec  4 09:51 .
drwxr-xr-x    1 root     root          4096 Dec  4 09:51 ..
-rwxr-xr-x    1 root     root             0 Dec  4 09:51 .dockerenv
drwxr-xr-x    1 root     root          4096 Dec  4 09:51 app
drwxr-xr-x    2 root     root          4096 Apr 23  2020 bin
drwxr-xr-x    5 root     root           360 Dec  4 09:51 dev
drwxr-xr-x    1 root     root          4096 Dec  4 09:51 etc
drwxr-xr-x    2 root     root          4096 Apr 23  2020 home
drwxr-xr-x    1 root     root          4096 Apr 23  2020 lib
drwxr-xr-x    5 root     root          4096 Apr 23  2020 media
drwxr-xr-x    2 root     root          4096 Apr 23  2020 mnt
drwxr-xr-x    2 root     root          4096 Apr 23  2020 opt
dr-xr-xr-x  157 root     root             0 Dec  4 09:51 proc
drwx------    2 root     root          4096 Apr 23  2020 root
drwxr-xr-x    1 root     root          4096 Dec  4 09:51 run
drwxr-xr-x    2 root     root          4096 Apr 23  2020 sbin
drwxr-xr-x    2 root     root          4096 Apr 23  2020 srv
dr-xr-xr-x   12 root     root             0 Dec  4 08:01 sys
drwxrwxrwt    2 root     root          4096 Apr 23  2020 tmp
drwxr-xr-x    1 root     root          4096 Apr 23  2020 usr
drwxr-xr-x    1 root     root          4096 Apr 23  2020 var
/app $ ls -la  /tmp/
total 8
drwxrwxrwt    2 root     root          4096 Apr 23  2020 .
drwxr-xr-x    1 root     root          4096 Dec  4 09:51 ..
/app $ echo 'test' >  /tmp/test.txt
/bin/sh: can't create /tmp/test.txt: Read-only file system
/app $

and description of pods

Name:         newrelic-bundle-nri-kube-events-b8fd6d44f-ct79c
Namespace:    new-relic
Priority:     0
Node:         gke-ado-dev21753c63--ado-dev21753c63--93f4b0df-psjl/10.20.0.6
Start Time:   Fri, 04 Dec 2020 10:01:33 +0200
Labels:       app=nri-kube-events
              app.kubernetes.io/instance=newrelic-bundle
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=nri-kube-events
              app.kubernetes.io/version=1.3.0
              helm.sh/chart=nri-kube-events-1.3.4
              pod-template-hash=b8fd6d44f
Annotations:  <none>
Status:       Running
IP:           10.56.6.19
IPs:
  IP:           10.56.6.19
Controlled By:  ReplicaSet/newrelic-bundle-nri-kube-events-b8fd6d44f
Containers:
  kube-events:
    Container ID:  docker://f1bb6fc7852e00867767ca0ae4edef60c5e68d7e48cf31fc72f0c719b9f3e6c4
    Image:         newrelic/nri-kube-events:1.3.0
    Image ID:      docker-pullable://newrelic/nri-kube-events@sha256:7fae6822b53f1f2a3857707bb033f9bcf7453cc9a1b6f161a7d9567545b787bc
    Port:          <none>
    Host Port:     <none>
    Args:
      -config
      /app/config/config.yaml
      -loglevel
      debug
    State:          Running
      Started:      Fri, 04 Dec 2020 11:59:37 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Fri, 04 Dec 2020 11:51:23 +0200
      Finished:     Fri, 04 Dec 2020 11:59:24 +0200
    Ready:          True
    Restart Count:  11
    Environment:    <none>
    Mounts:
      /app/config from config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from newrelic-bundle-nri-kube-events-token-6w25m (ro)
  infra-agent:
    Container ID:   docker://79515ecb8db1ec5f524563b4b18dfc0b09b91c451f670a7e4357af21ed7e10c2
    Image:          newrelic/k8s-events-forwarder:1.12.0
    Image ID:       docker-pullable://newrelic/k8s-events-forwarder@sha256:e1258b23b8e350d8e7f159c69b4a24e2178edd57afa9240dacdb549cf84e6938
    Port:           8001/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 04 Dec 2020 10:01:36 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      NRIA_LICENSE_KEY:  <set to the key 'licenseKey' in secret 'newrelic-bundle-nri-kube-events-config'>  Optional: false
    Mounts:
      /tmp from tmpfs-tmp (rw)
      /var/db/newrelic-infra/data from tmpfs-data (rw)
      /var/db/newrelic-infra/user_data from tmpfs-user-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from newrelic-bundle-nri-kube-events-token-6w25m (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      newrelic-bundle-nri-kube-events-config
    Optional:  false
  tmpfs-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  tmpfs-user-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  tmpfs-tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  newrelic-bundle-nri-kube-events-token-6w25m:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  newrelic-bundle-nri-kube-events-token-6w25m
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                    From     Message
  ----     ------   ----                   ----     -------
  Normal   Created  8m52s (x11 over 118m)  kubelet  Created container kube-events
  Normal   Started  8m52s (x11 over 118m)  kubelet  Started container kube-events
  Warning  BackOff  50s (x20 over 102m)    kubelet  Back-off restarting failed container
  Normal   Pulled   38s (x12 over 118m)    kubelet  Container image "newrelic/nri-kube-events:1.3.0" already present on machine

notice /tmp from tmpfs-tmp (rw) is defined for 'infra-agent' abut not for 'kube-events'.
trying to add this mount to 'kube-events'