AVC denied for crio on access of tmpfs (modules_object_t)

[containers]

AVC denied for crio on access of tmpfs (modules_object_t)

[rhatdan]

Very strange this looks like you have a file system labelled modules_object_t?
The only file systems that are supposed to have this label are:

/lib/modules(/.*)?		gen_context(system_u:object_r:modules_object_t,s0)
/usr/lib/modules(/.*)?		gen_context(system_u:object_r:modules_object_t,s0)

This AVC would indicate you were attempting to put a file system object labeled container_file_t on a file system labeled modules_object_t.

Do you have any idea what container you are executing when this happens?

@stephensmalley Any ideas?
@wrabcak ?

[mpepping]

In this case it is a Fluentd container, with some specific mounts to the host:

Containers:
  fluentd-elasticsearch:
    Container ID:   docker://7f9f77537cb563c60afacbd4616035be211f54cf60bf0bd98bdee389f9d211b7
    Image:          gcr.io/google-containers/fluentd-elasticsearch:v2.3.2
    [..]
    Mounts:
      /etc/fluent/config.d from config-volume (rw)
      /host/lib from libsystemddir (ro)
      /var/lib/docker/containers from varlibdockercontainers (ro)
      /var/log from varlog (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from fluentd-elasticsearch-token-7z7xg (ro)
Volumes:
  varlog:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log
    HostPathType:
  varlibdockercontainers:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/docker/containers
    HostPathType:
  libsystemddir:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/lib64
    HostPathType:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      fluentd-elasticsearch
    Optional:  false
  fluentd-elasticsearch-token-7z7xg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  fluentd-elasticsearch-token-7z7xg
    Optional:    false

Thought it affects all running containers. Will check.

[mpepping]

Since we're not seeing this issue with every cluster deployment (and cannot reproduce this easily) we're a bit dependent on the moments this occurs.

modules_objects_t is only defined for:

# grep modules_object_t /etc/selinux/targeted/contexts/files/*
/etc/selinux/targeted/contexts/files/file_contexts:/lib/modules(/.*)?	system_u:object_r:modules_object_t:s0
/etc/selinux/targeted/contexts/files/file_contexts:/usr/lib/modules(/.*)?	system_u:object_r:modules_object_t:s0

No mounted filesystems with modules_objects_t in it's context mountflag.

To make it even more obscure: The moment we see the avc denied for crio, we also see a simmilar one for systemd-logind:

node=node-3 type=AVC msg=audit(1586179201.387:34773): avc:  denied  { mount } for  pid=2270 comm="systemd-logind" name="/" dev="tmpfs" ino=1541968 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:object_r:modules_object_t:s0 tclass=filesystem permissive=0

	Was caused by:
		Missing type enforcement (TE) allow rule.

		You can use audit2allow to generate a loadable module to allow this access.

and for crio

node=node-3 type=AVC msg=audit(1586179450.059:34989): avc:  denied  { associate } for  pid=12036 comm="crio" name="/" dev="tmpfs" ino=118490 scontext=system_u:object_r:container_file_t:s0:c450,c575 tcontext=system_u:object_r:modules_object_t:s0 tclass=filesystem permissive=0

	Was caused by:
		Missing type enforcement (TE) allow rule.

		You can use audit2allow to generate a loadable module to allow this access.

A set of files and directories in /var/lib/kubelet/pods is labeled with the modules_object_t type:

[root]# find /var/lib/kubelet/pods -context "*modules_object_t*" -ls 
113507    0 drwxrwxrwt   3 root     root          140 Apr  6 02:05 /var/lib/kubelet/pods/8a708930-3909-4c21-b714-7b9110e65df4/volumes/kubernetes.io~secret/nginx-ingress-token-hg67p
114091    0 lrwxrwxrwx   1 root     root           31 Apr  6 02:05 /var/lib/kubelet/pods/8a708930-3909-4c21-b714-7b9110e65df4/volumes/kubernetes.io~secret/nginx-ingress-token-hg67p/..data -> ..2020_04_06_02_05_09.684044987
114090    0 lrwxrwxrwx   1 root     root           13 Apr  6 02:05 /var/lib/kubelet/pods/8a708930-3909-4c21-b714-7b9110e65df4/volumes/kubernetes.io~secret/nginx-ingress-token-hg67p/ca.crt -> ..data/ca.crt
114089    0 lrwxrwxrwx   1 root     root           12 Apr  6 02:05 /var/lib/kubelet/pods/8a708930-3909-4c21-b714-7b9110e65df4/volumes/kubernetes.io~secret/nginx-ingress-token-hg67p/token -> ..data/token
114088    0 lrwxrwxrwx   1 root     root           16 Apr  6 02:05 /var/lib/kubelet/pods/8a708930-3909-4c21-b714-7b9110e65df4/volumes/kubernetes.io~secret/nginx-ingress-token-hg67p/namespace -> ..data/namespace
114084    0 drwxr-xr-x   2 root     root          100 Apr  6 02:05 /var/lib/kubelet/pods/8a708930-3909-4c21-b714-7b9110e65df4/volumes/kubernetes.io~secret/nginx-ingress-token-hg67p/..2020_04_06_02_05_09.684044987
114087    4 -rw-r--r--   1 root     root          882 Apr  6 02:05 /var/lib/kubelet/pods/8a708930-3909-4c21-b714-7b9110e65df4/volumes/kubernetes.io~secret/nginx-ingress-token-hg67p/..2020_04_06_02_05_09.684044987/token
114086    4 -rw-r--r--   1 root     root           13 Apr  6 02:05 /var/lib/kubelet/pods/8a708930-3909-4c21-b714-7b9110e65df4/volumes/kubernetes.io~secret/nginx-ingress-token-hg67p/..2020_04_06_02_05_09.684044987/namespace
114085    4 -rw-r--r--   1 root     root         1025 Apr  6 02:05 /var/lib/kubelet/pods/8a708930-3909-4c21-b714-7b9110e65df4/volumes/kubernetes.io~secret/nginx-ingress-token-hg67p/..2020_04_06_02_05_09.684044987/ca.crt
[..]

Restarting the kubelet systemd unit doesn't resolve this issue, rebooting the node does.

[rhatdan]

Are these file systems being mounted with a context mount?

On the host could you do a
grep modules_object_t /proc/self/mounts /proc/self/mountinfo

[mpepping]

Context modules_object_t isn't active on any mount. The only mounts that have a context property are overlay and shm mounts and these use context="system_u:object_r:container_file_t".

The tmpfs mount belonging to the above snippet is:

tmpfs on /var/lib/kubelet/pods/8a708930-3909-4c21-b714-7b9110e65df4/volumes/kubernetes.io~secret/nginx-ingress-token-hg67p type tmpfs (rw,relatime,seclabel)

[rhatdan]

@stephensmalley Any ideas?

[stephensmalley]

If you don't have any context mounts with modules_object_t and you don't have any fs_use or genfscon rules in policy that specify modules_object_t then this seems like a kernel bug.

[rhatdan]

@wrabcak FYI?

[wrabcak]

Very strange.
Maybe @WOnder93 can check kernel bits.

[mpepping]

No genfscon or fs_use statements are used.

[mpepping]

With the risk of jinxing the happy streak: I haven't seen the issue occur in over +50 deployments after upgrading cri-o/k8s to 1.17.4.

Thanks for the input!