lxc/lxcri

CRI-O support for lxc

About

lxcri is a wrapper around LXC which can be used as a drop-in container runtime replacement for use by CRI-O.

OCI compliance

With liblxc starting from lxc-4.0.0-927-gb5daeddc5 it passes all sonobuoy conformance tests.

Build

You can use the provided Dockerfile to build an

runtime only image (lxcri + lxc)

docker build --build-arg installcmd=install_runtime

or with everything required for a kubernetes node (kubelet, kubeadm, cri-o, lxcri, lxc ...)

docker build

Note: The images are not pre-configured and you must follow the steps in setup for now.

Setup

To use lxcri as OCI runtime in cri-o see setup.md

API Usage

Please have a look at the runtime tests for now.

Notes

• It's currently only tested with cgroups v2.

lxcri's Issues

Unable to use lxcri

I've managed to build lxcri and lxcri-* binaries from the sources (by fixing some errors, I'll another issue for this), but either from github.com/lxc/lxcri:main or from @r10r fork drachenfels-de/lxcri:fixes, I get the same error

In Kubernetes events, I'm getting:

Error: container create failed: [lxcri-start] monitor process pid=1357587 failed (container error_num:1)
[lxcri:create.go:64] failed to run container process: monitor already died
Warning  Failed  4m49s (x3 over 5m15s)   kubelet  (combined from similar events): Error: container create failed: lxcri://57ab0cec5b0aa8d439c83402d04f584512f9ea9c5dc91e55a85ddd34ceae1363 [lxcri:create.go:77] failed to run container process: monitor already died

Digging a bit, lxcri gives me this:

{"l":"info","cmd":"create","cid":"116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d","t":"16:00:56.768","c":"runtime.go:196","m":"using cgroup root /sys/fs/cgroup"}
{"l":"info","cmd":"create","cid":"116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d","t":"16:00:56.781","c":"create.go:146","m":"device files are bind mounted"}
{"l":"warn","cmd":"create","cid":"116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d","t":"16:00:56.784","c":"create.go:480","m":"duplicate environment variable PATH (overwrite=true)"}
{"l":"info","cmd":"create","cid":"116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d","pid":1778291,"t":"16:00:56.788","c":"runtime.go:418","m":"monitor process started"}
lxc 116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d 20240219150056.830 WARN     start - ../src/lxc/start.c:lxc_spawn:1832 - File exists - Failed to allocate new network namespace id
lxc 116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d 20240219150057.954 ERROR    conf - ../src/lxc/conf.c:run_buffer:295 - Resource temporarily unavailable - Failed to popen() exec /usr/local/libexec/lxcri/lxcri-hook
lxc 116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d 20240219150057.955 ERROR    conf - ../src/lxc/conf.c:lxc_setup:4437 - Failed to run mount hooks
lxc 116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d 20240219150057.955 ERROR    start - ../src/lxc/start.c:do_start:1272 - Failed to setup container "116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d"
lxc 116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d 20240219150057.956 ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
lxc 116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d 20240219150057.965 ERROR    start - ../src/lxc/start.c:__lxc_start:2107 - Failed to spawn container "116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d"
lxc 116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d 20240219150057.966 WARN     start - ../src/lxc/start.c:lxc_abort:1036 - No such process - Failed to send SIGKILL via pidfd 17 for process 1778293
lxc 116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d 20240219150057.115 ERROR    af_unix - ../src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d 20240219150057.115 ERROR    commands - ../src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_state"
{"l":"info","cmd":"create","cid":"116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d","t":"16:00:57.216","c":"container.go:160","m":"monitor 1778291 died: exited:false exit_status:-1 signaled:true signal:hangup"}
{"l":"error","cmd":"create","cid":"116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d","t":"16:00:57.216","c":"cli.go:354","m":"failed to create container: [lxcri:create.go:77] failed to run container process: monitor already died"}
{"l":"info","cmd":"create","cid":"116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d","force":true,"cid":"116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d","t":"16:00:57.216","c":"runtime.go:497","m":"delete container"}
{"l":"error","cmd":"create","cid":"116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d","error":"[lxcri:create.go:77] failed to run container process: monitor already died","duration":452.438103,"t":"16:00:57.219","c":"cli.go:274","m":"command failed"}
{"l":"info","cmd":"delete","cid":"116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d","t":"16:00:57.236","c":"runtime.go:196","m":"using cgroup root /sys/fs/cgroup"}
{"l":"info","cmd":"delete","cid":"116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d","force":true,"cid":"116434c7b440e4d58018046134937624b8a588a676e54d9758ef1f59ec39be5d","t":"16:00:57.236","c":"runtime.go:497","m":"delete container"}

The real (first) error seem to be ERROR conf - ../src/lxc/conf.c:run_buffer:295 - Resource temporarily unavailable - Failed to popen() exec /usr/local/libexec/lxcri/lxcri-hook

I've try to take a look at the code (from lxc) but unfortunately I've not been able to understand the issue.

Container startup failed: ended on error (126)

Hi @r10r ,

Following the bringup of the k8s cluster, I am attempting to move our existing containers to use crio-lxc.

I am building my OCI image w/ buildah, using

• buildah from scratch
• buildah copy to copy in the existing LXC container rootfs
• buildah config --cmd "bash -x /home/rp/workspace/start-rp-webserver" to set the startup command of the container
• buildah commit CONTAINER_ID rp-runtime-lxc:latest

This however fails when deployed over the cluster :

$ k get pods
NAME                                  READY   STATUS             RESTARTS   AGE
[...]
rp-runtime                            0/1     Error              0          52m

There is not much error reported other than the 126 exit code:

$ k describe pods rp-runtime 
Name:         rp-runtime
Namespace:    kube-system
Priority:     0
Node:         vagrant-k8s/10.0.2.15
Start Time:   Tue, 09 Feb 2021 08:26:31 +0000
Labels:       run=rp-runtime
Annotations:  <none>
Status:       Failed
IP:           10.0.0.203
IPs:
  IP:  10.0.0.203
Containers:
  rp-runtime:
    Container ID:   cri-o://dd08b37614838e8c21eae450b83a5be3f62b51236bacad50b83e4a63a513c678
    Image:          localhost/rp-runtime-lxc:latest
    Image ID:       localhost/rp-runtime-lxc@sha256:6f1850cd085bfd565f57eb488f852cbed949d7bcde949d59c5a169a8b9440c32
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Error
      Exit Code:    126
      Started:      Tue, 09 Feb 2021 08:26:33 +0000
      Finished:     Tue, 09 Feb 2021 08:26:33 +0000
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-4g57m (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-4g57m:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-4g57m
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  53m   default-scheduler  Successfully assigned kube-system/rp-runtime to vagrant-k8s
  Normal  Pulling    53m   kubelet            Pulling image "localhost/rp-runtime-lxc:latest"
  Normal  Pulled     53m   kubelet            Successfully pulled image "localhost/rp-runtime-lxc:latest" in 76.43171ms
  Normal  Created    53m   kubelet            Created container rp-runtime
  Normal  Started    53m   kubelet            Started container rp-runtime

The crio-lxc log is also not very verbose about what happened:

lxc dd08b37614838e8c21eae450b83a5be3f62b51236bacad50b83e4a63a513c678 20210209082633.321 DEBUG    commands - commands.c:lxc_cmd_get_state:821 - Container "dd08b37614838e8c21eae450b83a5be3f62b51236bacad50b83e4a63a513c678" is in "RUNNING" st
ate
lxc dd08b37614838e8c21eae450b83a5be3f62b51236bacad50b83e4a63a513c678 20210209082633.321 DEBUG    commands - commands.c:lxc_cmd_rsp_recv:172 - Response data length for command "get_init_pid" is 0
{"l":"info","cmd":"state","cid":"dd08b37614838e8c21eae450b83a5be3f62b51236bacad50b83e4a63a513c678","pid":162725,"status":"running","t":"20210209082633.321","c":"runtime.go:550","m":"container state"}
{"l":"debug","cmd":"state","cid":"dd08b37614838e8c21eae450b83a5be3f62b51236bacad50b83e4a63a513c678","duration":1.309935,"t":"20210209082633.321","c":"cli.go:225","m":"cmd completed"}
lxc dd08b37614838e8c21eae450b83a5be3f62b51236bacad50b83e4a63a513c678 20210209082633.324 DEBUG    start - start.c:signal_handler:422 - Container init process 162727 exited
lxc dd08b37614838e8c21eae450b83a5be3f62b51236bacad50b83e4a63a513c678 20210209082633.324 INFO     error - error.c:lxc_error_set_and_log:28 - Child <162727> ended on error (126)
lxc dd08b37614838e8c21eae450b83a5be3f62b51236bacad50b83e4a63a513c678 20210209082633.324 DEBUG    network - network.c:lxc_delete_network:3988 - Deleted network devices
{"l":"info","cmd":"state","cid":"dd08b37614838e8c21eae450b83a5be3f62b51236bacad50b83e4a63a513c678","pid":162725,"status":"stopped","t":"20210209082633.360","c":"runtime.go:550","m":"container state"}
{"l":"debug","cmd":"state","cid":"dd08b37614838e8c21eae450b83a5be3f62b51236bacad50b83e4a63a513c678","duration":0.840545,"t":"20210209082633.360","c":"cli.go:225","m":"cmd completed"}
{"l":"info","cmd":"state","cid":"dd08b37614838e8c21eae450b83a5be3f62b51236bacad50b83e4a63a513c678","pid":162725,"status":"stopped","t":"20210209082633.364","c":"runtime.go:550","m":"container state"}
{"l":"debug","cmd":"state","cid":"dd08b37614838e8c21eae450b83a5be3f62b51236bacad50b83e4a63a513c678","duration":0.729601,"t":"20210209082633.364","c":"cli.go:225","m":"cmd completed"}

I suspect this is due to how I am setting the startup entry point for the container (we have a shell script). crictl inspect on the container shows what I have set (cf args):

  "info": {
    "sandboxID": "b845e01a217771503c3b555495b87c351656e28c5d82d42ca4b2ec41ed46bd90",
    "pid": 162725,
    "runtimeSpec": {
      "ociVersion": "1.0.2-dev",
      "process": {
        "user": {
          "uid": 1000,
          "gid": 1001,
          "additionalGids": [
            190,
            1000
          ]
        },
        "args": [
          "/bin/bash",
          "-x",
          "/home/rp/workspace/start-rp-webserver"

Until I can build a testcase that I can share the logs of, would you have any strategy to find where the problem is?

Can you also confirm having a shell-based entry point is supported? I don't see any restrictions in the OCI runtime spec but crio-lxc might have some.

Thanks much!

wrong help info

@brauner reports that the help info for start shows the help for state.

handle linux resources stanza

https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#example-3

this includes device access and cgroup settings such as cpu.shares

handle LISTEN_FDS

crio will optionally send nsfds for some of the namespaces that are shared with the pod. Use those if sent.

cri-o OCI runtime interface needs clarification

Clarify what runtime version / cmdline interface is actually used by cri-o.
There are two OCI runtime specifications.

Runtime and Lifecycle

• https://github.com/opencontainers/runtime-spec/blob/master/runtime.md

This is the simple version, which actually defines the 'kill' command interface used by cri-o.

OCI Runtime Command Line Interface

https://github.com/opencontainers/runtime-tools/blob/master/docs/command-line-interface.md

This is a the more complex interface, and it has not been changed since April/2008.
Some flags are defined in the original version of crio-lxc but never used by cri-o.
E.g. the 'kill' command defines the signal for 'kill' as flag '--signal' not as cmdline argument.

Add user namespace support for systemd containers

Running containers with systemd as init system currently requires privileged:true to be set in the containers SecurityContext. The container will by default run in the user namespace of the host.

• https://github.com/Drachenfels-GmbH/crio-lxc/issues/13#issuecomment-778325968
• https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod

Since process limits are set per user namespace, changing process limits in the container will affect the host.

https://systemd.io/CONTAINER_INTERFACE/#fully-unprivileged-container-payload

cri-o has added support for UID mappings (since TODO) in /etc/crio/crio.conf and sets them to the OCI runtime spec
LinuxIDMapping

# The UID mappings for the user namespace of each container. A range is
# specified in the form containerUID:HostUID:Size. Multiple ranges must be
# separated by comma.
uid_mappings = ""

# The GID mappings for the user namespace of each container. A range is
# specified in the form containerGID:HostGID:Size. Multiple ranges must be
# separated by comma.
gid_mappings = ""

crio-lxc in turn uses the mapping from the runtime spec to set lxc.idmap in the container config.

Currently I did not manage to get a plain lxc container running in user namespaces.
See lxc/lxc#3669

Run conformance tests

The kubernetes conformance tests should be run automatically.
I'm currently doing this on my development system, but it should run for each pull request
to avoid regressions. This involves automating the following steps:

• Build the container image from the provided Dockerfile with docker or buildah bud.
• Bootstrap a single node kubernetes cluster using the container image.
• Run the kubernetes conformance tests using the sonobuoy e2e plugin in the cluster.

Improve debugging.

Debugging is currently not straight forward when lxcri is started through crio.

1. The logging configuration can only be changed for alllxcri invocations.
2. Debugging the container using lxcri inspect has to be done on the scheduling node.

Proposal: Use annotations to control debugging. This should work per invocation and without the
need to access the target node.

kubectl run -i --tty ping --annotations org.linuxcontainers.lxcri.Debug  \
--image=busybox --restart=Never --rm --privileged=true --  ping -c100 192.168.1.206 

If the annotation org.linuxcontainers.lxcri.Debug is set we could:

• Enable LogConsole to use ConsoleLogger
• Set the log level to trace.
• Log the Runtime configuration when logging is set to trace
• Log the Container configuration when logging is set to trace

container doesn't show running

After a crio-lxc start, lxc-ls -f shows the container STOPPED even if it is still running (i.e. sleep 120)

Build breakage on Arch linux because of incorrect musl-gcc path

The build currently breaks on Arch Linux because of an incorrect musl-gcc path hardcoded in the project Makefile:

$ sudo make install                                                                                                                                                                                             
go fmt ./...                                                                                                                                                                                                                                  
go: downloading gopkg.in/lxc/go-lxc.v2 v2.0.0-20210205143421-c4b883be4881                                                                                                                                                                     
go: downloading golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4                                                                                                                                                                           
lxcontainer/mount_test.go                                                                                                                                                                                                                     
go build -a -ldflags '-X main.version=upstream-157-g19d0076-dirty' -o crio-lxc ./cmd                                                                                                                                                          
cc -Werror -Wpedantic -I/usr/local/include -L/usr/local/lib -llxc -lutil  -o crio-lxc-start cmd/start/crio-lxc-start.c                                                                                                                        
/usr/local/musl/bin/musl-gcc -Werror -Wpedantic -static -g -o crio-lxc-init cmd/init/crio-lxc-init.c                                                                                                                                          
make: /usr/local/musl/bin/musl-gcc: No such file or directory                                                                                                                                                                                 
make: *** [Makefile:28: crio-lxc-init] Error 127

[vagrant@vagrant-k8s crio-lxc]$ which musl-gcc
/usr/bin/musl-gcc

The following patch fixes the issue 0001-Fix-musl-gcc-path-for-Arch-linux.txt (git format-patch format)

runtime: Implement cgroup2 resource limits.

Implement resource limits for cgroup2

https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#control-groups

• Identify the tests within the sonobuoy testsuite that test for cgroup resource constraints.
• Verify whether either kubelet or crio already set resource limits, since there are no failing sonobuoy tests regarding cgroup resource limits in the default testset.

unused timeNamespace build error from staticcheck

When #50 was implemented, the assignment of timeNamespace was commented out, but not the declaration, thus an error is issued when calling $ PATH="$HOME/go/bin:$PATH" make

go fmt ./...
shfmt -w ./install.sh
golint ./...
go mod tidy
staticcheck ./...
namespaces.go:27:2: var timeNamespace is unused (U1000)

the simple workaround is to also comment out the declaration.

Publish images

Publish docker image for tagged versions.

Fixes to make compilation work

I've had trouble building lxcri from sources (since there are no package release yet)

In github.com/lxc/lxcri:main, when trying to compile libtool I get the error

libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I../../src -fPIC -DPIC -Wvla -std=gnu11 -fms-extensions -fdiagnostics-color -Wimplicit-fallthrough=5 -Wcast-align -Wstrict-prototypes -fno-strict-aliasing -fstack-clash-protection -fstack-protector-strong --param=ssp-buffer-size=4 -g -fcf-protection -Werror=implicit-function-declaration -Wlogical-op -Wmissing-include-dirs -Wold-style-definition -Winit-self -Wunused-but-set-variable -Wfloat-equal -Wsuggest-attribute=noreturn -Werror=return-type -Werror=incompatible-pointer-types -Wformat=2 -Wshadow -Wendif-labels -Werror=overflow -fdiagnostics-show-option -Werror=shift-count-overflow -Werror=shift-overflow=2 -Wdate-time -Wnested-externs -fasynchronous-unwind-tables -pipe -fexceptions -Warray-bounds -Wrestrict -Wreturn-local-addr -Wstringop-overflow -DLXCROOTFSMOUNT=\"/usr/local/lib/lxc/rootfs\" -DLXCPATH=\"/usr/local/var/lib/lxc\" -DLXC_GLOBAL_CONF=\"/usr/local/etc/lxc/lxc.conf\" -DLXCINITDIR=\"/usr/local/libexec\" -DLIBEXECDIR=\"/usr/local/libexec\" -DLXCTEMPLATEDIR=\"/usr/local/share/lxc/templates\" -DLXCTEMPLATECONFIG=\"/usr/local/share/lxc/config\" -DLOGPATH=\"/usr/local/var/log/lxc\" -DLXC_DEFAULT_CONFIG=\"/usr/local/etc/lxc/default.conf\" -DLXC_USERNIC_DB=\"/run/lxc/nics\" -DLXC_USERNIC_CONF=\"/usr/local/etc/lxc/lxc-usernet\" -DDEFAULT_CGROUP_PATTERN=\"\" -DRUNTIME_PATH=\"/run\" -DSBINDIR=\"/usr/local/sbin\" -DAPPARMOR_CACHE_DIR=\"/usr/local/var/cache/lxc/apparmor\" -I ../../src -I ../../src/lxc -I ../../src/lxc/storage -I ../../src/lxc/cgroups -DHAVE_APPARMOR -DHAVE_SECCOMP -DHAVE_SELINUX -pthread -g -O2 -Wvla -std=gnu11 -fms-extensions -Werror -MT liblxc_la-conf.lo -MD -MP -MF .deps/liblxc_la-conf.Tpo -c conf.c  -fPIC -DPIC -o .libs/liblxc_la-conf.o
conf.c: In function 'execveat_supported':
conf.c:3470:9: error: argument 3 null where non-null expected [-Werror=nonnull]
 3470 |         execveat(-1, "", NULL, NULL, AT_EMPTY_PATH);
      |         ^~~~~~~~
In file included from /usr/include/x86_64-linux-gnu/bits/sigstksz.h:24,
                 from /usr/include/signal.h:328,
                 from /usr/include/x86_64-linux-gnu/sys/param.h:28,
                 from conf.c:23:
/usr/include/unistd.h:300:12: note: in a call to function 'execveat' declared 'nonnull'
  300 | extern int execveat (int __fd, const char *__path, char *const __argv[],
      |            ^~~~~~~~
cc1: all warnings being treated as errors

This can be fixed but bumping lxc to latest 4.0.x, which is lxc 4.0.12 in build.sh

The second error I get is from lxc-go go bindings

# [gopkg.in/lxc/go-lxc.v2](http://gopkg.in/lxc/go-lxc.v2)
In file included from /root/go/pkg/mod/gopkg.in/lxc/[email protected]/container.go:11:
./lxc-binding.h:50:16: error: redefinition of 'struct lxc_groups_t'
   50 | typedef struct lxc_groups_t {
      |                ^~~~~~~~~~~~
In file included from /usr/local/include/lxc/lxccontainer.h:12,
                 from /root/go/pkg/mod/gopkg.in/lxc/[email protected]/container.go:9:
/usr/local/include/lxc/attach_options.h:80:16: note: originally defined here
   80 | typedef struct lxc_groups_t {
      |                ^~~~~~~~~~~~
In file included from /root/go/pkg/mod/gopkg.in/lxc/[email protected]/container.go:11:
./lxc-binding.h:53:3: error: conflicting types for 'lxc_groups_t'; have 'struct lxc_groups_t'
   53 | } lxc_groups_t;
      |   ^~~~~~~~~~~~
In file included from /usr/local/include/lxc/lxccontainer.h:12,
                 from /root/go/pkg/mod/gopkg.in/lxc/[email protected]/container.go:9:
/usr/local/include/lxc/attach_options.h:83:3: note: previous declaration of 'lxc_groups_t' with type 'lxc_groups_t'
   83 | } lxc_groups_t;
      |   ^~~~~~~~~~~~
make: *** [Makefile:42: lxcri] Error 2
error building at STEP "RUN make install": error while running runtime: exit status 2
ERRO[0136] exit status 2

They have to be upgraded to either latest https://pkg.go.dev/gopkg.in/lxc/[email protected] or even better, update go.mod to use

[github.com/lxc/go-lxc](http://github.com/lxc/go-lxc) v0.0.0-20230926171149-ccae595aa49e
[golang.org/x/sys](http://golang.org/x/sys) v0.12.0

This is what I did when working with https://github.com/drachenfels-de/lxcri:fixes which is ahead of this repository, but x/sys v0.12.0 also needs you to bump go version to 1.17

runtime: Implement mount option filtering

liblxc fails to mount filesystems if unsupported mount options are passed to lxc.mount.entry.
It does however support the option optional to continue with container startup even if a mountpoint fails.
See man lxc.container.conf

crio-lxc should support filtering of mount options, to handle unsupported mount options gracefully.

handle kill correctly

Currently, kill.go just resolves the container's init pid and sends it the "kill signal" from the container's spec. This is wrong for several reasons.

1. liblxc sticks an lxc-init in front of every task; it does this because it assumes that this lxc-init is going to be pid 1 in every application container, but that is not true in the pod case. The pod's lxc-init really is pid 1, but other containers (with their lxc-inits) spawned in this pod will not be pid 1.

2. We assume that killing the init will kill everything in the container, presumably because this is true for pid 1 in a pidns. But since in the k8s view of containers, the pod owns pid 1, and the containers don't. So our current implementation kills the contianer's lxc-init which is not pid 1, which simply results in the service in the container being reparented to the pod's lxc-init, which does not actually stop the service.

3. We send the signal to lxc-init, instead of the real pid 1. The OCI spec allows the container image to say what signal it would like to receive in order to be killed. Presumably then, the application code understands when it sees a particular signal, it needs to shut down. So we should send that signal to the application code instead of lxc-init, which just swallows it.

It seems like the easiest way to solve this is to configure crio-lxc to just not use lxc-init. Since the applications already understand that they don't really have an init, we don't need to babysit them. It doesn't look like it's currently possible to avoid using lxc-init, though.

Create a static release bundle.

Create a static release that can be deployed to any x86_64 server with a recent enough kernel.
Hash and sign releases with a GPG key.

• Build packages using github CI ?

lxc: Unknown capability perfmon

Deploying cilium fails with

ERROR conf - conf.c:dropcaps_except:2451 - Unknown capability perfmon

Workaround

Disable capabilities support by settingCRIO_LXC_CAPABILITIES=false in /etc/default/crio-lxc

Problem

Cri-o requests capablities that are unknown to the installed liblxc version.
Commit lxc/lxc@7b4cd46 added capabilities support for new capabilities introduced by linux 5.8 CAP_BPF and CAP_PERFMON. The forked liblxc version is not recent enough and must be upgraded.

Detailed Description

Capabilities are set by cri-o. For privileged containers e.g cilium all supported capabilities are set.
cri-o uses the library github.com/syndtr/gocapability to list all supported capabilities.

The gocapability library was updated in syndtr/gocapability#17 to support CAP_BPF and CAP_PERFMON.

cri-o then upgraded the gocapability dependency in cri-o/cri-o#4462

-       github.com/syndtr/gocapability v0.0.0-20180916011248-d98352740cb2
+       github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635

Conclusion

• Track whether new capabilities are introduced when changing the kernel version.
• Ensure that the liblxc version is recent enough for the underlying kernel and supports all available capabilities.

Replace cli library.

Replace the cli library urfave/cli with spf13/cobra.
urfave/cli is poorly maintained and cobra has some very nice features
and a cleaner interface. Migration should be easy.

tmpcopyup runc-specific extension prevents container from starting up

My systemd-based LXC container fails to start with the following error:

lxc ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68 20210210222236.599 DEBUG    storage - storage/storage.c:storage_query:233 - Detected rootfs type "dir"
lxc ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68 20210210222236.601 DEBUG    conf - conf.c:lxc_mount_rootfs:1262 - Mounted rootfs "/var/lib/containers/storage/overlay/1164bb7a6218ebd528572647a87e0335ca248c2e363c75da2d250f85072e1cf4/merged" onto "/usr/local/lib/lxc/rootfs" with options "(null)"
lxc ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68 20210210222236.618 DEBUG    conf - conf.c:mount_entry:2009 - Mounted "proc" on "/usr/local/lib/lxc/rootfs//proc" with filesystem type "proc"
lxc ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68 20210210222236.625 DEBUG    conf - conf.c:mount_entry:2009 - Mounted "tmpfs" on "/usr/local/lib/lxc/rootfs//dev" with filesystem type "tmpfs"
lxc ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68 20210210222236.656 DEBUG    conf - conf.c:mount_entry:2009 - Mounted "devpts" on "/usr/local/lib/lxc/rootfs//dev/pts" with filesystem type "devpts"
lxc ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68 20210210222236.681 DEBUG    conf - conf.c:mount_entry:2009 - Mounted "mqueue" on "/usr/local/lib/lxc/rootfs//dev/mqueue" with filesystem type "mqueue"
lxc ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68 20210210222236.685 DEBUG    conf - conf.c:mount_entry:2009 - Mounted "sysfs" on "/usr/local/lib/lxc/rootfs//sys" with filesystem type "sysfs"
lxc ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68 20210210222236.740 ERROR    utils - utils.c:safe_mount:1197 - Invalid argument - Failed to mount "tmpfs" onto "/usr/local/lib/lxc/rootfs//run"
lxc ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68 20210210222236.742 ERROR    conf - conf.c:mount_entry:1940 - Invalid argument - Failed to mount "tmpfs" on "/usr/local/lib/lxc/rootfs//run"
lxc ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68 20210210222236.743 ERROR    conf - conf.c:lxc_setup:3338 - Failed to setup mount entries
lxc ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68 20210210222236.744 ERROR    start - start.c:do_start:1267 - Failed to setup container "ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68"
lxc ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68 20210210222236.757 ERROR    sync - sync.c:__sync_wait:36 - An error occurred in another process (expected sequence number 5)
lxc ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68 20210210222236.765 ERROR    start - start.c:__lxc_start:2082 - Failed to spawn container "ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68"
lxc ac70e773b77df4a731f1c5d420baaa0c247515b3e696b8c4385f6cbe6a596b68 20210210222236.766 WARN     start - start.c:lxc_abort:1012 - No such process - Failed to send SIGKILL via pidfd 17 for process 511687

Looking at the LXC config which is generated, I see the following:

lxc.mount.entry = tmpfs /var/lib/containers/storage/overlay/b302f8e1e4ccdd5a182653c2664f57f00fdf0f3544af7026848e8844944d98d7/merged/run tmpfs rw,rprivate,noexec,nosuid,nodev,tmpcopyup,create=dir
lxc.mount.entry = tmpfs /var/lib/containers/storage/overlay/b302f8e1e4ccdd5a182653c2664f57f00fdf0f3544af7026848e8844944d98d7/merged/run/lock tmpfs rw,rprivate,noexec,nosuid,nodev,tmpcopyup,create=dir
lxc.mount.entry = tmpfs /var/lib/containers/storage/overlay/b302f8e1e4ccdd5a182653c2664f57f00fdf0f3544af7026848e8844944d98d7/merged/tmp tmpfs rw,rprivate,noexec,nosuid,nodev,tmpcopyup,create=dir
lxc.mount.entry = tmpfs /var/lib/containers/storage/overlay/b302f8e1e4ccdd5a182653c2664f57f00fdf0f3544af7026848e8844944d98d7/merged/var/log/journal tmpfs rw,rprivate,noexec,nosuid,nodev,tmpcopyup,create=dir

It looks like the tmpcopyup mount option is a non-OCI runc extension and this is what is making the mount fail:

• https://github.com/cri-o/cri-o/blob/master/server/container_create_linux.go#L159
• cri-o/cri-o#1434 (comment)

If that is true, would there be a workaround? Are you running systemd-based containers on your cluster?

Here are the logs (this is a shareable testcase, I can provide more if needed, including the rootfs
02.10_22.36.27.zip

Edit: I see tmpcopyup is also implemeneted by crun

tmpfs unsupported mount options tmpcopyup, rprivate

crio-o seems adds the mount options rprivate and tmpcopyup to tmpfs container mounts. See drachenfels-de/lxcri-defork#13

As far as I can see these options are added to all tmpfs mounts but /dev This includes:

/run
/run/lock
/tmp
/var/log/journal

The kernel log (dmesg) then shows the following errors:

[721537.855698] tmpfs: Unknown parameter 'rprivate'
[721602.257009] tmpfs: Unknown parameter 'tmpcopyup'

The mount options rprivate and tmpcopyup are not support for tmpfs. See man 5 tmpfs

Platform

Linux k8s-cluster2-controller 5.10.11-arch1-1 drachenfels-de/lxcri#7 SMP PREEMPT Wed, 27 Jan 2021 13:53:16 +0000 x86_64 GNU/Linux

Improve config file mechanism

I really like the crio config command to generate the configuration file based on the passed flags and environment variables.
This makes it very convenient to generate a matching config file.
lxcri should support a similar mechanism for convenience.

• Replace the environment file /etc/default/lxcri with a YAML config file in /etc/lxcri/lxcri.yaml.
• Add a cli cmd lxcri {args} config to create and edit the config file.
• Also add a flag and environment variable to overwrite the default location of the config file path, to support multiple environments.

Recreate of container fails with 'Error reserving ctr name'

Recreating a container where the process was killed manually fails with the error:

Response error: error reserving ctr

Related information:

• https://bugzilla.redhat.com/show_bug.cgi?id=1779421
• cri-o/cri-o#3183

root@container-node:~# crio --version
crio version 1.19.0-dev
Version:       1.19.0-dev
GitCommit:     a7b54a80cb5f4818b2175e31fe44a84f6755a5cd
GitTreeState:  dirty
BuildDate:     2020-07-07T13:34:13Z
GoVersion:     go1.14.4
Compiler:      gc
Platform:      linux/amd64
Linkmode:      dynamic

root@container-node:~# crictl version
Version:  0.1.0
RuntimeName:  cri-o
RuntimeVersion:  1.19.0-dev
RuntimeApiVersion:  v1alpha1

root@container-node:~# crio-lxc --version
crio-lxc version af7046654cd4967cfc993a7177a00fe32984826d-dirty

ul 10 15:54:05 container-node crio[130951]: time="2020-07-10 15:54:05.661488404+02:00" level=debug msg="Response error: error reserving ctr name k8s_sgg-alpine-exim_nginx-sandbox_default_hdishd83djaidwnduwk28bcsb_0 for id baca3e0064284658e211ecc61a56f1b350a222aad364e4e11cfb8aecec93d51a: name is reserved\nreserving container name\ngithub.com/cri-o/cri-o/server.(*Server).CreateContainer
/root/packages/cri-o/server/container_create.go:532\nk8s.io/cri-api/pkg/apis/runtime/v1alpha2._RuntimeService_CreateContainer_Handler.func1
/root/packages/cri-o/vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go:7799\ngithub.com/cri-o/cri-o/internal/log.UnaryInterceptor.func1
/root/packages/cri-o/internal/log/interceptors.go:57\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
/root/packages/cri-o/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25\ngithub.com/cri-o/cri-o/server/metrics.UnaryInterceptor.func1
/root/packages/cri-o/server/metrics/interceptors.go:24\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
/root/packages/cri-o/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
/root/packages/cri-o/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34\nk8s.io/cri-api/pkg/apis/runtime/v1alpha2._RuntimeService_CreateContainer_Handler
/root/packages/cri-o/vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go:7801\ngoogle.golang.org/grpc.(*Server).processUnaryRPC
/root/packages/cri-o/vendor/google.golang.org/grpc/server.go:1082\ngoogle.golang.org/grpc.(*Server).handleStream
/root/packages/cri-o/vendor/google.golang.org/grpc/server.go:1405\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1
/root/packages/cri-o/vendor/google.golang.org/grpc/server.go:746\nruntime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1373" file="go-grpc-middleware/chain.go:25" id=0c9aad0f-c590-444d-b111-05f724672c86 name=/runtime.v1alpha2.RuntimeService/CreateContainer

Fix signal (SIGINT) handling when running kubectl with terminal attached

Currently it's not possible to interrupt the following command using Control-C

kubectl run -i --tty ping --image=busybox --restart=Never --rm --privileged=true -- ping github.com

Check if lxcri-init must set the signal mask before executing the container process.
The container process should inherit the signal mask from lxcri-init.
See libc manual - 24.7.3 Process Signal Mask

Invalid argument - Failed to mount "sysfs"

Hi guys, I'm trying lxcri in my cluster, but got failed on container creating, here is kubelet error log:

Oct 18 19:42:17 slave2 kubelet[1206]: E1018 19:42:17.402654    1206 remote_runtime.go:116] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = container create failed: [lxcri:create.go:37] failed to configure container: failed  to set hostname: failed to open container uts namespace "": open : no such file or directory

and my cluster info:

# uname -a 
Linux slave2 5.4.61-050461-generic #202008260931 SMP Wed Aug 26 09:34:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

# kubelet --version 
Kubernetes v1.20.1

# crio -v
crio version 1.18.3
Version:       1.18.3
GitCommit:     61de18161fb4ccda720768c001713592b5a04e46
GitTreeState:  clean
BuildDate:     2020-07-17T13:47:42Z
GoVersion:     go1.14
Compiler:      gc
Platform:      linux/amd64
Linkmode:      static

# lxcri -v 
lxcri version v0.12.1-11-gcbb8850

make check' cannot run on master branch ("cgroupfs manager conmon cgroup should be 'pod' or empty")

Hi there,

Not sure if this is the proper place to get help on the package here, feel free to let me know if there is a better place

I am trying to install and run crio-lxc and attempted to run the tests. This fails with the following fatal error when CRI-O checks the runtime configuration:

$ PACKAGES_DIR=/home/vrubiolo/Documents/Dev/repos/community/crio-lxc/packages/ make check 2>&1 | tee make_check_log.txt
[...]
not ok 1 basic cri-o workings
# (from function `crictl' in file test/helpers.bash, line 57,
#  in test file test/basic.bats, line 12)
#   `crictl runp test/basic-pod-config.json' failed
# time="2020-11-16T14:13:52+01:00" level=info msg="Starting CRI-O, version: 1.20.0-dev, git: f261f0560237c5c76a20db7f7c54214c026fe0e4(clean)"
# time="2020-11-16 14:13:52.233314056+01:00" level=info msg="Node configuration value for hugetlb cgroup is true"
# time="2020-11-16 14:13:52.233325596+01:00" level=info msg="Node configuration value for pid cgroup is true"
# time="2020-11-16 14:13:52.233433168+01:00" level=info msg="Node configuration value for memoryswap cgroup is true"
# time="2020-11-16 14:13:52.237398249+01:00" level=info msg="Node configuration value for systemd CollectMode is true"
# time="2020-11-16 14:13:52.237852416+01:00" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
# time="2020-11-16 14:13:52.238011089+01:00" level=warning msg="Forcing ctr_stop_timeout to lowest possible value of 30s"
# time="2020-11-16 14:13:52.238027582+01:00" level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_NET_RAW, CAP_SETGID, CAP_SETUID, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_SYS_CHROOT, CAP_KILL"
# time="2020-11-16 14:13:52.238353453+01:00" level=info msg="Using conmon executable: /home/vrubiolo/Documents/Dev/repos/community/crio-lxc/packages//conmon/bin/conmon"
# time="2020-11-16 14:13:52.239838784+01:00" level=info msg="Conmon does support the --sync option"
# time="2020-11-16 14:13:52.239864465+01:00" level=info msg="Using pinns executable: /home/vrubiolo/Documents/Dev/repos/community/crio-lxc/packages//cri-o/bin/pinns"
# time="2020-11-16 14:13:52.239877819+01:00" level=info msg="Seccomp is disabled by the system or at CRI-O build-time"
# time="2020-11-16 14:13:52.239885843+01:00" level=info msg="AppArmor is disabled by the system or at CRI-O build-time"
# time="2020-11-16 14:13:52.239929963+01:00" level=fatal msg="Validating runtime config: cgroupfs manager conmon cgroup should be 'pod' or empty"
# time="2020-11-16T14:13:54+01:00" level=fatal msg="failed to connect: failed to connect: context deadline exceeded"

Could sb shed light on the error? It is not obvious to me what the problem is (wrong cgroup name for conmon but I don't know why).

Here is the full log : make_check_log.txt

Git repo information for the packages : git_info.txt

This is Fedora 32.

Thanks much for your help and guidance !

Warning: pause should be the first process in a pod

The pod logs will always contain this warning, since lxc always launches init.lxc or init.lxc.static as first process.
The command launched from init.lxc configured by lxc.init.cmd will always have pid 2 within the container.

The kubernetes /pause command that is used for pod creation issues this warning.
https://github.com/verb/kubernetes/blob/c1520e15ff35fa7ec19b6aa999362adef5963b9d/build/pause/pause.c#L36

runtime: Implement POSIX platform hooks

Implement and test POSIX platform hooks defined by the runtime spec.

https://github.com/opencontainers/runtime-spec/blob/master/config.md#posix-platform-hooks

Most of the OCI hooks could probably simply be mapped to the lxc CONTAINER HOOKS (man lxc.container.conf).

Make namespaces configurable for `lxcri exec`

Add command option(s) to lxcri exec to control which namespaces should be attached.
This should be implemented similar to nsenter or lxc-attach

sonobuoy conformance : analyse failing conformance tests

root@k8s-node1:~# crio --version
crio version 1.18.3
Version:       1.18.3
GitCommit:     61de18161fb4ccda720768c001713592b5a04e46
GitTreeState:  clean
BuildDate:     2020-08-07T11:44:34Z
GoVersion:     go1.14.7
Compiler:      gc
Platform:      linux/amd64
Linkmode:      dynamic

crio-lxc version v0.1.1-5-g2e29441-dirty

Sonobuoy Version: v0.19.0
MinimumKubeVersion: 1.17.0
MaximumKubeVersion: 1.19.99
GitSHA: e03f9ee353717ccc5f58c902633553e34b2fe46a
API Version:  v1.19.0

{
  "plugins": [
    {
      "plugin": "e2e",
      "node": "global",
      "status": "complete",
      "result-status": "failed",
      "result-counts": {
        "failed": 6,
        "passed": 297,
        "skipped": 4929
      },
      "progress": {
        "name": "e2e",
        "node": "global",
        "timestamp": "2020-09-01T12:06:25.746039005Z",
        "msg": "Test Suite completed",
        "total": 303,
        "completed": 297,
        "failures": [
          "[sig-scheduling] SchedulerPreemption [Serial] validates lower priority pod preemption by critical pod [Conformance]",
          "[k8s.io] Kubelet when scheduling a read only busybox container should not write to root filesystem [LinuxOnly] [NodeConformance] [Conformance]",
          "[sig-scheduling] SchedulerPreemption [Serial] validates basic preemption works [Conformance]",
          "[sig-storage] Secrets should be consumable from pods in volume as non-root with defaultMode and fsGroup set [LinuxOnly] [NodeConformance] [Conformance]",
          "[sig-apps] Daemon set [Serial] should rollback without unnecessary restarts [Conformance]",
          "[sig-storage] Projected secret should be consumable from pods in volume as non-root with defaultMode and fsGroup set [LinuxOnly] [NodeConformance] [Conformance]"
        ]
      }
    },
    {
      "plugin": "systemd-logs",
      "node": "k8s-node1",
      "status": "complete",
      "result-status": "passed",
      "result-counts": {
        "passed": 1
      }
    }
  ],
  "status": "complete",
  "tar-info": {
    "name": "202009011039_sonobuoy_05d76421-09ae-4630-9f8c-3f3825ab4803.tar.gz",
    "created": "2020-09-01T12:06:31.351085397Z",
    "sha256": "246f307f6259b937a647a5cec3d45f7794ca65688c9a9ac835d3ac1522344236",
    "size": 4252085
  }
}

we don't check errors sometimes

#7 (comment)

go-lxc fails to build: error: unknown type name 'lxc_groups_t'

Hi there,

I am attempting to follow the installation instructions on Debian 10 but this fails during the crio-lxc build with the following error:

$ sudo make install 2>&1 | tee ~/crio-lxc-build-failure.txt
go fmt ./...
go: finding gopkg.in/lxc/go-lxc.v2 v2.0.0
go build -a -ldflags '-X main.version=v0.9.5-0-g8629c80' -o crio-lxc ./cmd
go: finding gopkg.in/lxc/go-lxc.v2 v2.0.0
# gopkg.in/lxc/go-lxc.v2
/root/go/pkg/mod/github.com/!drachenfels-!gmb!h/[email protected]/container.go:1299:13: could not determine kind of name for C.go_lxc_attach
/root/go/pkg/mod/github.com/!drachenfels-!gmb!h/[email protected]/container.go:1435:13: could not determine kind of name for C.go_lxc_attach_no_wait
/root/go/pkg/mod/github.com/!drachenfels-!gmb!h/[email protected]/container.go:1362:13: could not determine kind of name for C.go_lxc_attach_run_wait
cgo: 
gcc errors for preamble:
In file included from /root/go/pkg/mod/github.com/!drachenfels-!gmb!h/[email protected]/container.go:11:
./lxc-binding.h:59:25: error: unknown type name 'lxc_groups_t'
   uid_t uid, gid_t gid, lxc_groups_t groups,
                         ^~~~~~~~~~~~
./lxc-binding.h:69:25: error: unknown type name 'lxc_groups_t'
   uid_t uid, gid_t gid, lxc_groups_t groups,
                         ^~~~~~~~~~~~
./lxc-binding.h:78:25: error: unknown type name 'lxc_groups_t'
   uid_t uid, gid_t gid, lxc_groups_t groups,
                         ^~~~~~~~~~~~

I suspect this is because the version of lxc that the build finds is not recent enough. I have however followed the steps to install lxc from source, albeit modified to be used w/ sudo:

git clone https://github.com/lxc/lxc.git
cd lxc
./autogen.sh
./configure --enable-bash=no --enable-tools=no \
  --enable-commands=no --enable-seccomp=yes \
  --enable-capabilities=yes --enable-apparmor=yes
sudo make install

echo /usr/local/lib | sudo tee /etc/ld.so.conf.d/local.conf
sudo ldconfig

cf vrubiolo@2fb6145

Full log at crio-lxc-build-failure.txt

Could you advise on what could be wrong here? Thanks much!

Issues seen during k8s deployment with kubeadm

Hi Ruben,

I am seeing some issues when attempting to deploy k8s using kubeadm during the preflight checks, namely:

[preflight] Some fatal errors occurred:
        [ERROR FileExisting-crictl]: crictl not found in system path
        [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
        [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
        [ERROR KubeletVersion]: the kubelet version is higher than the control plane version. This is not a supported version skew and may lead to a malfunctional cluster. Kubelet version: "1.20.1" Control plane version: "1.19.6"

• The first one is probably a PATH-related error or similar
• The 2nd and 3rd are likely some settings, I am looking at instructions like https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#letting-iptables-see-bridged-traffic to fix, any input welcome
• it's the last one which I am not sure about, given that the INSTALL.md instructions are supposed to pick up the latest of k8s binaries
  • on this front, it looks like the CHECKSUM value in this block is wrong as it fails the check a few lines below:

ARCH="linux-amd64"
RELEASE="1.20.1"
ARCHIVE=kubernetes-server-$ARCH.tar.gz
CHECKSUM="fb56486a55dbf7dbacb53b1aaa690bae18d33d244c72a1e2dc95fb0fcce45108c44ba79f8fa04f12383801c46813dc33d2d0eb2203035cdce1078871595e446e"
DESTDIR="/usr/local/bin"

Full log is below, let me know if you need more information. I am running with sudo kubeadm init --config cluster-init.yaml -v 5 2>&1 | tee kubeadm.log
kubeadm.log

sonobuoy: pod/sonobuoy-e2e-job Error - getting HTTP client: Couldn't parse CaCert PEM

sonobuoy run --wait fails with:

root@container-node:~# cat /var/log/containers/sonobuoy-e2e-job-b3daf56b6e8c406d_sonobuoy_sonobuoy-worker-20747ee04af93356ded9f462e0ac6b5fc0e3539e1e643ba5f7329924aa0cb2e5.log 
2020-08-05T16:44:22.723075467+02:00 stderr F Error: getting HTTP client: Couldn't parse CaCert PEM
2020-08-05T16:44:22.723322541+02:00 stderr F Usage:
2020-08-05T16:44:22.723322541+02:00 stderr F   sonobuoy worker global [flags]
2020-08-05T16:44:22.723322541+02:00 stderr F 
2020-08-05T16:44:22.723322541+02:00 stderr F Flags:
2020-08-05T16:44:22.723322541+02:00 stderr F   -h, --help   help for global
2020-08-05T16:44:22.723322541+02:00 stderr F 
2020-08-05T16:44:22.723322541+02:00 stderr F Global Flags:
2020-08-05T16:44:22.723322541+02:00 stderr F       --alsologtostderr                  log to standard error as well as files
2020-08-05T16:44:22.723322541+02:00 stderr F   -d, --debug                            Enable debug output (includes stack traces)
2020-08-05T16:44:22.723322541+02:00 stderr F       --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
2020-08-05T16:44:22.723322541+02:00 stderr F       --log_dir string                   If non-empty, write log files in this directory
2020-08-05T16:44:22.723322541+02:00 stderr F       --log_file string                  If non-empty, use this log file
2020-08-05T16:44:22.723322541+02:00 stderr F       --log_file_max_size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
2020-08-05T16:44:22.723322541+02:00 stderr F       --logtostderr                      log to standard error instead of files (default true)
2020-08-05T16:44:22.723322541+02:00 stderr F       --skip_headers                     If true, avoid header prefixes in the log messages
2020-08-05T16:44:22.723322541+02:00 stderr F       --skip_log_headers                 If true, avoid headers when opening log files
2020-08-05T16:44:22.723322541+02:00 stderr F       --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
2020-08-05T16:44:22.723322541+02:00 stderr F   -v, --v Level                          number for the log level verbosity (default 0)
2020-08-05T16:44:22.723322541+02:00 stderr F       --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging
2020-08-05T16:44:22.723322541+02:00 stderr F 
2020-08-05T16:44:22.723322541+02:00 stderr F time="2020-08-05T14:44:22Z" level=error msg="getting HTTP client: Couldn't parse CaCert PEM"

some question about lxcri

Hello
I have some question about lxcri:

1. Is it support both lxc and lxd?
2. Is it better than lxe?
3. Have you any community for lxcri?

Can lxcri work with containerd instead of cri-o?

List containers in runtime directory

lxcri currently can not list containers within the runtime root.
lxcri containers are lxc containers, so all lxc tools can operate on the runtime directory e.g

lxc-ls -f -P /run/lxcri/

But to get lxcri specific output I propose to implement something similar to crictl inspect:

• Add an API function to the runtime that loads a list of containers (along with the current state) from the runtime directory.
• Add a cli command that accepts a golang text/template, wich is rendered in the context of the selected container / list of containers.

Add SELinux support.

It should be as simple as mapping specs.Process.SelinuxLabel to lxc.selinux.context.