Kata-containers - failed to create containerd task: failed to create shim task

[jasek5]

Kata-containers - failed to create containerd task: failed to create shim task

[frezbo]

maybe try talosctl logs syslogd

[jasek5]

maybe try talosctl logs syslogd

Hi and thank you.

Those logs are from the one node only (that node which got kata pod assigned). Maybe there can be found logs when I "kubectl" deleted the example from the instruction. From my point of view - there are lot of same errors from the pod description output.

(I uniqued output for better readability via removing timestamps and: - gsed -E 's/"timestamp":"[^"]*"//; s/"time=\\"[^\\"]*\\"//' | awk '!seen[$0]++') because output is too long for GitHub.

<obfuscated IP>: ": dial unix /run/vc/vm/5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927/clh-api.sock: connect: no such file or directory\" name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":4,"severity":4,"tag":"kata",}
<obfuscated IP>: {"content": level=warning msg=\"clh.VmmPingGet API call failed\" error=\"Get \\\"http://localhost/api/v1/vmm.ping\\\": dial unix /run/vc/vm/5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927/clh-api.sock: connect: no such file or directory\" name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":4,"severity":4,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"State() start\" container=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=containerd-kata-shim-v2\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=warning msg=\"cloud-hypervisor init failed\" error=\"Failed to connect to API (timeout 10s): error: Get \\\"http://localhost/api/v1/vmm.ping\\\": dial unix /run/vc/vm/5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927/clh-api.sock: connect: no such file or directory reason: \" name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":4,"severity":4,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Stop Sandbox\" function=StopVM name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Stopping Cloud Hypervisor\" name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"virtiofsd quits\" name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers/hypervisor subsystem=virtiofsd\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"stop virtiofsDaemon\" name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=warning msg=\"The virtiofsd had stopped\" name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":4,"severity":4,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Cleanup VM\" name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"removing vm sockets\" name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=warning msg=\"removing vm socket failed\" error=\"remove /run/vc/vm/5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927/clh.sock: no such file or directory\" name=containerd-shim-v2 path=/run/vc/vm/5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927/clh.sock pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":4,"severity":4,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Cleanup vm path\" dir=/run/vc/vm/5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 link=/run/vc/vm/5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Removing network after failure in createSandbox\" name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Detaching endpoint\" endpoint-type=virtual name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers subsystem=network\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Endpoints removed\" name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers subsystem=network\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Deleting sandbox \u0026{%!s(*v2.Manager=\u0026{/sys/fs/cgroup /sys/fs/cgroup/kubepods/besteffort/pode536be43-39e2-4d84-93a5-76a89fd6db22/kata_5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927}) /kubepods/besteffort/pode536be43-39e2-4d84-93a5-76a89fd6db22/kata_5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 %!s(*specs.LinuxCPU=\u0026{\u003cnil\u003e \u003cnil\u003e \u003cnil\u003e \u003cnil\u003e \u003cnil\u003e \u003cnil\u003e   \u003cnil\u003e}) [{%!s(bool=false)  %!s(*int64=\u003cnil\u003e) %!s(*int64=\u003cnil\u003e) rwm} {%!s(bool=true) c %!s(*int64=0xc000272418) %!s(*int64=0xc000272470) rwm} {%!s(bool=true) c %!s(*int64=0xc000272478) %!s(*int64=0xc000272480) rwm} {%!s(bool=true) c %!s(*int64=0xc0002859e8) %!s(*int64=0xc0002859f0) rwm} {%!s(bool=true) c %!s(*int64=0xc000285a18) %!s(*int64=0xc000285a20) rwm} {%!s(bool=true) c %!s(*int64=0xc000285a48) %!s(*int64=0xc000285a50) rwm} {%!s(bool=true) c %!s(*int64=0xc000285a78) %!s(*int64=0xc000285a80) rwm} {%!s(bool=true) c %!s(*int64=0xc000285aa8)","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Already stopped\" name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=Cleanup function=Cleanup name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Cleanup agent\" name=containerd-shim-v2 path=/run/kata-containers/shared/sandboxes/5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927/shared pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=virtcontainers subsystem=fs_share\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Create() end\" container=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=containerd-kata-shim-v2\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"State() end\" container=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=containerd-kata-shim-v2\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Delete() start\" container=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=containerd-kata-shim-v2\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Delete() end\" container=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=containerd-kata-shim-v2\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Shutdown() start\" container=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 name=containerd-shim-v2 pid=71424 sandbox=5d1954fc3d679ef74121dd1928b6c663ba6f9213b0b1861c0b8b738d6e2aa927 source=containerd-kata-shim-v2\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"loaded configuration\" file=/usr/local/share/kata-containers/configuration.toml format=TOML name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=katautils\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=debug default-kernel-parameters= name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=katautils\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"container rootfs: /run/containerd/io.containerd.runtime.v2.task/k8s.io/358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6/rootfs\" source=virtcontainers subsystem=oci\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"shm-size detected: 67108864\" source=virtcontainers subsystem=oci\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=warning msg=\"Could not add /dev/kvm to the devices cgroup\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=cgroups\n","facility":0,"hostname":"localhost","priority":4,"severity":4,"tag":"kata",}
<obfuscated IP>: {"content": level=warning msg=\"Could not add /dev/mshv to the devices cgroup\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=cgroups\n","facility":0,"hostname":"localhost","priority":4,"severity":4,"tag":"kata",}
<obfuscated IP>: {"content": level=warning msg=\"Could not add /dev/vfio/vfio to the devices cgroup\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=cgroups\n","facility":0,"hostname":"localhost","priority":4,"severity":4,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"restore sandbox failed\" error=\"open /run/vc/sbs/358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6/persist.json: no such file or directory\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers subsystem=sandbox\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"creating Sandbox\" function=CreateVM name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Sandbox not found creating\" function=CreateVM name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Adding HybridVSock\" cid=3 name=containerd-shim-v2 path=/run/vc/vm/358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6/clh.sock pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"get Capabilities\" function=Capabilities name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Adding share volume to hypervisor: kataShared\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"veth interface found\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers subsystem=network\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Attaching endpoint\" endpoint-type=virtual hotplug=false name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers subsystem=network\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"connect TCFilter to VM network\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers subsystem=network\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Adding Endpoint of type \u0026{virtual  {{{150 1500 1000 eth0 32:f1:71:7b:46:f6 up|broadcast|multicast 69699 151 0 \u003cnil\u003e  0xc000386b40 0 0 1 0xc0003cf3b0 ether \u003cnil\u003e up 0 0 2 2 65536 65535 [] 0 \u003cnil\u003e} veth} {[]  [] []} 0xc00015c840 [10.244.4.174/32 eth0 fe80::30f1:71ff:fe7b:46f6/64] [{Ifindex: 150 Dst: \u003cnil\u003e Src: \u003cnil\u003e Gw: 10.244.4.159 Flags: [] Table: 254 Realm: 0} {Ifindex: 150 Dst: 10.244.4.159/32 Src: \u003cnil\u003e Gw: \u003cnil\u003e Flags: [] Table: 254 Realm: 0} {Ifindex: 150 Dst: fe80::/64 Src: \u003cnil\u003e Gw: \u003cnil\u003e Flags: [] Table: 254 Realm: 0}] [{150 10 64 5 0 0 ff02::16 33:33:00:00:00:16 \u003cnil\u003e 0 0 0}]} {{ea75b5bb-6169-4493-a274-91168c97ca59 br0_kata {tap0_kata 32:f1:71:7b:46:f6 []} [0xc00006a740] []} {eth0 3e:00:d8:e6:c6:70 []} 2} false false}\" endpoint-type=\"\u0026{virtual  {{{150 1500 1000 eth0 32:f1:71:7b:46:f6 up|broadcast|multicast 69699 151 0 \u003cnil\u003e  0xc000386b40 0 0 1 0xc0003cf3b0 ether \u003cnil\u003e up 0 0 2 2 65536 65535 [] 0 \u003cnil\u003e} veth} {[]  [","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Storing the Cloud Hypervisor network configuration: \u0026{Tap:\u003cnil\u003e Ip:0xc000408d70 Mask:0xc000408dd0 Mac:0xc000408d60 HostMac:\u003cnil\u003e Mtu:\u003cnil\u003e Iommu:0xc0003352f1 NumQueues:0xc0003352e8 QueueSize:0xc0003352ec VhostUser:0xc0003352f0 VhostSocket:\u003cnil\u003e VhostMode:0xc000408de0 Id:\u003cnil\u003e PciSegment:\u003cnil\u003e RateLimiterConfig:\u003cnil\u003e}\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"endpoints found after scan\" endpoints=\"[0xc0004c7b08]\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers subsystem=network\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Endpoints added\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers subsystem=network\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"State() start\" container=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=containerd-kata-shim-v2\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Starting VM\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers subsystem=sandbox\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Get Sandbox Console\" function=GetVMConsole id=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"starting Sandbox\" function=StartVM name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Starting virtiofsDaemon\" function=setupVirtiofsDaemon name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info name=containerd-shim-v2 path=/usr/local/libexec/virtiofsd pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=virtiofsd\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info args=\"--syslog --cache=auto --shared-dir=/run/kata-containers/shared/sandboxes/358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6/shared --fd=3 --thread-pool-size=1 --announce-submounts\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=virtiofsd\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info name=containerd-shim-v2 path=/usr/local/bin/cloud-hypervisor pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info args=\"--api-socket /run/vc/vm/358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6/clh-api.sock -v\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=warning msg=\"clh.VmmPingGet API call failed\" error=\"Get \\\"http://localhost/api/v1/vmm.ping\\\": dial unix /run/vc/vm/358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6/clh-api.sock: connect: no such file or directory\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":4,"severity":4,"tag":"kata",}
<obfuscated IP>: {"content":"Waiting for vhost-user socket connection...","facility":1,"hostname":"k8s-node-02","priority":14,"severity":6,"tag":"virtiofsd",}
<obfuscated IP>: {"content": level=warning msg=\"cloud-hypervisor init failed\" error=\"Failed to connect to API (timeout 10s): error: Get \\\"http://localhost/api/v1/vmm.ping\\\": dial unix /run/vc/vm/358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6/clh-api.sock: connect: no such file or directory reason: \" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":4,"severity":4,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Stop Sandbox\" function=StopVM name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Stopping Cloud Hypervisor\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"virtiofsd quits\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=virtiofsd\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"stop virtiofsDaemon\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=warning msg=\"The virtiofsd had stopped\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":4,"severity":4,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Cleanup VM\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"removing vm sockets\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=warning msg=\"removing vm socket failed\" error=\"remove /run/vc/vm/358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6/clh.sock: no such file or directory\" name=containerd-shim-v2 path=/run/vc/vm/358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6/clh.sock pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":4,"severity":4,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Cleanup vm path\" dir=/run/vc/vm/358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 link=/run/vc/vm/358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Removing network after failure in createSandbox\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Detaching endpoint\" endpoint-type=virtual name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers subsystem=network\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Endpoints removed\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers subsystem=network\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Already stopped\" name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Deleting sandbox \u0026{%!s(*v2.Manager=\u0026{/sys/fs/cgroup /sys/fs/cgroup/kubepods/besteffort/pode536be43-39e2-4d84-93a5-76a89fd6db22/kata_358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6}) /kubepods/besteffort/pode536be43-39e2-4d84-93a5-76a89fd6db22/kata_358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 %!s(*specs.LinuxCPU=\u0026{\u003cnil\u003e \u003cnil\u003e \u003cnil\u003e \u003cnil\u003e \u003cnil\u003e \u003cnil\u003e   \u003cnil\u003e}) [{%!s(bool=false)  %!s(*int64=\u003cnil\u003e) %!s(*int64=\u003cnil\u003e) rwm} {%!s(bool=true) c %!s(*int64=0xc0002a4330) %!s(*int64=0xc0002a4338) rwm} {%!s(bool=true) c %!s(*int64=0xc0002a4340) %!s(*int64=0xc0002a4348) rwm} {%!s(bool=true) c %!s(*int64=0xc000196ed8) %!s(*int64=0xc000196ee0) rwm} {%!s(bool=true) c %!s(*int64=0xc0001971a8) %!s(*int64=0xc0001971b0) rwm} {%!s(bool=true) c %!s(*int64=0xc0001971d8) %!s(*int64=0xc0001971e0) rwm} {%!s(bool=true) c %!s(*int64=0xc0001972c8) %!s(*int64=0xc0001972d0) rwm} {%!s(bool=true) c %!s(*int64=0xc0001972f8","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=Cleanup function=Cleanup name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers/hypervisor subsystem=cloudHypervisor\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=info msg=\"Cleanup agent\" name=containerd-shim-v2 path=/run/kata-containers/shared/sandboxes/358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6/shared pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=virtcontainers subsystem=fs_share\n","facility":0,"hostname":"localhost","priority":6,"severity":6,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Create() end\" container=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=containerd-kata-shim-v2\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Delete() start\" container=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=containerd-kata-shim-v2\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Delete() end\" container=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=containerd-kata-shim-v2\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"State() end\" container=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=containerd-kata-shim-v2\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content": level=debug msg=\"Shutdown() start\" container=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 name=containerd-shim-v2 pid=71990 sandbox=358e7b4f79ddf12c19197634fd1b32b1ae081764dc8805347a8fedcab5efd5d6 source=containerd-kata-shim-v2\n","facility":0,"hostname":"localhost","priority":7,"severity":7,"tag":"kata",}
<obfuscated IP>: {"content":": info: guest-ping called\n","facility":1,"hostname":"localhost","priority":14,"severity":6,"tag":"unknown",}
<obfuscated IP>: {"content":": info: guest-fsfreeze called\n","facility":1,"hostname":"localhost","priority":14,"severity":6,"tag":"unknown",}

[jasek5]

I've resolved the issue. It turns out the problem was on my end. As a Proxmox user, enabling nested virtualization for VMs is essential, and I had that set up correctly. However, the issue arose because I had mistakenly attached the wrong CPU type to my Talos nodes. This mismatch prevented nested virtualization from working properly. Once I corrected this by aligning the CPU to the host type of my Talos VMs, the kata-containers started working without a hitch.

Sorry for the trouble.
Best regards!