WSL2 crashes at launch after enabling distrod `[Distrod][ERROR] Failed to spawn the command.`

[nullpo-head]

WSL2 crashes at launch after enabling distrod `[Distrod][ERROR] Failed to spawn the command.`

[nullpo-head]

Hmm.. It looks systemd crashed for some reason. Possibly due to some Genie's initialization script which wasn't completely removed, but I'm not sure, I'm not so familiar with genie. Or, it could be just a bug of Distrod.

Could you enable the logging of both log_level and kmsg_log_level of Distrod following the doc please? After that, instead of enabling Distrod, run it as a one-shot command by the following commands.

/opt/distrod/bin/distrod-exec /bin/bash bash

It should crash in the same way as before, but with debug logging this time.
Could you share the log of /dev/kmsg then, please? This time I'd like to see the systemd's log as well, so please don't filter the log with Distrod:

sudo cat /dev/kmsg > kmsg_log.txt  # this doesn't return. Press ctrl + c after a few seconds

[eizanprime]

Thank you for the reply ! I have gathered the logs, looks like some mount problem (???) I don't really understand though.. Thank you very much for your help !

[Distrod][DEBUG] starting /init from distrod-exec
[Distrod][DEBUG] WSL envs: "WSLENV" = "WT_SESSION::WT_PROFILE_ID"
[Distrod][DEBUG] WSL envs: "WSL_DISTRO_NAME" = "Archlinux"
[Distrod][DEBUG] WSL envs: "WSL_INTEROP" = "/run/WSL/8_interop"
[Distrod][DEBUG] Container::with_mount source: Some(HostPath("/run/distrod/cmdline")), target: ContainerPath("/proc/cmdline"), fstype: None, flags: MS_BIND, is_file: true
[Distrod][TRACE] mount_distrod_run_files: path: "/opt/distrod/run/systemd"
[Distrod][TRACE] mount_distrod_run_files: path: "/opt/distrod/run/systemd/system"
[Distrod][TRACE] mount_distrod_run_files: path: "/opt/distrod/run/systemd/system/portproxy.service"
[Distrod][DEBUG] Container::with_mount source: Some(HostPath("/opt/distrod/run/systemd/system/portproxy.service")), target: ContainerPath("/run/systemd/system/portproxy.service"), fstype: None, flags: MS_BIND, is_file: true
[Distrod][TRACE] mount_distrod_run_files: path: "/opt/distrod/run/tmpfiles.d"
[Distrod][TRACE] mount_distrod_run_files: path: "/opt/distrod/run/tmpfiles.d/x11.conf"
[Distrod][DEBUG] Container::with_mount source: Some(HostPath("/opt/distrod/run/tmpfiles.d/x11.conf")), target: ContainerPath("/run/tmpfiles.d/x11.conf"), fstype: None, flags: MS_BIND, is_file: true
[Distrod][DEBUG] DistroLauncher::launch
[Distrod][DEBUG] Container::with_mount source: Some(HostPath("/run/distrod/distrod_wsl_env-uid0")), target: ContainerPath("/run/distrod/distrod_wsl_env-uid0"), fstype: None, flags: MS_BIND, is_file: true
[Distrod][DEBUG] Spawning the command or the waiter.
[Distrod][DEBUG] Executing a command in the distro.
[Distrod][TRACE] mounting source: Some(
    ContainerPath(
        "/run/distrod/cmdline",
    ),
), mount: ContainerMount { source: Some(HostPath("/run/distrod/cmdline")), target: ContainerPath("/proc/cmdline"), fstype: None, flags: MS_BIND, data: None, is_file: true }
[Distrod][DEBUG] Failed to ignore signal Sys(EINVAL)
[Distrod][TRACE] mounting source: Some(
    ContainerPath(
        "/opt/distrod/run/systemd/system/portproxy.service",
    ),
), mount: ContainerMount { source: Some(HostPath("/opt/distrod/run/systemd/system/portproxy.service")), target: ContainerPath("/run/systemd/system/portproxy.service"), fstype: None, flags: MS_BIND, data: None, is_file: true }
[Distrod][DEBUG] Failed to ignore signal Sys(EINVAL)
[Distrod][DEBUG] Distro::exec_command.
[Distrod][DEBUG] Container::exec_command.
[Distrod][TRACE] mounting source: Some(
    ContainerPath(
        "/opt/distrod/run/tmpfiles.d/x11.conf",
    ),
), mount: ContainerMount { source: Some(HostPath("/opt/distrod/run/tmpfiles.d/x11.conf")), target: ContainerPath("/run/tmpfiles.d/x11.conf"), fstype: None, flags: MS_BIND, data: None, is_file: true }
[Distrod][TRACE] skipping an identical mount: Some(
    ContainerPath(
        "/run/distrod/distrod_wsl_env-uid0",
    ),
), ContainerMount {
    source: Some(
        HostPath(
            "/run/distrod/distrod_wsl_env-uid0",
        ),
    ),
    target: ContainerPath(
        "/run/distrod/distrod_wsl_env-uid0",
    ),
    fstype: None,
    flags: MS_BIND,
    data: None,
    is_file: true,
}
[Distrod][DEBUG] Triple fork done.
[Distrod][ERROR] Pre_second_fork failed.

Caused by:
    0: Failed to enter the init's namespace
    1: Failed to open ns/mnt
    2: ENOENT: No such file or directory
[Distrod][ERROR] Failed to spawn the command.

Caused by:
    No such file or directory (os error 2)
[Distrod][DEBUG] The pipe for wait has been closed. Possibly the proxy process has been killed by SIGKILL.

9,333,23059002104,-;Distrod: [DEBUG] starting /init from distrod-exec
9,334,23059002432,-;Distrod: [DEBUG] WSL envs: "WSLENV" = "WT_SESSION::WT_PROFILE_ID"
9,335,23059002443,-;Distrod: [DEBUG] WSL envs: "WSL_DISTRO_NAME" = "Archlinux"
9,336,23059002464,-;Distrod: [DEBUG] WSL envs: "WSL_INTEROP" = "/run/WSL/8_interop"
9,337,23059002752,-;Distrod: [DEBUG] Container::with_mount source: Some(HostPath("/run/distrod/cmdline")), target: ContainerPath("/proc/cmdline"), fstype: None, flags: MS_BIND, is_file: true
9,338,23059002882,-;Distrod: [TRACE] mount_distrod_run_files: path: "/opt/distrod/run/systemd"
9,339,23059002919,-;Distrod: [TRACE] mount_distrod_run_files: path: "/opt/distrod/run/systemd/system"
9,340,23059002951,-;Distrod: [TRACE] mount_distrod_run_files: path: "/opt/distrod/run/systemd/system/portproxy.service"

9,341,23059002969,-;Distrod: [DEBUG] Container::with_mount source: Some(HostPath("/opt/distrod/run/systemd/system/portproxy.service")), target: ContainerPath("/run/systemd/system/portproxy.service"), fstype: None, flags: MS_BIND, is_file: true
9,342,23059003003,-;Distrod: [TRACE] mount_distrod_run_files: path: "/opt/distrod/run/tmpfiles.d"
9,343,23059003031,-;Distrod: [TRACE] mount_distrod_run_files: path: "/opt/distrod/run/tmpfiles.d/x11.conf"
9,344,23059003060,-;Distrod: [DEBUG] Container::with_mount source: Some(HostPath("/opt/distrod/run/tmpfiles.d/x11.conf")), target: ContainerPath("/run/tmpfiles.d/x11.conf"), fstype: None, flags: MS_BIND, is_file: true
9,345,23059003103,-;Distrod: [DEBUG] DistroLauncher::launch
9,346,23059003184,-;Distrod: [DEBUG] Container::with_mount source: Some(HostPath("/run/distrod/distrod_wsl_env-uid0")), target: ContainerPath("/run/distrod/distrod_wsl_env-uid0"), fstype: None, flags: MS_BIND, is_file: true
9,347,23059003775,-;Distrod: [DEBUG] Spawning the command or the waiter.
9,348,23059004154,-;Distrod: [DEBUG] Executing a command in the distro.
9,349,23059004161,-;Distrod: [TRACE] mounting source: Some(\x0a    ContainerPath(\x0a        "/run/distrod/cmdline",\x0a    ),\x0a), mount: ContainerMount { source: Some(HostPath("/run/distrod/cmdline")), target: ContainerPath("/proc/cmdline"), fstype: None, flags: MS_BIND, data: None, is_file: true }
9,350,23059004190,-;Distrod: [DEBUG] Failed to ignore signal Sys(EINVAL)
9,351,23059004224,-;Distrod: [DEBUG] Failed to ignore signal Sys(EINVAL)
9,352,23059004232,-;Distrod: [DEBUG] Distro::exec_command.
9,353,23059004239,-;Distrod: [TRACE] mounting source: Some(\x0a    ContainerPath(\x0a        "/opt/distrod/run/systemd/system/portproxy.service",\x0a    ),\x0a), mount: ContainerMount { source: Some(HostPath("/opt/distrod/run/systemd/system/portproxy.service")), target: ContainerPath("/run/systemd/system/portproxy.service"), fstype: None, flags: MS_BIND, data: None, is_file: true }
9,354,23059004261,-;Distrod: [DEBUG] Container::exec_command.
9,355,23059004278,-;Distrod: [TRACE] mounting source: Some(\x0a    ContainerPath(\x0a        "/opt/distrod/run/tmpfiles.d/x11.conf",\x0a    ),\x0a), mount: ContainerMount { source: Some(HostPath("/opt/distrod/run/tmpfiles.d/x11.conf")), target: ContainerPath("/run/tmpfiles.d/x11.conf"), fstype: None, flags: MS_BIND, data: None, is_file: true }
9,356,23059004320,-;Distrod: [TRACE] skipping an identical mount: Some(\x0a    ContainerPath(\x0a        "/run/distrod/distrod_wsl_env-uid0",\x0a    ),\x0a), ContainerMount {\x0a    source: Some(\x0a        HostPath(\x0a            "/run/distrod/distrod_wsl_env-uid0",\x0a        ),\x0a    ),\x0a    target: ContainerPath(\x0a        "/run/distrod/distrod_wsl_env-uid0",\x0a    ),\x0a    fstype: None,\x0a    flags: MS_BIND,\x0a    data: None,\x0a    is_file: true,\x0a}
9,357,23059004367,-;Distrod: [DEBUG] Triple fork done.
9,358,23059004565,-;Distrod: [ERROR] Pre_second_fork failed.\x0a\x0aCaused by:\x0a    0: Failed to enter the init's namespace\x0a    1: Failed to open ns/mnt\x0a    2: ENOENT: No such file or directory
9,359,23059004610,-;Distrod: [DEBUG] The pipe for wait has been closed. Possibly the proxy process has been killed by SIGKILL.
9,360,23059004613,-;Distrod: [ERROR] Failed to spawn the command.\x0a\x0aCaused by:\x0a    No such file or directory (os error 2)
12,361,23118825225,-;WSL2: Performing memory compaction.

[nullpo-head]

Hi, I cannot find anything wrong in your log. I will check if I can reproduce the problem by installing / uninstalling Genie later.

[eizanprime]

Thank you checking, do you know where it failed so I can try to work around it ?

[nullpo-head]

There are two possibilities as far as I can guess.

1. Systemd is crashing immediately after it is launched.
   • In this case, I suspect Genie's some systemd generator script is causing crash, because it should assume that Genie is running when it detects systemd running.
2. Systemd doesn't crash, but the file /proc/[pid(e.g: 1)]/ns/mnt actually doesn't exist for some reason.
   • In this case, we must investigate the root cause.

In both cases, there is no workaround. What I can suggest as a workaround is to install a clean new distro by Distrod.

[nullpo-head]

To check whether Systemd crashes or not, could you share the result of ps aux after Distrod crashes, by wsl -e ps aux command? The name of the systemd process is like /sbin/init ....setenv=...
it systemd exists, could you share the output of wsl -e ls /proc/[pid of the systemd you found]/ns as well, please?

[oDn]

I seem to be having the exact same problem running the fedora 35 image from linuxcontainers with distrod.

Here's the output of wsl -e ls /proc/[pid of the systemd you found]/ns:

PS C:\Users\user> wsl -d Distrod -e ps faux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   8944   332 ?        Ssl  15:38   0:00 /init
root         7  0.0  0.0   8944   228 tty1     Ss   15:38   0:00 /init
user+     8  0.0  0.0 233576  1828 tty1     R    15:38   0:00  \_ ps faux
PS C:\Users\user> wsl -d Distrod -e sudo ls /proc/1/ns
ipc  mnt  net  pid  user  uts
PS C:\Users\user> wsl -d Distrod -e sudo ls /proc/7/ns
ls: cannot access '/proc/7/ns': No such file or directory

There doesn't seem to be any parameters passed to /init though.

[nullpo-head]

@oDn thanks for sharing the outputs! Could you also try wsl -e uname -a, please? I suspect your Linux kernel is too old, or running WSL1, and thus Systemd crashed.

What I can tell from your outputs.

1. /init is not systemd, but WSL's init process, and there's no /sbin/init. That means Systemd crashed.

2. > wsl -d Distrod -e sudo ls /proc/1/ns
   ipc  mnt  net  pid  user  uts
   

   You can see pid_for_children is missing. This indicates that your WSL kernel is older than 4.12, or your running WSL1.

[oDn]

@nullpo-head, I think you're spot on !

PS C:\Users\user> wsl --list --verbose
  NAME                   STATE           VERSION
* Ubuntu                 Running         2
  distrod-fedora         Stopped         1

I was sure I'd set WSL's default version to 2 a while ago but that obviously wasn't the case.

PS C:\Users\user> wsl --set-default-version 2
For information on key differences with WSL 2 please visit https://aka.ms/wsl2
The operation completed successfully.
PS C:\Users\user> wsl --unregister distrod-fedora
Unregistering...

Now that I've created a new VM using WSL 2, everything seems to be running fine !

PS C:\Users\user> wsl --list --verbose
  NAME                   STATE           VERSION
* Ubuntu                 Running         2
  distrod-fedora         Running         2

[user@hostname distrod_wsl_launcher-x86_64]$ systemctl status
● hostname
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Mon 2021-11-29 16:18:33 CET; 31s ago
   CGroup: /
           ├─init.scope
           │ └─1 /sbin/init systemd.setenv=WSLENV=WT_SESSION::WT_PROFILE_ID systemd.setenv=WSL_INTEROP=/run/WSL/25_interop systemd.setenv=WSL_DISTRO_NAME=distrod-fedora --unit=multi-user.target
           └─system.slice
             ├─systemd-udevd.service
             │ └─66 /usr/lib/systemd/systemd-udevd
             ├─dbus-broker.service
             │ ├─78 /usr/bin/dbus-broker-launch --scope system --audit
             │ └─91 dbus-broker --log 4 --controller 9 --machine-id f79933900b0e4a7bb4ec85e6727bf35a --max-bytes 536870912 --max-fds 4096 --max-matches 16384 --audit
             ├─systemd-homed.service
             │ └─75 /usr/lib/systemd/systemd-homed
             ├─systemd-journald.service
             │ └─54 /usr/lib/systemd/systemd-journald
             └─systemd-logind.service
               └─76 /usr/lib/systemd/systemd-logind

Thanks !

[nullpo-head]

By the way, as for the original issue reporter, @eizanprime , it's unlikely the cause is WSL1, since they used genie once, which doesn't run on WSL1. I'm still waiting for their comments.

[eizanprime]

I am finally back, sorry for being late (was a bit busy these days, I'm actually work in tokyo too)

here is the output of the ps command, the process doesn't look there..

root         1  1.3  0.0    896   528 ?        Sl   18:20   0:00 /init
root        16  0.0  0.0    896    84 ?        Ss   18:20   0:00 /init
root        17  0.0  0.0    896    84 ?        R    18:20   0:00 /init
eizanpr+    18  0.0  0.0  10292  3396 pts/0    Rs+  18:20   0:00 ps aux

I would recreate a new distro if I could but my current system is quite customized and I don't really want to restart from scratch (as everything is vmdk in wsl2 I suppose a good old dd on the disk would not work so well)

[skyblaster]

In attempting to bring up a Fedora 36 or Rawhide container without first upgrading the 35 base from linuxcontainers.org (which BTW puts systemd into a degraded state), I'm seeing the same issue with images from: https://koji.fedoraproject.org/koji/packageinfo?packageID=26387

After extracting layer.tar from the tar.xz and performing the following steps:

> wsl --import rawhide C:\Temp\wslTemp C:\Users\yourusername\Downloads\fedora-rawhide.tar

> wsl -d rawhide

$ dnf update -y && dnf install passwd sudo -y
$ adduser -G wheel yourusername
$ echo -e "[user]\ndefault=yourusername" >> /etc/wsl.conf
$ passwd yourusername
$ exit

> wsl -t rawhide
> wsl -d rawhide

$ curl -L -O "https://raw.githubusercontent.com/nullpo-head/wsl-distrod/main/install.sh"
$ chmod +x install.sh
$ sudo ./install.sh install

I get symlink errors after enabling distrod:

$ sudo /opt/distrod/bin/distrod enable
[Distrod][WARN] Faled to mask systemd-remount-fs.service. Error: Failed to symlink '"/etc/systemd/system/systemd-remount-fs.service"'.

Caused by:
    No such file or directory (os error 2)
[Distrod][WARN] Faled to mask systemd-modules-load.service. Error: Failed to symlink '"/etc/systemd/system/systemd-modules-load.service"'.

Caused by:
    No such file or directory (os error 2)
[Distrod][WARN] Faled to mask [email protected]. Error: Failed to symlink '"/etc/systemd/system/[email protected]"'.

Caused by:
    No such file or directory (os error 2)
[Distrod][WARN] Faled to mask [email protected]. Error: Failed to symlink '"/etc/systemd/system/[email protected]"'.

Caused by:
    No such file or directory (os error 2)
[Distrod][WARN] Faled to mask console-getty.service. Error: Failed to symlink '"/etc/systemd/system/console-getty.service"'.

Caused by:
    No such file or directory (os error 2)
[Distrod] Distrod has been enabled. Now your shell will start under systemd.

Here are the errors from subsequent launch 1:

> wsl -d rawhide
[Distrod][ERROR] Failed to spawn the command.

Caused by:
    No such file or directory (os error 2)

And subsequent launch 2:

> wsl -d rawhide
[Distrod][ERROR] Failed to spawn the command.

Caused by:
    No such file or directory (os error 2)
[Distrod][ERROR] Pre_second_fork failed.

Caused by:
    0: Failed to enter the init's namespace
    1: Failed to open ns/uts
    2: ESRCH: No such process

This is from a clean Windows 11 Pro install, so wsl2 is implied and confirmed.

Microsoft Windows [Version 10.0.22000.527]

> wsl -e uname -a
Linux 8600K 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[machsix]

Any progress for this issue? I'm having similar problem. The distro I use is from this https://github.com/machsix/ubuntu-impi. You can download the rootfs here https://github.com/machsix/ubuntu-impi/actions/runs/2157438149

[Mw3y]

This error just started to appear on my laptop running windows 11. I haven't changed anything: it just doesn't launch anymore.
code 137 (0x00000089)