Running Singularity in a user namespace created by an unprivileged user doesn't work

[stxue1]

Running Singularity in a user namespace created by an unprivileged user doesn't work

[dtrudg]

Please could you give a bit more context:

• Are the failing commands run inside a Docker container, or are they run on the host at this point? If in a Docker container, do they work instead on the host?
• What is the overall goal here... i.e. why do you need to use Singularity with FUSE mounts insides an unprivileged Docker container? Can you explain why singularity must be run inside an unprivileged Docker container, and what FUSE mounts are needed... do you mean Singularity's FUSE mounts or external FUSE mounts?

[dtrudg]

Did some quick digging, and may be able to solve this without additional context.

For the unshare -r case the following patch should solve the issue, and will be included in the next release:

diff --git a/internal/pkg/ociimage/unpack.go b/internal/pkg/ociimage/unpack.go
index fdbc132d7..55cbd0736 100644
--- a/internal/pkg/ociimage/unpack.go
+++ b/internal/pkg/ociimage/unpack.go
@@ -16,7 +16,9 @@ import (
        umocilayer "github.com/opencontainers/umoci/oci/layer"
        "github.com/opencontainers/umoci/pkg/idtools"
        "github.com/sylabs/singularity/v4/internal/pkg/util/fs"
+       "github.com/sylabs/singularity/v4/pkg/util/namespaces"
        "github.com/sylabs/singularity/v4/pkg/sylog"
+
 )
 
 // isExtractable checks if we have extractable layers in the image. Shouldn't be
@@ -69,7 +71,8 @@ func UnpackRootfs(_ context.Context, srcImage v1.Image, destDir string) (err err
        }
 
        // Allow unpacking as non-root
-       if os.Geteuid() != 0 {
+       insideUserNs, _ := namespaces.IsInsideUserNamespace(os.Getpid())
+       if os.Geteuid() != 0 || insideUserNs {
                mapOptions.Rootless = true
 
                uidMap, err := idtools.ParseMapping(fmt.Sprintf("0:%d:1", os.Geteuid()))

[dtrudg]

The unshare -c case is a separate issue.

With unshare -c you will need to use the -u flag for singularity - this is required to tell a setuid installation of singularity to run in usernamespace mode, instead of setuid mode.

unshare -c singularity run -u alpine_latest.sif

At present this is not working, as a wrapped extraction of an image is failing to honour the -u flag. I'll track this as a separate issue.

You can workaround this issue, at present, via creating a sandbox dir and then running with -u, e.g.:

$ singularity build -s test-sandbox docker://ubuntu:latest

$ unshare -c singularity run -u test-sandbox/
Singularity> echo "HELLO"
HELLO
Singularity> 

[stxue1]

Hi, thanks for the quick response!

* Are the failing commands run inside a Docker container, or are they run on the host at this point? If in a Docker container, do they work instead on the host?

I first encountered this issue when running the commands inside a Docker container, but was able to reproduce the exact same issue on a host machine. Neither the container or the host work, but the host seemed more easily reproducible so I included that as the bug report instead.

* What is the overall goal here... i.e. why do you need to use Singularity with FUSE mounts insides an unprivileged Docker container? Can you explain why singularity must be run inside an unprivileged Docker container, and what FUSE mounts are needed... do you mean Singularity's FUSE mounts or external FUSE mounts?

I'm trying to create a Docker container with an application preinstalled. The app is a workflow engine and uses Singularity as one of it's container backends to run tasks. I noticed that sandboxing images could take up a small amount of time for each task, so I wanted to see how much of a difference having Singularity mount with FUSE would be. In theory the container could be set up as privileged, which would allow it to work, but it would be better if that weren't the case.

I applied the patch and unshare -r and unshare -c work properly on my host machine and within the Docker container. Making Singularity use FUSE with unshare -c --keep-caps -m singularity -v -d run -u docker://hello-world does return an error on cleanup:

DEBUG   [U=1000,P=508312]  UnmountWithFuseLazy()         Executing FUSE unmount command: /usr/bin/fusermount3 -z -u /tmp/rootfs-3641030937/root
/usr/bin/fusermount3: failed to unmount /tmp/rootfs-3641030937/root: Invalid argument
ERROR   [U=1000,P=508312]  CleanupHost()                 While running host cleanup tasks: while unmounting fuse directory: /tmp/rootfs-3641030937/root: exit status 1
DEBUG   [U=1000,P=508312]  CleanupHost()                 Exiting CleanupHost - Cleanup failure
ERROR   [U=1000,P=508199]  Master()                      Unprivileged host cleanup failed: host cleanup failed

My guess is because Singularity tries to cleanup twice as the same unmount command appears above:

DEBUG   [U=1000,P=508311]  UnmountWithFuseLazy()         Executing FUSE unmount command: /usr/bin/fusermount3 -z -u /tmp/rootfs-3641030937/root

Log

This is probably a separate issue, but it doesn't break Singularity's functionality since the /tmp/rootfs-* folder is properly cleaned up. But everything looks to be working properly now, thanks!

[dtrudg]

Thanks for the feedback. The fusermount error is indeed something that will need to be addressed differently / separately. I'll open a separate issue for that, and we can tackle it a bit later on.