`fchmodat2` seccomp filter breaks sandboxed builds with glibc 2.38

[max-privatevoid]

`fchmodat2` seccomp filter breaks sandboxed builds with glibc 2.38

[Ma27]

Turns out, it's not glibc, but libseccomp: libseccomp seems to maintain a syscall table internally and validates syscall names from seccomp_rule_add against that. In other words, linking nix against libseccomp 2.5.5 fixed the problem.
On 23.11 we have 2.5.4, so

• we should backport the upgrade to 2.5.5. Short of an update to the syscall table nothing changed.
• update the flake to use libseccomp 2.5.5 immediately (to unblock @max-privatevoid and @Gerg-L amonst others)
• reject libseccomp 2.5.4 or older in the build system: Nix must not be linked against that since builds on Linux will always fail.

I have a personal appointment now, but I'll take care of that later. Just wanted to share the good news already :)

[Ma27]

Oh no... sorry!
Will try to reproduce it today and come up with a fix!

[edolstra]

For reproducing, note that Nix's own VM tests are showing this issue: https://hydra.nixos.org/eval/1805828#tabs-still-fail, so e.g. nix build .#hydraJobs.tests.containers should show it.

[Gerg-L]

Didn't mean to be rude at all @Ma27 was just on mobile and didn't have much time this morning
good work on the PR's

[Ma27]

I've noticed you're using a Nix built within nixpkgs, while I'm using the flake output, which means your Nix is built with glibc 2.39 and mine (and presumably Gerg's) uses 2.38. So the bug is probably in the glibc 2.38 compatibility code.

Good point!
Will look into this later today.

[Ma27]

Yep, glibc 2.38 is the problem. I'm somewhat surprised by this, but I'll need to dig a little deeper into what seccomp does under the hood.

cc @nixos/nix-team I do intend to work on that (I hope to push something by today or tomorrow), does that work for you? I'd like to avoid duplicated efforts.

Yes, the commit is mentioned multiple times above

I'm aware that the commit was mentioned above. However it wasn't clear whether both client and daemon use it and if it's exactly the rev at fault or if it was cherry-picked to a different Nix checkout (yes, I do that sometimes to test stuff out). Also unclear if the flake was used or an override from nixpkgs (which was the culprit btw!).

While I don't think it was meant that way, your tone came off somewhat passive-aggressive. Especially considering that you could've also given the details I asked for to help me to reproduce this.

[Gerg-L]

Can reproduce with Linux 6.6.28 previous commit (731c389) works fine

[max-privatevoid]

I also get error: getting sandbox mount namespace: No such file or directory instead sometimes, but can't say whether that's related to this.

[cole-h]

FWIW, latest CI seems to agree as well: https://github.com/NixOS/nix/actions/runs/8773741325/job/24075042219 (installer-test on the latest commit as of time of writing, but also appears in runs on previous commits as well).

cc @Ma27

[Ma27]

So, my naïve attempt to reproduce it failed.

Steps I did to try to reproduce it

{
  vm = { pkgs, ... }: {
    virtualisation.writableStore = true;
    virtualisation.memorySize = 8192;
    virtualisation.diskSize = 12 * 1024;
    environment.etc."foo.nix".source = ./test3.nix;
    nix.package = pkgs.nixVersions.unstable;
    boot.kernelPackages = pkgs.linuxKernel.packages.linux_6_6;
  };
}

With test3.nix being

with import (builtins.fetchTarball https://github.com/nixos/nixpkgs/archive/nixos-unstable.tar.gz) {};
let
  builder = runCommand "fnord" { } ''
    ${pkgs.gcc}/bin/gcc ${writeText "foo.c" ''
      #include <stdio.h>
      #include <stdlib.h>
      #include <sys/stat.h>
      #include <sys/syscall.h>
      #include <errno.h>

      int main(void) {
          char *name = getenv("OUT_FROM_ENV");
          FILE *fd = fopen(name, "w");
          fprintf(fd, "hello");
          fclose(fd);
          long rs = syscall(__NR_fchmodat2, NULL, name, S_ISUID, 0);
          printf("result: %ld, errno: %ld\n", rs, errno);
      }
    ''} -O0 -g -o $out
  '';
in
runCommand "foobar" { } ''
  OUT_FROM_ENV=$out ${builder}
''

(used it to test the change originally).

With the following change on nixpkgs master (db176ef1d6545cc4f3c0f930661e83ff5d0bcd76):

diff --git a/pkgs/tools/package-management/nix/default.nix b/pkgs/tools/package-management/nix/default.nix
index 61053e491dbb..22abd93a96f6 100644
--- a/pkgs/tools/package-management/nix/default.nix
+++ b/pkgs/tools/package-management/nix/default.nix
@@ -262,6 +262,17 @@ in lib.makeExtensible (self: ({
     hash = "sha256-ObaVDDPtnOeIE0t7m4OVk5G+OS6d9qYh+ktK67Fe/zE=";
   };
 
+  unstable = common rec {
+    version = "2.22.0";
+    suffix = "pre20240321_${lib.substring 0 8 src.rev}";
+    src = fetchFromGitHub {
+      owner = "NixOS";
+      repo = "nix";
+      rev = "6fd2f42c2defd210e17ec95653110fc58858dba9";
+      hash = "sha256-DjkxYMcG52APiADdEtXL1FNVSxNXRBw78LYctly93j0=";
+    };
+  };
+
   # The minimum Nix version supported by Nixpkgs
   # Note that some functionality *might* have been backported into this Nix version,
   # making this package an inaccurate representation of what features are available
@@ -280,8 +291,6 @@ in lib.makeExtensible (self: ({
       nix;
 
   stable = addFallbackPathsCheck self.nix_2_18;
-
-  unstable = self.nix_2_21;
 } // lib.optionalAttrs config.allowAliases {
   nix_2_4 = throw "nixVersions.nix_2_4 has been removed";
 

VM is built via NIX_PATH=nixpkgs=$(pwd) nixos-build-vms vm.nix.
Running nix-build /etc/foo.nix inside doesn't trigger the issue you're seeing.

Kernel is currently 6.6.28, but also checked it with latest 6.1.

Now I have a few questions (most likely not all of them are relevant, but it may help to pinpoint the issue a little better):

• Which OS did you use (if NixOS, which nixpkgs revision precisely)?
• Which architecture?
• Kernel 6.6.28, correct?
• Do you use Nix master (which rev btw?) for both client and daemon?
• Are remote builds involved?
• Can you reproduce the issue with the build I added above? If not, can you share a build that breaks (perhaps a flake URL)?

[Gerg-L]

• Do you use Nix master (which rev btw?) for both client and daemon?

Yes, the commit is mentioned multiple times above

[max-privatevoid]

I reworked your VM test a bit.

{
  vm = { pkgs, ... }: {
    virtualisation.writableStore = true;
    virtualisation.memorySize = 8192;
    virtualisation.diskSize = 12 * 1024;
    environment.etc."foo.nix".text = ''
      derivation {
        name = "test";
        system = builtins.currentSystem;
        builder = "/bin/sh";
        args = [ "-c" "echo ''${toString builtins.currentTime} > $out" ];
      }
    '';
    nix.package = (builtins.getFlake "github:NixOS/nix/ba6804518772e6afb403dd55478365d4b863c854").packages.${pkgs.system}.nix;
    boot.kernelPackages = pkgs.linuxKernel.packages.linux_6_6;
  };
}

Built with nixos-build-vms --option nix-path nixpkgs=flake:github:nixos/nixpkgs/5c24cf2f0a12ad855f444c30b2421d044120c66f vm.nix. The bug also happens on Linux 6.8, so this behavior probably doesn't depend on the kernel version.

I've noticed you're using a Nix built within nixpkgs, while I'm using the flake output, which means your Nix is built with glibc 2.39 and mine (and presumably Gerg's) uses 2.38. So the bug is probably in the glibc 2.38 compatibility code.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-04-22-nix-team-meeting-minutes-140/44016/1