"sh: write error: Invalid argument" errors when register

[multiarch]

"sh: write error: Invalid argument" errors when register

[junaruga]

https://travis-ci.com/hoshsadiq/qemu-testing/builds/124639278

@hoshsadiq in your Travis log, below command outputs error message then finishes with exit status: 0. that is what we have to fix.

$ docker run --rm --privileged multiarch/qemu-user-static:$QUS_VERSION --reset -p yes
Status: Downloaded newer image for multiarch/qemu-user-static:v4.0.0-4
sh: write error: Invalid argument
...

That means docker run --rm --privileged multiarch/qemu-user-static:$QUS_VERSION --reset -p yes failed to create /proc/sys/fs/binfmt_misc/qemu-* files.
I do not know why for now. To know it, we need to debug /qemu-binfmt-conf.sh injecting set -x in it.

But in your case I found why the issue happened.

Here is my repository forked from your qemu testing repository.
https://github.com/junaruga/qemu-testing/tree/feature/debug
https://travis-ci.org/junaruga/qemu-testing/builds/577460704

diff --git a/.travis.yml b/.travis.yml
index 8a2fc7a..b31a117 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,4 +1,4 @@
-dist: trusty
+dist: xenial
 # sudo: required
 language: minimal

After changing dist from trusty to xenial, you can run it without errors.

[lqhandsome]

must uname -r > 4.8

[junaruga]

@heyi-arm hi the behavior you reported is expected behavior.

The error message sh: write error: Invalid argument is not understandable.
But the actual meaning is when "docker run --rm --privileged multiarch/qemu-user-static:register" try to add "binfmt_misc" entry files /proc/sys/fs/binfmt_misc/qemu-$arch, those files already exist.
/proc/sys/fs/binfmt_misc/qemu-$arch files are special files . [1]
When updating the file, we have to delete the file at once.

I am not sure which Linux distribution you are using.
When installing RPM qemu-user-staticon CentOS or Fedora, it adds the /proc/sys/fs/binfmt_misc/qemu-$arch files as a post script after yum install qemu-user-static.
There is deb package qemu-user-static package on Debian or Ubuntu.
It is possible those files already exist when you run "docker run --rm --privileged multiarch/qemu-user-static:register".

Here is the way to change flags: to flags: F for /proc/sys/fs/binfmt_misc/qemu-aarch64.

# prompt is executed by root user. sudo does not work for the operation.
But if you try this operation, that is by your own responsibility.

If the environment is really clean without binfmt_misc files. Maybe condition is like this.

$ ls /proc/sys/fs/binfmt_misc
register  status

# echo -1 > /proc/sys/fs/binfmt_misc/qemu-aarch64

$ ls /proc/sys/fs/binfmt_misc/qemu-aarch64
ls: cannot access '/proc/sys/fs/binfmt_misc/qemu-aarch64': No such file or directory

# echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:" > /proc/sys/fs/binfmt_misc/register

$ ls /proc/sys/fs/binfmt_misc/qemu-aarch64
/proc/sys/fs/binfmt_misc/qemu-aarch64

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags:  
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

# echo -1 > /proc/sys/fs/binfmt_misc/qemu-aarch64

$ ls /proc/sys/fs/binfmt_misc/qemu-aarch64
ls: cannot access '/proc/sys/fs/binfmt_misc/qemu-aarch64': No such file or directory

# echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:F" > /proc/sys/fs/binfmt_misc/register

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /qus/bin/qemu-aarch64-static
flags: F
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

I would close this ticket, as it is expected behavior.
But you like to reopen, you can do it.
Thanks for reporting!

By the way today we release a new image "multiarch/qemu-user-static" that can build and run standard architecture specific container instead of multiarch compatible imagtes.

Like this.

$ uname -m
x86_64

$ docker run --rm -t arm64v8/ubuntu uname -m
standard_init_linux.go:211: exec user process caused "exec format error"

$ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

$ docker run --rm -t arm64v8/ubuntu uname -m
aarch64

• [1] https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html

[junaruga]

The register image is using below QEMU's scripts/qemu-binfmt-conf.sh script internally.
So, if the message is not understandable, you might be able to report to QEMU project.

Below is the logic to add binfmt_misc entry.

https://github.com/qemu/qemu/blob/master/scripts/qemu-binfmt-conf.sh#L269-L270

qemu_register_interpreter() {
    echo "Setting $qemu as binfmt interpreter for $cpu"
    qemu_generate_register > /proc/sys/fs/binfmt_misc/register
}

[hoshsadiq]

@junaruga I'm not sure I follow all of what you said. Could you please expand a little bit on it?

My build output can be found here: https://travis-ci.com/hoshsadiq/qemu-testing/builds/124639278

As you can the /proc/sys/fs/binfmt_misc/ is empty and doesn't have the files as suggested in your comment (unless I misunderstood).

[junaruga]

This repository's .travis.yml might be also useful.
https://github.com/junaruga/ci-multi-arch-test

[hoshsadiq]

Ah. I was being stupid. I was meant to be using bionic. Seems this is supported on Xenial and above. Thank you!

[lugeng]

echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:F" > /proc/sys/fs/binfmt_misc/register
bash: line 0: echo: write error: Invalid argument

ls /proc/sys/fs/binfmt_misc/qemu-aarch64
ls: cannot access /proc/sys/fs/binfmt_misc/qemu-aarch64: No such file or directory

But
echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:" > /proc/sys/fs/binfmt_misc/register

ls /proc/sys/fs/binfmt_misc/qemu-aarch64
/proc/sys/fs/binfmt_misc/qemu-aarch64 works

cannot set flag with F

[yuanzhedong]

@lugeng I got the same error, did you find a solution? Thanks!

[junaruga]

The F flag is a relatively new feature of the kernel binfmt_misc. Maybe it does not work if the host os kernel version (= uname -r) < 4. See #100 (comment) for the solution.

[Deepak275]

facing the same issue, kernel is 4.4, OS: Ubuntu 18.04.4 LTS. Tried running both, with and without -p yes, but no progress. Any help is appreciated ?

[gasgithub]

I have similar issue in docker.io/library/docker:19.03.14 container (Linux reynwz-containerize-pod 3.10.0-1160.62.1.el7.x86_64 #1 SMP Wed Mar 23 09:04:02 UTC 2022 x86_64 Linux) running the following but getting errors:

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

Status: Downloaded newer image for multiarch/qemu-user-static:latest 
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha sh: write error: Invalid argument
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm sh: write error: Invalid argument
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc-static as binfmt interpreter for sparc sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc64-static as binfmt interpreter for sparc64 sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64 sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le sh: write error: Invalid argument
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k sh: write error: Invalid argument
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32 sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el sh: write error: Invalid argument
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64 sh: write error: Invalid argument
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4 sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb sh: write error: Invalid argument
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x sh: write error: Invalid argument
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64 sh: write error: Invalid argument
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be sh: write error: Invalid argument
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa sh: write error: Invalid argument
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32 sh: write error: Invalid argument
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64 sh: write error: Invalid argument
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa sh: write error: Invalid argument
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb sh: write error: Invalid argument
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze sh: write error: Invalid argument
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel sh: write error: Invalid argument
Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k sh: write error: Invalid argument
Setting /usr/bin/qemu-hexagon-static as binfmt interpreter for hexagon sh: write error: Invalid argument

Any hints whats the problem and what can I do to fix this issue?

My environment is:

+ docker version
Client: Docker Engine - Community
 Version:           19.03.14
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        5eb3275
 Built:             Tue Dec  1 19:14:24 2020
 OS/Arch:           linux/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.15
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       99e3ed8
  Built:            Sat Jan 30 03:18:13 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.9
  GitCommit:        ea765aba0d05254012b0b9e595e995c09186427f
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[orgads]

@gasgithub Your kernel is probably too old. Try to upgrade it.

[Deepak275]

must uname -r > 4.8
@lqhandsome Yeah, that was the issue, thanks. :)