What is the purpose of /usr/bin in the multiarch/qemu-user-static:... docker images

[multiarch]

What is the purpose of /usr/bin in the multiarch/qemu-user-static:... docker images

[rmoriz]

I think the readme should cover the -p yes option of:

https://github.com/qemu/qemu/blob/659b11e7a7239529cfdb4968418268ff9aa22d88/scripts/qemu-binfmt-conf.sh#L261-L263

which translates to flag F:
https://www.kernel.org/doc/html/v4.18/admin-guide/binfmt-misc.html

The Kernel now uses the binary of the host file system and there is no need to copy the qemu-static binary inside the image.

[junaruga]

Now the pull-request was merged.

The new image multiarch/qemu-user-static works. :)

$ sudo docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
$ docker pull arm64v8/ubuntu
$ docker run --rm -t arm64v8/ubuntu uname -m
aarch64

$ sudo podman run --rm --privileged multiarch/qemu-user-static --reset -p yes
$ podman pull arm64v8/fedora
$ podman run --rm -t arm64v8/fedora uname -m
aarch64

Current status

• Check/ensure that ADD does extract the tarball and not only copy it ✔️
• Add all the binaries in multiarch/qemu-user-static:* images to image multiarch/qemu-user-static:register. ✔️
• Deprecate the concept of 'Compatible images' and remove the section from the README. : not yet
• Deprecate the 'Examples & articles' section of the README and provide some up-to-date article that enforces using -p. : not yet

[eine]

Just dropping my two cents here:

If ADD is used to copy a local tarball, it is automatically extracted. See docs.docker.com/engine/reference/builder/#add:

If is a local tar archive in a recognized compression format (identity, gzip, bzip2 or xz) then it is unpacked as a directory. Resources from remote URLs are not decompressed.

However, this has not always being like that. See stackoverflow.com/questions/45987019/dockerfile-add-tar-gz-does-not-extract-on-ubuntu-vm-with-docker.

Preciselly, build 52 was the last one with docker version 17.03 and version 17.09 has been used since then.

So, I think that @stefanklug might be describing the originally intended behaviour.

---

FROM multiarch/qemu-user-static:x86_64-aarch64 as qemu
FROM arm64v8/ubuntu
COPY --from=qemu /usr/bin/qemu-* /usr/bin
#continue with my now qemu equipped docker image

I think that this is equivalent to:

FROM arm64v8/ubuntu
COPY --from=multiarch/qemu-user-static:x86_64-aarch64 /usr/bin/qemu-* /usr/bin
#continue with my now qemu equipped docker image

Also:

FROM alpine:3.5 as qemu_extract
COPY --from=multiarch/qemu-user-static:x86_64-aarch64 /usr/bin qemu_user_static.tgz
RUN tar -xzvf qemu_user_static.tgz

FROM arm64v8/ubuntu
COPY --from=qemu_extract qemu-* /usr/bin
#continue with my now qemu equipped docker image

[eine]

Thanks for the feedback. I did not understand it well.
I saw that dbhi/qus repository. That is quite excellent.
We multiarch project have many things to learn from the dbhi/qus project.

Precisely, dbhi/qus is a superset of project multiarch. See https://github.com/dbhi/qus/blob/master/register.sh#L3. Hence, to be fair, we must say that dbhi/qus did learn many things from this repo, before including additional features. The most relevant difference is that dbhi/qus provides images for different host architectures, and multiarch is focused on amd64 hosts.

Maybe people in the the thread want to create multiarch/qemu-user-static image like dbhi/qus 's aptman/qus image that is a union of qemu-*static binaries and register, right?

I do think so. aptman/qus:f4.0.0 is exactly that. It includes the register.sh script along with all the binaries in v4.0.0-5.

It has a feature to remove binfmt_misc entries. It has a option to register only specific arch's binfmt_misc entry. It is really good.

These new features were proposed upstream (see https://patchew.org/QEMU/20190312195009.GA6@99bbefa4bcea/ and https://patchew.org/search?q=project%3AQEMU+qemu-binfmt-conf.sh), but are not merged yet.

I just used podman instead of docker for my preference.

...

Glad to know that it works with podman too. Another interesting context is Windows 10. With multiarch, it is not easy to setup and execute e.g. arm64v8/fedora, because the user cannot access a terminal in the VM where docker containers are executed (e.g. ls /proc/sys/fs/binfmt_misc). However, dbhi/qus copies the binary from the container to the VM directly. As a result, foreign images/containers can be used on docker-for-windows.

When using "flag: F", qemu-*-static binary is not required anymore, right?
I did not know that. I see "/qus/bin/qemu-aarch64-static" does not exist.

Flag -F (-p) copies the binary to memory as soon as it is registered. Hence, the binary is required when register.sh is executed, but it is not required by any other container executed after it. This is why aptman/qus images include both the script and the binaries.

You can find details in the list of Tests: column 'Dependency' shows if the mechanism depends on having some specific binary file available, or if packages are installed through apt (dnf on Fedora).

So, what are our concrete actions to fix this ticket?

• Check/ensure that ADD does extract the tarball and not only copy it:

  qemu-user-static/update.sh

  Line 34 in 5ee00f7

  ADD https://github.com/${REPO}/releases/download/v${VERSION}/${from_arch}_qemu-${to_arch}-static.tar.gz /usr/bin

Possible enhancements:

• Add all the binaries in multiarch/qemu-user-static:* images to image multiarch/qemu-user-static:register.
• Deprecate the concept of 'Compatible images' and remove the section from the README.
• Deprecate the 'Examples & articles' section of the README and provide some up-to-date article that enforces using -p.

[junaruga]

@1138-4eb thanks for clarifying the tasks and information about windows and etc! I will take a look.
When I send the pull-requests, I want to ask you (all you guys in this thread) to review the pull-requests.

Thanks.

[junaruga]

• Deprecate the concept of 'Compatible images' and remove the section from the README. : not yet
• Deprecate the 'Examples & articles' section of the README and provide some up-to-date article that enforces using -p. : not yet

I sent a pull-request now to update documents to fix above tasks!
#78

[jpawlowski]

As you anyway need to prepare your build host, I believe the binary files are automatically available from the host and can be copied over to the image if you prepare your host like this:

docker run --rm --privileged --name qemu multiarch/qemu-user-static:register --reset

[stefanklug]

Thanks a lot for your responses but I still don't get it.
I see two options:

1. Using the -p flag. For this to work I ensure all qemu-xxx-static live in /usr/bin on the host machine. After calling docker run --rm --privileged --name qemu multiarch/qemu-user-static:register --reset -p, all binfmts are registered and the F flag is set. From now on every xxx app in an container is served using the qemu-xxx-static from the host. This has two downsides:

   • The need to install the qemu-xxx-static on the host, albeit these are only needed by the containers
   • If the container depends on a given minimum qemu version (i.e. I saw instabilities with some older aarch64 versions) I need to ensure the host provides this version. This breaks encapsulation which is the main purpose of containers

2. Not using the -p flag. In this case docker run --rm --privileged --name qemu multiarch/qemu-user-static:register --resetjust registers the binfmts in the kernel, no F flag and loading the interpreter happens at runtime in container space. Now every container has to provide its own /usr/bin/qemu-xxx-static. This is nice as it provides full encapsulation, but raises the question where to get the correct qemu-xxx-static for the container. I thought that was the main reason for the multiarch/qemu-user-static:x86_64-xxx containers.

Which brings me back to my initial question:
What is the reason for an executable.tar.gz named /usr/bin in these containers. (I would expect them to contain /usr/bin/qemu-xxx-static.tar.gz or even better /usr/bin/qemu-xxx-static )

[eine]

I have been doing some further tests. See 1138-4EB/test-qemu.

---

@stefanklug:
Using the -p flag. For this to work I ensure all qemu-xxx-static live in /usr/bin on the host machine. After calling docker run --rm --privileged --name qemu multiarch/qemu-user-static:register --reset -p, all binfmts are registered and the F flag is set. From now on every xxx app in an container is served using the qemu-xxx-static from the host.

Did you actually try this? I found it necessary to:

• Set -p yes. -p alone won't work.
• Add -v=/usr/bin/qemu-aarch64-static:/usr/bin/qemu-aarch64-static so that binaries are found in the container.

NOTE: I believe that it can be improved to share a single dir with all the qemu-*-static binaries.

---

@stefanklug:
This has two downsides:
* The need to install the qemu-xxx-static on the host, albeit these are only needed by the containers

You can use exactly the same binaries that you would put inside the container. So, you don't need to install all the qemu-*-static on the host. You just download the ones you want/need to a temporal folder. In travis-ci.com/1138-4EB/test-qemu only qemu-aarch64-static is downloaded and registered (see FIX and MULTILIB_FIX).

It is still a downside that any other process in the host will use these binaries. However, the advantage is that you don't need to copy anything in the docker images. You can use them straighaway. Furthermore, you can run multiple foreign containers with a single binary, instead of copying it to all the images.

   * If the container depends on a given minimum qemu version (i.e. I saw instabilities with some older aarch64 versions) I need to ensure the host provides this version. This breaks encapsulation which is the main purpose of containers

You can put the qemu-*-static binaries of the version you want in a temporal folder on the host (and only for the foreign architectures you need). Then, temporally use those binaries system-wide, as commented above. When you are done, just reset the registered formats. See QEMU_BIN_DIR in register script.

---

2. Not using the -p flag. In this case `docker run --rm --privileged --name qemu multiarch/qemu-user-static:register --reset`just registers the binfmts in the kernel, no F flag and loading the interpreter happens at runtime in container space. **Now every container has to provide its own /usr/bin/qemu-xxx-static**.

If you don't use -p, you can still share the binary/ies from the host with the containers. See MULTILIB_VOLUME in travis-ci.com/1138-4EB/test-qemu. The advantages of this approach are that you can use a single binary for multiple containers, you can use the version of qemu that you want, and other processes on the host can use different versions of qemu. The downside is that I don't know how to make it work with docker build.

Moreover, the qemu-*-static binaries can be saved in a docker volume. This allows to avoid saving them on the host. But it also allows to run multiple containers with --volumes-from.

This is nice as it provides full encapsulation, but raises the question where to get the correct qemu-xxx-static for the container. I thought that was the main reason for the multiarch/qemu-user-static:x86_64-xxx containers.

It seems that the latest qemu-user-static from Fedora is used (see .travis.yml#L8 and .travis.yml#L14-L16). However, I am unsure about what's the difference between qemu-* and x86_64_qemu-* as it seems that they should be the same (see publish.sh#L24-L29), but sizes do not match.

[junaruga]

Answer for 1st comment's question.

Is /usr/bin just a typo in update.sh? Should it be /usr/bin/qemu_user_static.tgz

Back to first question, summarizing the comments.

https://github.com/multiarch/qemu-user-static/blob/master/update.sh#L36

FROM scratch
ADD ${from_arch}_qemu-${to_arch}-static.tar.gz /usr/bin/

As @1138-4eb mentioned,

If ADD is used to copy a local tarball, it is automatically extracted. See docs.docker.com/engine/reference/builder/#add:

ADD has a logic to extract tar.gz file.

Wouldn't it be better to have the uncompressed binary there? This would allow cool multi-stage dockerfiles. (See below)

As a result is yes, it is put as the uncompressed binary there.

Here is how to check it.

$ docker run --rm -t multiarch/qemu-user-static:aarch64 /usr/bin/qemu-aarch64-static --version
qemu-aarch64 version 4.0.0 (qemu-4.0.0-5.fc31)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers

I also imagine maybe the author wanted to create a kind of container version that has same functions with qemu-user-static RPM or deb package.
That is the philosophy of this repository, and that's why

• multiarch/qemu-user-static:register to register binfmt entries.
• multiarch/qemu-user-static: to manage /usr/bin/qemu-<cpu>-static.

$ sudo dnf install qemu-user-static

$ rpm -ql qemu-user-static
/usr/bin/qemu-aarch64-static
/usr/bin/qemu-aarch64_be-static
...
/usr/lib/binfmt.d/qemu-aarch64-static.conf
/usr/lib/binfmt.d/qemu-aarch64_be-static.conf
...

$ sudo apt-get install qemu-user-static

$ dpkg-query -L qemu-user-static
...
/usr/bin/qemu-aarch64-static
/usr/bin/qemu-alpha-static
/usr/bin/qemu-arm-static
...

Adding document for -p option

I think the readme should cover the -p yes option of:

For @rmoriz 's comment, yes, we should document this flag "-p" to enable flags: F .

https://www.kernel.org/doc/html/v4.18/admin-guide/binfmt-misc.html

Here is the latest manual as a reference.
https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html

I think -p is important.

Because on my Fedora 30 environment, below is the configuration file to register the entry to /proc/sys/fs/binfmt_misc/qemu-aarch64. You see "F" flags. Some programs using binfmt_misc expects "F" flag is enabled.

$ cat /usr/lib/binfmt.d/qemu-aarch64-static.conf
:qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:F

Then Feodra's `qemu-user-staic' registers entry with "flags: F" when it is installed, like this. Actually it is not shell script by "echo ...".

# echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:F" > /proc/sys/fs/binfmt_misc/register

# cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags: F
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

But current docker run --rm --privileged multiarch/qemu-user-static:register (--reset) executes

Reset entries on the host computer like this if --reset option is specified.

# find /proc/sys/fs/binfmt_misc -type f -name 'qemu-*' -exec sh -c 'echo -1 > {}' \;
# ls /proc/sys/fs/binfmt_misc
register  status

Then register entry with no flag (flags: ).

# echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:" > /proc/sys/fs/binfmt_misc/register

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64 
enabled
interpreter /usr/bin/qemu-aarch64-static
flags: 
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

That difference of the flag sometimes causes an issue.

[junaruga]

As we answered @stefanklug 's question, can we close this ticket?

[eine]

@junaruga, I think that the main issue here is that all the sources in this repositorty (multiarch/qemu-user-static) assume that a qemu-*-static binary is required either on the host or inside the "user" docker images. IMHO, this made sense when the repo was created, but it is not required anymore, thanks to the -F/-p option.

Moreover, if multiarch/qemu-user-static:register included both the scripts and the binaries, it would be possible to deprecate the concept of Compatible Images; any docker image would be compatible by default. See #49:

docker run --rm --privileged aptman/qus -- -r
docker run --rm --privileged aptman/qus -s -- -p aarch64

docker build -t arm/test -<<EOF
FROM arm64v8/ubuntu:bionic

# Do your ARM stuff here

EOF

[junaruga]

@1138-4eb

I think that the main issue here is that all the sources in this repositorty (multiarch/qemu-user-static) assume that a qemu-*-static binary is required either on the host or inside the "user" docker images. IMHO, this made sense when the repo was created, but it is not required anymore, thanks to the -F/-p option.

Thanks for the feedback. I did not understand it well.
I saw that https://github.com/dbhi/qus repository. That is quite excellent.
We multiarch project have many things to learn from the https://github.com/dbhi/qus project.

Maybe people in the the thread want to create multiarch/qemu-user-static image like https://github.com/dbhi/qus 's aptman/qus image that is a union of qemu-*static binaries and register, right?

https://github.com/dbhi/qus

Available docker images
...

• aptman/qus: union of the two previous images.

It has a feature to remove binfmt_misc entries. It has a option to register only specific arch's binfmt_misc entry. It is really good.

I just used podman instead of docker for my preference.

$ podman run --rm --privileged aptman/qus -- -r

$ ls /proc/sys/fs/binfmt_misc
register  status

$ sudo podman run --rm --privileged aptman/qus -s -- -p aarch64
Setting /qus/bin/qemu-aarch64-static as binfmt interpreter for aarch64

$ ls /proc/sys/fs/binfmt_misc
qemu-aarch64  register  status

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /qus/bin/qemu-aarch64-static
flags: F
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

standard aarch64 container (that is not multiarch's compatible container) works.

$ podman run --rm -t arm64v8/fedora uname -m
aarch64

but it is not required anymore, thanks to the -F/-p option.

When using "flag: F", qemu-*-static binary is not required anymore, right?
I did not know that. I see "/qus/bin/qemu-aarch64-static" does not exist.

$ ls /qus/bin/qemu-aarch64-static
ls: cannot access '/qus/bin/qemu-aarch64-static': No such file or directory

Moreover, if multiarch/qemu-user-static:register included both the scripts and the binaries, it would be possible to deprecate the concept of Compatible Images; any docker image would be compatible by default. See #49:

I agree with this. Managing the mutliarh's compatible container images is hard.
When a distribution's arch images are well-managed on DockerHub and etc, multiarch project does not manage own image.

So, what are our concrete actions to fix this ticket?

[junaruga]

@1138-4eb and guys, I sent the pull-requst now to implement the new features! I want to ask you to review it.
I am going to send another pull-request to update document, after this pull-request will be merged.
Many thanks for that.
#75

[junaruga]

I just merged the PR to fix above tasks too.
See the updated document :) https://github.com/multiarch/qemu-user-static .

I think I fixed every task in this ticket now.
Can we close this ticket?

[eine]

I think I fixed every task in this ticket now.
Can we close this ticket?

I believe so. Congratulations for your effort on bringing this project up to date!

[junaruga]

Okay, congratulations for the project! I would close this ticket.