multiarch/qemu-user-static

:earth_africa: `/usr/bin/qemu-*-static`

qemu-user-static's Issues

Running mips/mipsel images fails

When trying to run mips/mipsel images, docker can't start the container.

Matt$ docker run -it --rm multiarch/debian-debootstrap:mipsel-jessie /bin/uname -a
panic: standard_init_linux.go:175: exec user process caused "no such file or directory" [recovered]
	panic: standard_init_linux.go:175: exec user process caused "no such file or directory"

goroutine 1 [running, locked to thread]:
panic(0x88f8a0, 0xc820152d90)
	/usr/local/go/src/runtime/panic.go:481 +0x3e6

After a while thinking about an error with the images, I finally found that the no such file or directory was related to the qemu binary.

We can check that the expected one is present in the image:

Matt$ docker run -it --rm multiarch/debian-debootstrap:mipsel-jessie /usr/bin/qemu-mipsel-static /bin/uname -a
Linux b9de7767a6d1 4.4.39-moby #1 SMP Fri Dec 16 07:34:12 UTC 2016 mips GNU/Linux

What happens in fact is that when running docker run --privileged --rm multiarch/qemu-user-static:register, 2 qemu binaries get registered for the mips magic/mask.The same goes for mipsel.

I moved a bit the lines from register.sh to emphasize this:

   echo ':mips:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:'${QEMU_BIN_DIR}'/qemu-mips-static:' > /proc/sys/fs/binfmt_misc/register
echo ':mipsn32:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:'${QEMU_BIN_DIR}'/qemu-mipsn32-static:' > /proc/sys/fs/binfmt_misc/register
   echo ':mipsel:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:'${QEMU_BIN_DIR}'/qemu-mipsel-static:' > /proc/sys/fs/binfmt_misc/register
echo ':mipsn32el:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:'${QEMU_BIN_DIR}'/qemu-mipsn32el-static:' > /proc/sys/fs/binfmt_misc/register

What happens when starting a mips image is that qemu-mipsn32-static is not found.

I worked around that issue (c.f. libjpeg-turbo/libjpeg-turbo#106) by creating a new image that adds another copy of qemu-mipsel-static with the name qemu-mipsn32el-static.

I don't know how this should/could be fixed (well, other than removing/commenting the n32 variants). Adding the real copy of qemu-mipsn32el-static to the image leads to qemu exiting with an error.

git pull --tags error: would clobber existing tag

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind bug

Description:

When I was running the pipeline for old qemu versions, then when I tried git pull --tags, the command showed the error "would clobber existing tag"

Steps to reproduce the issue:

1. I did qemu v4.0.0-4 that is new release. But the current latest released version is v4.0.0-5. #87

2. $ git pull --tags

Describe the results you received:

$ git pull --tags
From github.com:multiarch/qemu-user-static
 ! [rejected]        v4.0.0-4   -> v4.0.0-4  (would clobber existing tag)

Describe the results you expected:

The pull is success.

Environment:

$ git --version
git version 2.21.0

$ git remote -v
jaruga	[email protected]:junaruga/qemu-user-static.git (fetch)
jaruga	[email protected]:junaruga/qemu-user-static.git (push)
meeDamian	[email protected]:meeDamian/qemu-user-static.git (fetch)
meeDamian	[email protected]:meeDamian/qemu-user-static.git (push)
origin	[email protected]:multiarch/qemu-user-static.git (fetch)
origin	[email protected]:multiarch/qemu-user-static.git (push)

• Container application: Docker/Podman/Singularity (Leave only one)

Docker

But the issue is not related to my local environment's docker command.

Output of docker version, podman version or singularity version

$ docker version
Client: Docker Engine - Community
 Version:           19.03.0
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        aeac9490dc
 Built:             Wed Jul 17 18:16:02 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.0
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       aeac9490dc
  Built:            Wed Jul 17 18:14:40 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Additional information optionally:

Using multiarch s390x (and ppc64) on little endian systems - seg fault?

My host is an Ubuntu Bionic VM in ESXi 6.7 on a system with a Xeon processor.
I am using a relatively new version of docker:

Docker version 18.09.6, build 481bc77

I have installed qemu-user-static:

jking@ubuntu:~/bdde$ dpkg -l | grep qemu
ii  qemu-user-static                      1:2.11+dfsg-1ubuntu7.14           amd64        QEMU user mode emulation binaries (static version)

I have run the required register and I can see things in /proc. Let's take s390x for example:

jking@ubuntu:~/bdde$ cat /proc/sys/fs/binfmt_misc/qemu-s390x
enabled
interpreter /usr/bin/qemu-s390x-static
flags:
offset 0
magic 7f454c4602020100000000000000000000020016
mask ffffffffffffff00fffffffffffffffffffeffff

All the examples I could find do trivial checks to see if things are working. When I try to do a simple "yum list" it fails:

[root@5f54f1622a90 bdde]# yum list
Illegal instruction (core dumped)                               [===                                                        ] ---  B/s |   0  B     --:-- ETA

When I try to install something like gcc it fails:

[root@f91e5a0023fa bdde]# yum install gcc
Illegal instruction (core dumped)                               [===                                                        ] ---  B/s |   0  B     --:-- ETA

I am a member of the Boost Community Maintenance Team and I am working on adding a big-endian docker container to CI resources so we can test big and little endian issues in CI. For example there's an open bug in boostorg/uuid about endian handling, and I have no access to a big endian system. I'd like to use a docker container since it is portable and can run in Travis CI.

How can I diagnose and correct this issue? I know this may not be specific to the Fedora image; I could not find any Ubuntu based images that work either (most are too old to be useful).

Thanks.

How to enable i386?

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind question

Description:

How can I register the qemu-static-i386?

Steps to reproduce the issue:

$ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes          
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k

Describe the results you received:

List of different architectures that works perfectly

Describe the results you expected:

The i386 arch

Environment:

• QEMU version:
• Container application: Docker

Output of docker version, podman version or singularity version

docker version                    ✔  10066  17:36:04
Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        74b1e89e8a
 Built:             Thu Jul 25 21:17:37 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       74b1e89e8a
  Built:            Thu Jul 25 21:27:55 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.7.m
  GitCommit:        85f6aa58b8a3170aec9824568f7a31832878b603.m
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Additional information optionally:

Support ARM architectures as a host archicture

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind enhancement

Description:

There are tickets that has an error because the executed host architecture is not x86_64. Right now only x86_64 are supported. Because we only use qemu-user-static RPM x86_64 package in .travis.yml.

https://github.com/multiarch/qemu-user-static/blob/master/.travis.yml#L25

- PACKAGE_URI="https://kojipkgs.fedoraproject.org/packages/qemu/4.0.0/5.fc31/x86_64/qemu-user-static-4.0.0-5.fc31.x86_64.rpm"

To support other architectures, we need to add new piple line to .travis.yml, downloading the arch's qemu-user-static as input.

I think a similar project might support ARM architectures as a host architecture. I am not sure.
https://github.com/dbhi/qus

Additional information optionally:

Related tickets: #8 , #36 ,

Stable mips / riscv64 contaienr images to add README?

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind enhancement

Description:

Related to #15 .
I want to add examples to use mips and riscv64 images to README.md.

• mips: /proc/sys/fs/binfmt_misc/qemu-mips*
• riscv64: /proc/sys/fs/binfmt_misc//qemu-riscv64

Are there some official images for those architectures, such as arm64v8/ubuntu for aarch64?
Thanks.

Register ppc64le too

Hi and thanks for the awesome work!
I'm trying to build an image FROM ppc64le/debian:jessie that only installs iptables.
COPY qemu-ppc64le-static /usr/bin/ is there but docker run --rm --privileged multiarch/qemu-user-static:register --reset doesn't register ppc64le, only ppc which obviously doesn't work.
Is there any issues with registering ppc64le or could we do it?

qemu-s390x-static 2.9.1 release is an old version

qemu-s390x-static release 2.9.1 and other released binaries have a release version of 2.9.0 within the version string. Is this intended?

I am attempting to bump to Debian stretch within kubernetes and am hitting an issue with GNU/Linux vs SYSV binaries. 2.9.1 of qemu is supposed to fix this issue.

~/Downloads $ ./qemu-s390x-static --version                            
qemu-s390x version 2.9.0(qemu-2.9.0-1.fc27)

"go install" command fails while running inside s390x docker container on x86_64 host using qemu

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind bug

Description:

Steps to reproduce the issue:

1. Register x86_64 host with the latest qemu-user-static.
   docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

2. Build the following Docker Image using following Dockerfile.s390x using command docker build -t test/crossbuild:latest-s390x -f Dockerfile.s390x .

Dockerfile.s390x

FROM alpine:3.11 as qemu

ARG QEMU_VERSION=5.0.0-2
ARG QEMU_ARCHS="s390x"

RUN apk --update add curl

#Enable non-native runs on amd64 architecture hosts
RUN for i in ${QEMU_ARCHS}; do curl -L https://github.com/multiarch/qemu-user-static/releases/download/v${QEMU_VERSION}/qemu-${i}-static.tar.gz | tar zxvf - -C /usr/bin; done
RUN chmod +x /usr/bin/qemu-*

FROM s390x/golang:1.14.2-alpine3.11
MAINTAINER LoZ Open Source Ecosystem (https://www.ibm.com/developerworks/community/groups/community/lozopensource)

ARG MANIFEST_TOOL_VERSION=v1.0.2

#Enable non-native builds of this image on an amd64 hosts.
#This must be the first RUN command in this file!
COPY --from=qemu /usr/bin/qemu-*-static /usr/bin/

#Install su-exec for use in the entrypoint.sh (so processes run as the right user)
#Install bash for the entry script (and because it's generally useful)
#Install curl to download glide
#Install git for fetching Go dependencies
#Install ssh for fetching Go dependencies
#Install mercurial for fetching go dependencies
#Install wget since it's useful for fetching
#Install make for building things
#Install util-linux for column command (used for output formatting).
#Install grep and sed for use in some Makefiles (e.g. pulling versions out of glide.yaml)
#Install shadow for useradd (it allows to use big UID)
RUN apk update && apk add --no-cache su-exec curl bash git openssh mercurial make wget util-linux tini file grep sed shadow
RUN apk upgrade --no-cache

#Disable ssh host key checking
RUN echo 'Host *' >> /etc/ssh/ssh_config \
  && echo '    StrictHostKeyChecking no' >> /etc/ssh/ssh_config

#Disable cgo so that binaries we build will be fully static.
ENV CGO_ENABLED=0

#Recompile the standard library with cgo disabled.  This prevents the standard library from being
#marked stale, causing full rebuilds every time.
RUN go install -v std

#Install glide
RUN go get github.com/Masterminds/glide
ENV GLIDE_HOME /home/user/.glide

#Install dep
RUN go get github.com/golang/dep/cmd/dep

#Install ginkgo CLI tool for running tests
RUN go get github.com/onsi/ginkgo/ginkgo

#Install linting tools.
RUN wget -O - -q https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s v1.20.0
RUN golangci-lint --version

#Install license checking tool.
RUN go get github.com/pmezard/licenses

#Install tool to merge coverage reports.
RUN go get github.com/wadey/gocovmerge

#Install CLI tool for working with yaml files
RUN go get github.com/mikefarah/yaml

#Delete all the Go sources that were downloaded, we only rely on the binaries
RUN rm -rf /go/src/*

#Install vgo (should be removed once we take Go 1.11)
RUN go get -u golang.org/x/vgo

#Ensure that everything under the GOPATH is writable by everyone
RUN chmod -R 777 $GOPATH

RUN curl -sSL https://github.com/estesp/manifest-tool/releases/download/${MANIFEST_TOOL_VERSION}/manifest-tool-linux-s390x > manifest-tool && \
    chmod +x manifest-tool && \
    mv manifest-tool /usr/bin/

COPY entrypoint.sh /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/sbin/tini", "--", "/usr/local/bin/entrypoint.sh"]

The build just hangs at RUN go install -v std

3. Also, while running the same command inside s390x container on x86_64 host,, error Illegal instruction (core dumped) is thrown.

Register x86_64 host with the latest qemu-user-static.
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

docker run -it -v /home/test/qemu-s390x-static:/usr/bin/qemu-s390x-static s390x/golang:1.14.2-alpine3.11

Inside s390x container:

apk update && apk add --no-cache su-exec curl bash git openssh mercurial make wget util-linux tini file grep sed shadow
apk upgrade --no-cache

#Disable ssh host key checking
echo 'Host *' >> /etc/ssh/ssh_config
echo '    StrictHostKeyChecking no' >> /etc/ssh/ssh_config

#Disable cgo so that binaries we build will be fully static.
export CGO_ENABLED=0

#Recompile the standard library with cgo disabled.  This prevents the standard library from being
#marked stale, causing full rebuilds every time.
go install -v std

Describe the results you received:
Step 2 just halts while running step RUN go install -v std while building Dockerfile.s390x.
Step 3 gives the following error:
Illegal instruction (core dumped)

Describe the results you expected:
The command should have been successful without any output.

Environment:
x86_64 Ub18.04 4.15.0-101-generic Ubuntu SMP x86_64 GNU/Linux

• QEMU version: 5.0.0-2
• Container application: Docker

Output of docker version, podman version or singularity version

Client: Docker Engine - Community
 Version:           19.03.11
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        42e35e61f3
 Built:             Mon Jun  1 09:12:22 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.11
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       42e35e61f3
  Built:            Mon Jun  1 09:10:54 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Additional information optionally:
When I build the same Dockerfile.s390x on an s390x machine, it is built successfully.

Latest docker image outdated?

On a fresh C2M with docker run --rm --privileged multiarch/qemu-user-static:register done I get this error /build/qemu-Ee59aw/qemu-2.0.0+dfsg/tcg/optimize.c:362: tcg fatal error (qemu bug for sure) but it seems the version is 2.0.0 and not 2.5.0

failed to create new OS thread (have 2 already; errno=22)

I'm trying to run the following code:

package main

import "fmt"

func main() {  
    fmt.Println("Hello, world!")
}

# GOARCH=arm go build ./hello.go
# qemu-arm-static ./hello 

Which works with v2.7 and older.

/tmp/qemu-arm-static2.7 ./hello
Hello, world!

However, on v2.8 and newer I'm seeing

/tmp/qemu-arm-static2.8.0 ./hello
runtime: failed to create new OS thread (have 2 already; errno=22)
fatal error: newosproc

runtime stack:
runtime.throw(0xf6458, 0x9)
	/usr/lib/golang/src/runtime/panic.go:547 +0x78
runtime.newosproc(0x10328000, 0x10337fe0)
	/usr/lib/golang/src/runtime/os1_linux.go:149 +0x180
runtime.newm(0x1103fc, 0x0)
	/usr/lib/golang/src/runtime/proc.go:1516 +0x12c
runtime.main.func1()
	/usr/lib/golang/src/runtime/proc.go:125 +0x24
runtime.systemstack(0x155e00)
	/usr/lib/golang/src/runtime/asm_arm.s:247 +0x80
runtime.mstart()
	/usr/lib/golang/src/runtime/proc.go:1051

goroutine 1 [running]:
runtime.systemstack_switch()
	/usr/lib/golang/src/runtime/asm_arm.s:192 +0x4 fp=0x103247ac sp=0x103247a8
runtime.main()
	/usr/lib/golang/src/runtime/proc.go:126 +0x5c fp=0x103247d4 sp=0x103247ac
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_arm.s:990 +0x4 fp=0x103247d4 sp=0x103247d4

I'm running on go version
go version go1.6.3 linux/amd64

on centos 7.3.1611

What is the purpose of /usr/bin in the multiarch/qemu-user-static:... docker images

Right now these images contain one file /usr/bin which is the tar.gz of the corresponding qemu-user-static binary. I don't understand the reason behind this. My thoughts on this:

• Is /usr/bin just a typo in update.sh? Should it be /usr/bin/qemu_user_static.tgz

  qemu-user-static/update.sh

  Line 34 in 5ee00f7

  ADD https://github.com/${REPO}/releases/download/v${VERSION}/${from_arch}_qemu-${to_arch}-static.tar.gz /usr/bin

• Wouldn't it be better to have the uncompressed binary there? This would allow cool multi-stage dockerfiles. (See below)

In my dockerfile I would love to do something like:

FROM multiarch/qemu-user-static:x86_64-aarch64 as qemu
FROM arm64v8/ubuntu
COPY --from=qemu /usr/bin/qemu-* /usr/bin
#continue with my now qemu equipped docker image

What I have to do right now to get the same effect:

FROM multiarch/qemu-user-static:x86_64-aarch64 as qemu

FROM alpine:3.5 as qemu_extract
COPY --from=qemu /usr/bin qemu_user_static.tgz
RUN tar -xzvf qemu_user_static.tgz

FROM arm64v8/ubuntu
COPY --from=qemu_extract qemu-* /usr/bin
#continue with my now qemu equipped docker image

Maybe I'm just missing something obvious or didn't understand the purpose of these images.

Regards,
Stefan

Update to v2.9

Hi, I'm a big fan of your QEMU builds, thank you!

Now that v2.9.0 was released this week, would you mind upgrading to the latest version?
Cheers!

Can we do a release of v2.7.0?

Hi, is it possible to soon do a new release for v2.7?
That would be cool

qemu-mips64el-static qemu: uncaught target signal 4 (Illegal instruction) - core dumped

/kind bug

1. run mips binary test in host, success!

[root@localhost ~]# file mips
mips: ELF 64-bit LSB executable, MIPS, MIPS-III version 1 (SYSV), statically linked, not stripped
[root@localhost ~]# 
[root@localhost ~]# ./mips
Hello, MIPS!
[root@localhost ~]# 
[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

2. docker run error info, failure!

[root@localhost ~]# docker run --rm  -it --net=host --privileged    -v /usr/bin/qemu-mips64el-static:/usr/bin/qemu-mips64el-static  rhel_mips:7.4  date
qemu: uncaught target signal 4 (Illegal instruction) - core dumped

• QEMU version:

[root@localhost ~]# docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:20:01 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:20:01 2017
 OS/Arch:      linux/amd64

same error info on Docker CE 17.12

> 2.9.1 speed regression with qemu-arm-static?

I'm trying to use qemu-arm-static in a debian stretch based docker image, built from a debootstrap, using raspbian mirrors.

I've noticed that qemu-arm-static versions newer than 2.9.1 are really slow. It takes 10x or 20x as long for apt-get operations to run, for example. My host machine is x86_64, Ubuntu 16.04 (4.13.0-37-generic).

Just wondering if anyone else has seen this regression? I can provide more information if it's useful.

--reset flag remove too much!

Steps to replicate:
Fedora 25
wine-systemd installed

1. Contents of /proc/sys/fs/binfmt_misc should be:
   status register windows windowsPE

2. Run docker run --rm --privileged multiarch/qemu-user-static:register --reset

3. Contents of /proc/sys/fs/binfmt_misc now missing windows and windowsPE

when running --reset, you will effectively remove all but status and register in /proc/sys/fs/binfmt_misc - if you like me have a setup that use the same folder --reset will remove all those files.

--reset flag should just remove qemu files, right ? If so here is a fix:

entries="aarch64 arm m68k mips64 mipsel mipsn32el ppc64 sh4 sparc alpha armeb mips mips64el mipsn32 ppc ppc64le s390x sh4eb"

if [ "${1}" = "--reset" ]; then
    (
    cd /proc/sys/fs/binfmt_misc
    for file in $entries; do
        if [ -f $file ]; then
            echo -1 > "$file"
        fi
    done
    )
fi

Releases for qemu 2.10+

Hello,

I was wondering if the maintainers of this project would consider releasing updates for qemu 2.10 and/or 2.11? The support for various powerPC architectures is improved.

Thanks!

Big endian arch to add multiarch Dockerhub

I want to add big endian arch environment (ppc64 (not ppc64le), s390x) for this project.
But I have no idea how to add it.
Can we add the big endian arch docker container to the DockerHub [1]?

Thank you.

[1] https://hub.docker.com/r/multiarch/ubuntu-debootstrap/tags/

multiarch/qemu-user-static talk slides and video

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind announcement

Description:

Last week, I had a talk session at Flock event (= Fedora project community event) to share how multiarch/qemu-user-static is great. Yes. this project :)

https://flock2019.sched.com/event/SJLx/lets-add-fedora-multiarch-containers-to-your-ci

When we want to know how multiarch/qemu-user-static works, we might need to know below tech topics.

• QEMU (qemu-$arch-static)
• binfmt_misc, how the "flags: F" in a binfmt_misc file works, and why the flag is useful.
• How arch container works with QEMU on host OS.
• How multiarch/qemu-user-static image works.

I believe below my slide with some pictures for the talk would help you to understand the topics.

As people in Fedora would tend to prefer podman than docker for the container command, the examples in the slides are for podman. But you can read it replacing it docker.

https://github.com/junaruga/fedora-workshop-multiarch/blob/master/slides/Lets-add-Fedora-multiarch-to-CI.pdf

As the talk was recorded, when it is prepared, I would share it here.

I could not accomplish it without people's help in this project and dbhi/qus project.
Thanks!

Cheers.

qemu-x86_64-static: can't COPY from qemu-user-static:x86_64-x86_64 image

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind bug

Description:

I'm trying to create dockerfile for base image to be built by CI (drone) for multiple architectures. I want to have a single dockerfile so I'm using ARGs to pass arch value into dockerfile.

Steps to reproduce the issue:

1. Create test.dockerfile

ARG arch
ARG qemu
FROM multiarch/qemu-user-static:x86_64-${qemu} as qemu
FROM ${arch}/mono

ARG qemu
COPY --from=qemu /usr/bin/qemu-${qemu}-static /usr/bin

2. run docker build -f test.dockerfile --build-arg arch=amd64 --build-arg qemu=x86_64 -t test:amd64 .

Describe the results you received:

Step 6/6 : COPY --from=qemu /usr/bin/qemu-${qemu}-static /usr/bin
 COPY failed: Forbidden path outside the build context: usr/bin/qemu-x86_64-static ()

Describe the results you expected:
qemu-x86_64-static is being copied to /usr/bin

If I run docker build -f test.dockerfile --build-arg arch=arm32v7 --build-arg qemu=arm -t test:amd64 . everything works fine.

Environment:

• QEMU version: (if you can know it): ?
• Container application: Docker

Output of docker version, podman version or singularity version

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:34 2019
 OS/Arch:           darwin/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Additional information optionally:

Issue while installing docker inside s390x container under qemu

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind bug

Description:

Not able to install docker inside s390x container under qemu on amd64

Steps to reproduce the issue:

1. docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

2. docker run --it --privileged s390x/ubuntu:18.04

3. apt-get update

4. apt-get install -y
   apt-transport-https
   ca-certificates
   curl
   gnupg-agent
   software-properties-common

5. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

6. add-apt-repository
   "deb [arch=s390x] https://download.docker.com/linux/ubuntu
   $(lsb_release -cs)
   stable"

7. apt-get update

8. apt-get install -y --no-install-recommends docker-ce=18.06.*

Describe the results you received:
These errors are seen while installing docker

invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.

docker does not come up after running service docker start

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Describe the results you expected:
Docker is expected to come up after running service docker start

Environment:

• QEMU version: (if you can know it):
• Container application: Docker

Output of docker version

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:25:46 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 01:24:19 2020
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Additional information optionally:
I tried the following fixes before installing docker inside the s390x container but they didn't work.

printf '#!/bin/sh\nexit 0' > /usr/sbin/policy-rc.d
export RUNLEVEL=1

register: file exists

I'm trying to understand how to build docker images on x86 cups for different architectures like ARM.

I tried the following command on my macbook:

sudo docker run --rm --privileged multiarch/qemu-user-static:register

and one of the output is:

Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
sh: write error: File exists

It looks like the generated file already exists. My problem is that I can't find qemu-arm-static in /usr/bin/.

So, where was it saved?

Thanks

Multiarch Build Alpine Image

I'm trying to build/test a Docker image on travis. The build fails on an ARM alpine image:

The command "docker run --rm --privileged multiarch/qemu-user-static:register" exited with 0.

12.37s$ docker run --rm -v $PWD:/usr/src/myapp -w /usr/src/myapp -v go:/go ${GOIMG} bash -c "cd /usr/src/myapp && go get -d -v -t && go test --cover -v ./... --run UnitTest && go build -v -o docker-flow-proxy"

Unable to find image 'kutsudock/rpi-alpine-go:latest' locally

latest: Pulling from kutsudock/rpi-alpine-go

Status: Downloaded newer image for kutsudock/rpi-alpine-go:latest

standard_init_linux.go:175: exec user process caused "no such file or directory"

I suspect it's something that's missing from the alpine image. Any ideas how I can get this running?

Is this a cmake issue or QEMU issue? Broken compile on armhf, works on Aarch64 under Buildx/QEMU

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind question

Possible issue or bug with QEMU - need help identifying where issue is

Description:

I'm using latest multiarch/qemu-user-static installed via Docker container.

I run some builds using QEMUs via Docker buildx. Docker builds work fine for AArch64, but I consistently get a failure like below when attempting to build for 32-bit ARM (hard-float).

The containers are successfully built for AMD64 (ie. with no emulation) and Aarch64 (using QEMU emulation), but, fail for the ARMHF (ARMv7 32-bit) build again using QEMU emulation. I am using Focal 20.04 as the host and also building a Focal 20.04 container.

I have specifically used cmake 3.16.3 (from Focal 20.04 repo) and also compiled and used cmake 3.17.1

For reference, I am also using Conan as the C++ package manager - Version 1.24.1 / Docker 19.03.8 with the very latest buildx / Moby-Buildkit v0.7.1... All I believe are current

Please find attached the output of my failure

buildx_armhf_failure.txt

Finally, I have done my best to enable the--trace-expand option on cmake and redirect stdout and stderr - As you can see its not perfect but I'm hoping it will give you something to go on..

stdout.out.txt
stderr.out (1).txt

Steps to reproduce the issue:

1. Generally following this guide using multiarch/qemu-user-static - https://medium.com/@artur.klauser/building-multi-architecture-docker-images-with-buildx-27d80f7e2408

2. I can provide access to a Git containing the build Dockerfile - Please let me know the required Git account

3. Typically setting buildx to support linux ARM/v7, Aarch64 and amd64

4. Run using: docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 -t rhastie/container .

Describe the results you received:

Container is successfully built for amd64 (ie. x86_64) no QEMU emulation
Container is successfully built for Aarch64 via QEMU emulation
Container build errors and terminates for Arm/v7 via QEMU emulation

Describe the results you expected:

I would expect the container to be built for both architectures - This may be either cmake issue or a QEMU issue. Unusual that it works for Aarch64 and not armhf under QEMU.

Environment:

x86-64 Running Ubuntu Focal 20.04
Docker 19.03.8
Moby-Buildkit v0.7.1
cmake 3.16.3 and cmake 3.17.1 tested
latest buildx CLI from - https://github.com/docker/buildx
latest QEMU-user-static from - https://github.com/multiarch/qemu-user-static
Container being built using Focal 20.04

• QEMU version: (if you can know it):
• Container application: Docker
  Output of docker version, podman version or singularity version

Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 23:42:35 2020
 OS/Arch:           linux/amd64
 Experimental:      true

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 22:48:33 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2
  GitCommit:        
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:        
 docker-init:
  Version:          0.18.0
  GitCommit: 

Additional information optionally:
I also have an issue raised here: https://gitlab.kitware.com/cmake/cmake/-/issues/20568

4.0.0-5 / xattrs errors on Alpine (GCC)

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind bug

Description:

After I included 4.0.0-5 into my base images, some future builds like for python fails to install gcc with an xattrs set attributes errors. After I revert to previus version, it works.

Steps to reproduce the issue:

I saw the issue only on Azure Pipelines, but I had no time to investigate the issue deeper. I leaf only a issue that maybe other known that it could have an issue with this version.

register image seems to be missing qemu binaries

/kind question

Description:

the :register image seems to be missing the qemu-* binaries

Steps to reproduce the issue:

~ 👺 diff -Nru <(docker pull multiarch/qemu-user-static:register && docker run --rm --privileged --entryp
oint /bin/sh multiarch/qemu-user-static:register -c find) <(docker pull multiarch/qemu-user-static && doc
ker run --rm --privileged --entrypoint /bin/sh multiarch/qemu-user-static -c find)
--- /dev/fd/63  2019-10-01 10:34:37.885057387 +0900
+++ /dev/fd/62  2019-10-01 10:34:37.885057387 +0900
@@ -1,9 +1,45 @@
-register: Pulling from multiarch/qemu-user-static
-Digest: sha256:c77eb2da3597aa370f07ef970e2e0adf155172eb9d3c40e43d97aa43eef6b0c9
-Status: Image is up to date for multiarch/qemu-user-static:register
+Using default tag: latest
+latest: Pulling from multiarch/qemu-user-static
+Digest: sha256:0d5897acf5f822cec80fe40031856d417fa3c8c8cfa5d4dc66e30c74a72a450e
+Status: Image is up to date for multiarch/qemu-user-static:latest
 .
 ./usr
 ./usr/sbin
+./usr/bin
+./usr/bin/qemu-sh4eb-static
+./usr/bin/qemu-ppc64le-static
+./usr/bin/qemu-microblaze-static
+./usr/bin/qemu-sh4-static
+./usr/bin/qemu-nios2-static
+./usr/bin/qemu-mips-static
+./usr/bin/qemu-aarch64-static
+./usr/bin/qemu-sparc64-static
+./usr/bin/qemu-s390x-static
+./usr/bin/qemu-sparc-static
+./usr/bin/qemu-xtensa-static
+./usr/bin/qemu-ppc-static
+./usr/bin/qemu-arm-static
+./usr/bin/qemu-tilegx-static
+./usr/bin/qemu-i386-static
+./usr/bin/qemu-cris-static
+./usr/bin/qemu-alpha-static
+./usr/bin/qemu-armeb-static
+./usr/bin/qemu-sparc32plus-static
+./usr/bin/qemu-mips64-static
+./usr/bin/qemu-ppc64abi32-static
+./usr/bin/qemu-mipsn32el-static
+./usr/bin/qemu-xtensaeb-static
+./usr/bin/qemu-riscv64-static
+./usr/bin/qemu-mips64el-static
+./usr/bin/qemu-hppa-static
+./usr/bin/qemu-ppc64-static
+./usr/bin/qemu-mipsn32-static
+./usr/bin/qemu-m68k-static
+./usr/bin/qemu-microblazeel-static
+./usr/bin/qemu-riscv32-static
+./usr/bin/qemu-mipsel-static
+./usr/bin/qemu-aarch64_be-static
+./usr/bin/qemu-or1k-static
 ./root
 ./dev
 ./dev/core

Describe the results you received:

./usr/bin/qemu-* files are missing in the :register image

Describe the results you expected:

./usr/bin/qemu-* files are included in the :register image

Environment:

• Container application: Docker

Output of docker version

~ 🍊 docker version
Client:
 Version:           18.09.3
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        774a1f4
 Built:             Thu Feb 28 06:34:04 2019
 OS/Arch:           linux/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          18.09.3
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       774a1f4
  Built:            Thu Feb 28 05:59:55 2019
  OS/Arch:          linux/amd64
  Experimental:     false

The message #75 seems that they should be included:

Add all the binaries in multiarch/qemu-user-static:* images to image multiarch/qemu-user-static:register.

This will cause :register --reset -p true to fail w/ sh: write error: No such file or directory is this the expected behavior?

Bump to 2.8

QEMU 2.8 is available, could you please update the images?

See http://ftp.fr.debian.org/debian/pool/main/q/qemu/qemu-user-static_2.8+dfsg-1_amd64.deb

Unsupported ioctl: cmd=0x400454ca, Docker on Intelx86-64 Linux runs Armv7 Containers

/kind bug

Description:

Steps to reproduce the issue:

1. I can build virtual network interface in local. (kernel : tun module)
2. I can also build virtual network interface in others amd64 container (ex: ubuntu 16.04)
3. I successfully run my program on docker avmv7 container.

as the title, when I runs Armv7 Container

root@a4cea7e5c656:/# tunctl
Unsupported ioctl: cmd=0x400454ca
TUNSETIFF: Function not implemented

Is the architecture not yet supported ?
or I need to add ioctl mapping manually？

Environment:

linux version : Linux 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux (ubuntu 18.04)

qemu version : lastest

docker image : armv7/armhf-ubuntu tag: 16.04

docker version :
Client: Docker Engine - Community
Version: 19.03.5
API version: 1.40
Go version: go1.12.12
Git commit: 633a0ea838
Built: Wed Nov 13 07:29:52 2019
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.5
API version: 1.40 (minimum version 1.12)
Go version: go1.12.12
Git commit: 633a0ea838
Built: Wed Nov 13 07:28:22 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683

Unknown Host QEMU IFLA Type

I have one application that I compile in an ubuntu-core:arm64 container that, when run inside that container, causes the following errors:

Unknown host QEMU_IFLA type: 47
Unknown host QEMU_IFLA type: 48
Unknown host QEMU_IFLA type: 43

Copying the executable to an arm64 allows me to run it correctly, but on the x86_64 host, it runs but causes this error (and seems to have problems with network communication, although I am not sure it is related to this problem).

Image version tag to change from "vX.Y.Z-R" to "X.Y.Z-R"

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind enhancement

Description:

We recently released new feature to add version tag images (#74 (comment)) by the PR #84 .

I added the version tag vX.Y.Z-R with people's agreements.
The philosophy came from fX.Y.Z-R and dX.Y.Z-R of https://github.com/dbhi/qus .
I thought if our version tag meant common version tag, I thought vX.Y.Z-R was good to mean the common version.
But in case of this project, I think X.Y.Z-R is better than vX.Y.Z-R, after rethinking it. Because the version string without v is very common in the container version rule such as https://hub.docker.com/_/alpine .

I admit I made mistake.

So, you like I like to change the image version tag to change from "vX.Y.Z-R" to "X.Y.Z-R".
Then I want to run the pipeline for 3.1.1-2, 4.0.0-4 and 4.0.0-5 again.

How do you think?

sh: write error: Invalid argument - Centos 7

/kind bug

Description:

Similar error to #38, but different circumstances.

Operating system info:

$ cat /etc/centos-release
CentOS Linux release 7.7.1908 (Core)

$ uname -a
Linux saturn 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ ll /proc/sys/fs/binfmt_misc
.rw-r--r-- 0 root root 16 Dec 15:06 jexec
.-w------- 0 root root 16 Dec 15:28 register
.rw-r--r-- 0 root root 16 Dec 15:05 status

$ rpm -qa | grep qemu
# nothing, there are no pre-existing qemu packages installed

Then when I run this as root:
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

I get:

Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
sh: write error: Invalid argument
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
sh: write error: Invalid argument
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
sh: write error: Invalid argument
sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc-static as binfmt interpreter for sparc
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc64-static as binfmt interpreter for sparc64
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
sh: write error: Invalid argument
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
sh: write error: Invalid argument
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
sh: write error: Invalid argument
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
sh: write error: Invalid argument
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
sh: write error: Invalid argument
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
sh: write error: Invalid argument
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
sh: write error: Invalid argument
sh: write error: Invalid argument
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
sh: write error: Invalid argument
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
sh: write error: Invalid argument
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
sh: write error: Invalid argument
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
sh: write error: Invalid argument
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
sh: write error: Invalid argument
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
sh: write error: Invalid argument
Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k
sh: write error: Invalid argument

What am I missing? This command works on my other linux mint distro, but my CI nodes are running Centos7.

Arm64

Hi,

I am trying run qemu-user-static:register on arm based machine (aarch64) and I am getting the below error please take a look
root@arm-ucpe-test:# uname -a
Linux arm-ucpe-test 4.10.0-38-generic #4216.04.1-Ubuntu SMP Tue Oct 10 16:33:57 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux

root@arm-ucpe-test:~# docker run --rm --privileged multiarch/qemu-user-static:register
standard_init_linux.go:178: exec user process caused "no such file or directory"

So is this tested on aarch64?
Please let me know if you need more information.

Thanks,
Anand

qemu-arm-static 2.8 and Java+Maven setup not working

Hi,

We have systems that would stop working if we want to use the qemu-arm-static v2.8.0 that is used in this project.
I've created a project to demonstrate the problem.

I'm not quite sure where the problem is to be honest, is it with Docker, qemu or Java ? I don't know for sure. But what I do know, is that it was working with 2.5, 2.6 and 2.7 and stopped working with 2.8

The demonstration project linked above is a very simplified version of what we use and was created specifically to help the debugging of this issue. So feel free to clone it and let me know if you need more information.

The Dockerfile start from "multiarch/debian-debootstrap:armhf-jessie" and simply add oracle-java and maven. The qemu-arm-static binaries are overridden using volume bind (-v).

The test.sh will take care of everything to demonstrate the problem

Register reset errors

$ docker run --rm --privileged multiarch/qemu-user-static:register --reset
sh: write error: File exists
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
sh: write error: File exists
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
sh: write error: File exists
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
sh: write error: File exists
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
sh: write error: File exists
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
sh: write error: File exists
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
sh: write error: File exists
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
sh: write error: File exists
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
sh: write error: File exists
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
sh: write error: File exists
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
sh: write error: File exists
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
sh: write error: File exists
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
sh: write error: File exists
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
sh: write error: File exists
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
sh: write error: File exists
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
sh: write error: File exists
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
sh: write error: File exists
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
sh: write error: File exists

$ docker info
Containers: 3
 Running: 0
 Paused: 0
 Stopped: 3
Images: 26
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.13.0-16-generic
Operating System: Ubuntu 17.10
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.15GiB
Name: me
ID: UGF7:3IR1:SSKZ:BOMH:N667:5IDN:TYYC:VHBG:BHCX:RAQV:NHRY:3V49
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: aledbf
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Any idea how can be fixed?
Thanks in advance

Various errors when compiling on armv7h using qemu-arm-static

Hi there. For some time now I have had a couple of errors pop up when running with qemu-arm-static inside an armv7h chroot for crosscompiling armv7h software.

Most notable and annoying ones are:

QT
I get missing modules (these vary depending on project) when compiling QT applications:

Project ERROR: Unknown module(s) in QT: core gui widgets
Project ERROR: Unknown module(s) in QT: core

The exact same procedure, with qemu-aarch64-static in an aarch64 chroot works fine.
It also builds fine on an armv7h device without qemu at all.

GTK
I get unrecognised image formats when compiling some GTK apps. Like this one:

[3/71] Generating manager_resources_h with a custom command.
FAILED: resources/manager_resources.h 
glib-compile-resources ../resources/pamac.manager.gresource.xml --sourcedir ../resources --internal --generate --target resources/manager_resources.h
failed to load "../resources/package-available.png": Couldn?t recognize the image file format for file ?../resources/package-available.png?
../resources/pamac.manager.gresource.xml: Child process exited with code 1.
[4/71] Generating manager_resources_c with a custom command.
FAILED: resources/manager_resources.c 
glib-compile-resources ../resources/pamac.manager.gresource.xml --sourcedir ../resources --internal --generate --target resources/manager_resources.c --dependency-file resources/manager_resources.c.d
failed to load "../resources/package-available.png": Couldn?t recognize the image file format for file ?../resources/package-available.png?
../resources/pamac.manager.gresource.xml: Child process exited with code 1.
[11/71] Compiling Vala source ../src/common_daemon.vala ../src/package.vala ../src/pamac_config.vala ../src/alpm_config.vala ../src/aur.vala ../src/system_daemon.vala.
ninja: build stopped: subcommand failed.

Again, it builds fine with aarch64 instead and on the armv7h device directly.

I am at a loss here, I even tried different versions of the qemu-arm-static file, but they all had the same issues.
I'm using qemu 2.12, qemu-user-static 2.12 and systemd 239 (for systemd-nspawn).

Support for QEMU 3.0

Since QEMU 3.0 is released, it would be nice to update it. Thanks!

pull error

docker image pull multiarch/qemu-aarch64-static

Error response from daemon: pull access denied for multiarch/qemu-aarch64-static, repository does not exist or may require 'docker login'

Does I need to set some configuration?

Alpine - qemu-s390x-static - apk fetch commands fails with IO error on x86 host

on x86 , I am trying to cross compile for s390x Alpine. The apk command fails while fetch IO errors.

Note: Alpine image works perfectly well on an s390x host.

**Issue: (Below commands executed on x86 host) **
Dockerfile:

FROM s390x/alpine
COPY qemu-s390x-static /usr/bin/
CMD uname -a

uname -a
Linux dk-vm 4.4.0-134-generic #160-Ubuntu SMP Wed Aug 15 14:58:00 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

docker build -t test-alpine -f Dockerfile-test .
Sending build context to Docker daemon  23.27MB
Step 1/3 : FROM s390x/alpine
latest: Pulling from s390x/alpine
cdf21ace9418: Already exists
ea178433f2f0: Already exists
Digest: sha256:903e1eeceeb7208949673dcb557659b5f4c8e9065a45432a0668acdd0069268c
Status: Downloaded newer image for s390x/alpine:latest
 ---> 2fc01d06b47d
Step 2/3 : COPY qemu-s390x-static /usr/bin/
 ---> 0dfcc829de99
Step 3/3 : CMD uname -a
 ---> Running in 1034e895153a
Removing intermediate container 1034e895153a
 ---> e8d1f3431d7a
Successfully built e8d1f3431d7a
Successfully tagged test-alpine:latest
root@dk-vm:~/jenkins/docker# docker run -it --name alpine test-alpine sh
/ # clear
/ # apk add --no-cache git openssh-client curl unzip bash ttf-dejavu coreutils tini
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/s390x/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/main/s390x/APKINDEX.tar.gz: IO ERROR
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/s390x/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/community/s390x/APKINDEX.tar.gz: IO ERROR
ERROR: unsatisfiable constraints:
  bash (missing):
    required by: world[bash]
  coreutils (missing):
    required by: world[coreutils]
  curl (missing):
    required by: world[curl]
  git (missing):
    required by: world[git]
  openssh-client (missing):
    required by: world[openssh-client]
  tini (missing):
    required by: world[tini]
  ttf-dejavu (missing):
    required by: world[ttf-dejavu]
  unzip (missing):
    required by: world[unzip]
  musl-1.1.19-r10:
    breaks: musl-utils-1.1.19-r10[musl?1.1.19-r10] musl-utils-1.1.19-r10[so:libc.musl-s390x.so.1] busybox-1.28.4-r0[so:libc.musl-s390x.so.1]
            libressl2.7-libcrypto-2.7.4-r0[so:libc.musl-s390x.so.1] apk-tools-2.10.0-r0[so:libc.musl-s390x.so.1]
            libressl2.7-libssl-2.7.4-r0[so:libc.musl-s390x.so.1] scanelf-1.2.3-r0[so:libc.musl-s390x.so.1]
            zlib-1.2.11-r1[so:libc.musl-s390x.so.1] alpine-baselayout-3.1.0-r0[so:libc.musl-s390x.so.1]
  busybox-1.28.4-r0:
    breaks: world[busybox] alpine-baselayout-3.1.0-r0[/bin/sh]
  alpine-baselayout-3.1.0-r0:
    breaks: world[alpine-baselayout]
  alpine-keys-2.1-r1:
    breaks: world[alpine-keys]
  libressl2.7-libcrypto-2.7.4-r0:
    breaks: apk-tools-2.10.0-r0[so:libcrypto.so.43] libressl2.7-libssl-2.7.4-r0[so:libcrypto.so.43]
  libressl2.7-libssl-2.7.4-r0:
    breaks: apk-tools-2.10.0-r0[so:libssl.so.45]
  zlib-1.2.11-r1:
    breaks: apk-tools-2.10.0-r0[so:libz.so.1]
  apk-tools-2.10.0-r0:
    breaks: world[apk-tools]
  scanelf-1.2.3-r0:
    breaks: musl-utils-1.1.19-r10[scanelf]
  musl-utils-1.1.19-r10:
    breaks: libc-utils-0.7.1-r0[musl-utils]
  libc-utils-0.7.1-r0:
    breaks: world[libc-utils]
/ # exit

x86_64 not registered

x86_64 does not get registered.
Can you please add x86_64?

Unclean exit when registering with Docker ZFS storage driver

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind bug

Description:

Steps to reproduce the issue:

1. Run docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

2. Observe error:

Unable to find image 'multiarch/qemu-user-static:latest' locally
latest: Pulling from multiarch/qemu-user-static
bdbbaa22dec6: Pull complete
42399a41a764: Pull complete
ed8a5179ae11: Pull complete
1ec39da9c97d: Pull complete
df7dd9470aac: Pull complete
Digest: sha256:25d6e8bb037094525cd70da43edc06a62122028cb9ad434605affbd4fffb3a4f
Status: Downloaded newer image for multiarch/qemu-user-static:latest
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
Setting /usr/bin/qemu-sparc-static as binfmt interpreter for sparc
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /usr/bin/qemu-sparc64-static as binfmt interpreter for sparc64
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k
ERRO[0024] Error waiting for container: container 8daa8a39dacdae857a57bbebe300c5871efe08626931741205b6e50a0a83dbd4: driver "zfs" failed to remove root filesystem: exit status 1: "/usr/bin/zfs fs destroy -r nerv/ROOT/arch/caeeb050ad4ee9a4c3c9b00e69472f75f09287ac95264dca8863801842880e84" => cannot destroy 'nerv/ROOT/arch/caeeb050ad4ee9a4c3c9b00e69472f75f09287ac95264dca8863801842880e84': dataset is busy

Describe the results you received:
Above error is displayed when attempting to clean up the container.

Describe the results you expected:
Clean container removal and no error upon running the command.

Environment:

• QEMU version: (if you can know it): v4.2.0-2
• Container application: Docker (Leave only one)

Output of docker version, podman version or singularity version

Client:
 Version:           19.03.5-ce
 API version:       1.40
 Go version:        go1.13.4
 Git commit:        633a0ea838
 Built:             Fri Nov 15 03:19:09 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.5-ce
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.4
  Git commit:       633a0ea838
  Built:            Fri Nov 15 03:17:51 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.2.m
  GitCommit:        d50db0a42053864a270f648048f9a8b4f24eced3.m
 runc:
  Version:          1.0.0-rc9
  GitCommit:        d736ef14f0288d6993a1845745d6756cfc9ddd5a
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Additional information optionally:
I have attempted this on multiple machines with a zfs storage driver and they all appear to be experiencing the same issue, I've even attempted older versions of QEMU as far back as v4.1.0-1 and can reproduce this issue.

Cannot run suid programs

We are trying to run sudo inside the alpine docker image and it fails with:
sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges

This may be related:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=683205

document how to use multiarch docker image with CI

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind enhancement

Description:

I think it worth mention in README, that Docker Hub and QEMU have different notion of architecture. And I found that by abusing docker volume, I can register a single qemu-user-static binfmt_misc entry easily.

I came up with GitHub Actions workflow like this:

on: push

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        docker_arch: [amd64, i386, arm32v6, arm32v7, arm64v8, ppc64le, s390x] # arm32v5 is not supported by alpine
    steps:
    - name: Register binfmt_misc entry for qemu-user-static 
      env:
        DOCKER_ARCH: ${{ matrix.docker_arch }}
      run: |
        case ${DOCKER_ARCH} in
        	amd64|i386)
        		QEMU_ARCH=
        		;;
        	arm32*)
        		QEMU_ARCH=arm
        		;;
        	arm64*)
        		QEMU_ARCH=aarch64
        		;;
        	*)
        		QEMU_ARCH=${DOCKER_ARCH}
        		;;
        esac
        if [ -n "${QEMU_ARCH}" ]; then
        	docker rm $(docker create --volume qemu-user-static:/usr/bin multiarch/qemu-user-static:${QEMU_ARCH} dummy)
        	docker run --rm --privileged --volume qemu-user-static:/usr/bin:ro multiarch/qemu-user-static:register --persistent yes
        fi
    - name: Build
      env:
        DOCKER_ARCH: ${{ matrix.docker_arch }}
      run: |
        docker run --rm --interactive "${DOCKER_ARCH}/alpine:latest" <<-'EOF'
        	echo 'Hello World!'
        EOF

"sh: write error: Invalid argument" errors when register

Hi, on my x86_64 server:

$ uname -a
Linux a010291 4.4.0-127-generic #153-Ubuntu SMP Sat May 19 10:58:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ docker version
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:17:20 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:15:30 2018
OS/Arch: linux/amd64

Run the command "docker run --rm --privileged multiarch/qemu-user-static:register" encounters errors:

sh: write error: Invalid argument
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
sh: write error: Invalid argument
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
sh: write error: Invalid argument
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
sh: write error: Invalid argument
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
sh: write error: Invalid argument
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
sh: write error: Invalid argument

But run command ""docker run --rm --privileged multiarch/qemu-user-static:register --reset" actually installs all the interpreters successfully:

Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa

qemu-user-static is not static anymore

See multiarch/centos#2 (comment)

It's a recent issue:

• v2.9.0 is KO
• v2.8.4 is OK

"write pipe: bad file descriptor" on aarch64

Environment: multicore ARMv8 machine running Ubuntu 16.04.

Trying to follow directions, and I get this:

emv@armv8hello:~$ docker run --rm --privileged multiarch/qemu-user-static:register
Unable to find image 'multiarch/qemu-user-static:register' locally
register: Pulling from multiarch/qemu-user-static
8ddc19f16526: Already exists 
5ac0057b3331: Pull complete 
Digest: sha256:ee56b87e8debafb7c6ed2a2d2b7c2b5f9d471e4bee0cb84956f97d0316f4bc59
Status: Downloaded newer image for multiarch/qemu-user-static:register
write pipe: bad file descriptor

I have a feeling I'm missing something, perhaps a longer README about getting started.

thanks!

Ed

cannot execute binary file: Exec format error

When I executed any of qemu

Creating issue template

I like to create a GitHub issue template for this repository.

Because maybe we want to know

• A used internal qemu version
• Used docker or docker compatible software (podman, singularity) and the version.
• Which compatible images are used, or own image with qemu-user-static
• What command was executed.
• What the expectation and actual result are.

https://help.github.com/en/articles/about-issue-and-pull-request-templates
https://help.github.com/en/articles/creating-issue-templates-for-your-repository

I think below is a good example about the content though it is a old version of the template.
https://github.com/containers/libpod/blob/master/.github/ISSUE_TEMPLATE.md

Below is a case for the new version.
https://github.com/WorksOnArm/cluster/tree/master/.github/ISSUE_TEMPLATE

Shall we create it?
I believe this makes our work better.

Speed is very slow in user-mode emulation for mips64 arch

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind question

Description:

Using qemu-mips64el-static and image mips64el aoqi/debian-mips64el:oldstable to compile bazel application, it will very slow with javac ... in debug output(it's not still finished in 10m above). However, it will costs 1m ~ 2m to finish javac command in x86_64 container). In mips64el container, speed is too slow.

Steps to reproduce the issue:

1. start x86_64 debian:stretch(since debian:stretch has java 8), and start mips64el aoqi/debian-mips64el:oldstable(https://hub.docker.com/r/aoqi/debian-mips64el) with qemu-mips64el-static in another terminal

2. install openjdk-8-jdk etc. development tools, and download it from https://github.com/bazelbuild/bazel/releases/download/2.0.0/bazel-2.0.0-dist.zip and then extract it

3. start compile with below similar command

root@container:/home/bazel-2.0# pwd
/home/bazel-2.0
root@container:/home/bazel-2.0# PATH=/usr/lib/ccache:$PATH EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk --explain /tmp/LOG --verbose_explanations --compilation_mode fastbuild --verbose_failures --show_progress --subcommands --show_timestamps --color yes" VERBOSE=yes BAZEL_DEBUG_JAVA_COMPILATION=1 BAZEL_JAVAC_OPTS="-J-Xmx6g -J-Xms4g" bash compile.sh

Describe the results you received:

Using mips64el aoqi/debian-mips64el:oldstable to compile it, it will very slow with javac ... in debug output. However, it will costs 1m ~ 2m to finish javac command in x86_64 container):

root@container:/home/bazel-2.0# ...... bash compile
... ...

tools/java/runfiles/Runfiles.java
tools/java/runfiles/Util.java
/usr/lib/jvm/java-8-openjdk-amd64/bin/javac -classpath ... ...
... ...

Describe the results you expected:

I know about that performance will decrease using qemu user mode, but it's abnormal for so slow speed in qemu-mips64el-static. It is a normal phenomenon?

Environment:

• x86_64 host

[yancy@asus github]$ cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core) 
[yancy@asus github]$ uname  -a
Linux asus 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

https://kojipkgs.fedoraproject.org/packages/qemu/4.0.0/5.fc31/x86_64/qemu-user-static-4.0.0-5.fc31.x86_64.rpm

[yancy@asus github]$ qemu-mips64el-static --version
qemu-mips64el version 4.0.0 (qemu-4.0.0-5.fc31)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers

[yancy@asus github]$ cat /proc/sys/fs/binfmt_misc/qemu-mips64el 
enabled
interpreter /usr/bin/qemu-mips64el-static
flags: 
offset 0
magic 7f454c4602010100000000000000000002000800
mask ffffffffffffff000000000000000000feffffff

[yancy@asus github]$ rpm -qa docker-ce
docker-ce-19.03.5-3.el7.x86_64
[yancy@asus github]$ docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:25:41 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:24:18 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

• qemu-mips64el container

root@container:/home/bazel-2.0-mips64# uname  -m
mips64
root@container:/home/bazel-2.0-mips64# java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1~deb9u1-b10)
OpenJDK 64-Bit Zero VM (build 25.222-b10, interpreted mode)
root@container:/home/bazel-2.0-mips64# qemu-mips64el-static --version
qemu-mips64el version 4.0.0 (qemu-4.0.0-5.fc31)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers

qemu: uncaught target signal 5

Hello @lafin,

When running my travis job with qemu version 2.12.0 (arm) the following error occured, while running with 2.9.1-1 it succeeds.

Step 19/23 : RUN npm install
 ---> Running in 0a4abea8c736
#
# Fatal error in , line 0
# Missing deoptimization information for OptimizedFrame::Summarize.
#
qemu: uncaught target signal 5 (Trace/breakpoint trap) - core dumped
Trace/breakpoint trap (core dumped)
The command '/bin/sh -c npm install' returned a non-zero code: 133

Can you please have a look?

Thanks in advance,
Ray