Is this a cmake issue or QEMU issue? Broken compile on armhf, works on Aarch64 under Buildx/QEMU

[multiarch]

Is this a cmake issue or QEMU issue? Broken compile on armhf, works on Aarch64 under Buildx/QEMU

[junaruga]

I'm using latest multiarch/qemu-user-static installed via Docker container.

Are you sure? This repository is for "multiarch/qemu-user-static" container image.

I run some builds using QEMUs via Docker buildx.

In my memory docker buildx has using the forked version of qemu internally. This qemu. Not original one.
https://github.com/moby/qemu

Generally following this guide using multiarch/qemu-user-static - https://medium.com/@artur.klauser/building-multi-architecture-docker-images-with-buildx-27d80f7e2408

The used "qemu-user-static" is not this repository's one.

$ sudo apt-get install -y qemu-user-static

But for example if you are running Ubuntu as host bionic, here is the "qemu-user-static" deb package.
https://packages.ubuntu.com/bionic-updates/qemu-user-static

If you have a question related to buildx, the place to ask is here.
https://github.com/docker/buildx

[rhastie]

Yes... definitely using "multiarch/qemu-user-static" container image

Search inside the article (https://medium.com/@artur.klauser/building-multi-architecture-docker-images-with-buildx-27d80f7e2408) for "Docker Image Based Installation" - You'll see the references to multiarch/qemu-user-static. I'm using this approach, however it is confusing as several approaches are described.

[rhastie]

I think I've found the culprit... Building using bionic means a glibc version < 2.28, Focal is >2.28 - It appears the READDIR is returning NULL

https://bugs.launchpad.net/qemu/+bug/1805913

Is there any easy way round this bug/issue?

[junaruga]

I still do not understand your steps.

For Steps to reproduce the issue:, could you describe the steps by only command lines and result like this?

$ command1
result1

$ command2
result2

$ command3
result3

...

And I can not access to your log files. I do not know why.

stdout.out.txt
stderr.out (1).txt

Could you upload the log to your https://gist.github.com ?

[junaruga]

And it's good if you share your the minimal Dockerfile to reproduce the issue too.

[junaruga]

https://medium.com/@artur.klauser/building-multi-architecture-docker-images-with-buildx-27d80f7e2408

Docker Image Based Installation
As an alternative to installing the QEMU and binfmt-support packages on your host system you can use a docker image to satisfy the corresponding requirements. There are several docker images that do the job, among them multiarch/qemu-user-static and docker/binfmt.

If you faced the error as a result of docker buildx ..., in my experience, multiarch/qemu-user-statuc is not related to the result. Because docker buildx uses the internal binfmt_misc and qemu. I think the referred article might be wrong.

If you executed the multiarch/qemu-user-static image, possibly you see /proc/sys/fs/binfmt_misc/qemu-$arch files on your host. So, it might be good to remove the files at once.

https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html

You can remove one entry or all entries by echoing -1 to /proc/.../the_name or /proc/sys/fs/binfmt_misc/status.

# echo -1 > /proc/sys/fs/binfmt_misc/qemu-$arch

Please do it on your responsibility.

[rhastie]

Junaruga,

Gist is here: https://gist.github.com/rhastie/04e5af6fa4f61e8e278a1a5ee27484f2

It is quite an involved Dockerfile inside a private repo which is using cmake to run a compile - If you share your GitHub ID I can add you to the repo - Focal version which is erroring is on the focal branch. Currently master is Bionic - I'm having to revert to Bionic.

All I am changing between the two Dockerfile versions is the base-image

[junaruga]

Now I can see stderr.out.1.txt. Thank you.
Is the log the output of docker buildx ..., right?

I do not want to see your private repository. It's okay not to share it.

This CI service can offer ARM 32 bit native host environment. It's free for public repository. It might not free for the private repository. But it's worth to try.
https://drone.io/

[junaruga]

The example.
https://github.com/junaruga/ci-multi-arch-test/blob/master/.drone.yml#L27

[rhastie]

Now I can see stderr.out.1.txt. Thank you.
Is the log the output of docker buildx ..., right?

I do not want to see your private repository. It's okay not to share it.

This CI service can offer ARM 32 bit native host environment. It's free for public repository. It might not free for the private repository. But it's worth to try.
https://drone.io/

Yes - This is part of the docker buildx output for just ARMv7 build

Thanks for the help - I'll look at https://drone.io

[riku76]

Just wanted to comment that I have encountered this same problem and it seems to be tied to cmake version. 3.13.4 works fine, but cmake 3.18.1 fails to exact same error in the same container.

[TheBrokenRail]

This is reported on CMake's GitLab here: https://gitlab.kitware.com/cmake/cmake/-/issues/20568

[umarcor]

See https://github.com/dbhi/docker#building-docker-images-foron-arm-targets:

Building DynamoRIO on an arm32v7 container on a 64 bit host requires to set -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=armv7, as explained at DynamoRIO/dynamorio#3381.