setuptools-rust fails to cross-compile cryptography using crossenv

[th0ma7]

setuptools-rust fails to cross-compile cryptography using crossenv

[messense]

It's hard to investigate without a working environment, do you have a list of steps to reproduce your build issue locally using docker?

[th0ma7]

@messense Yes indeed I do, and let me know if you need anything else:

Clone my copy of our spksrc framework repository:

$ git clone https://github.com/th0ma7/spksrc.git
$ cd spksrc
$ git checkout rustc-wheels

Create the updated docker image:

$ docker build - < Dockerfile
...
Successfully built 882a0451f1bb

Access the docker image:

$ docker run -it -v $(pwd):/spksrc 882a0451f1bb /bin/bash

Build a python310 package that will invoke building cryptography==38.0.1

root@2ff602f23b71:/spksrc# cd spk/python310
root@2ff602f23b71:/spksrc/spk/python310/# make arch-armv7-7.0

Options of interest for building are:

• make arch-aarch64-7.0
• make arch-x64-7.0
• make arch-evansport-7.0

To build for every archs and every support Synology DSM Linux version (e.g. 6.1 & 7.0): make all-supported

Note that build logs are being output on the console but are also available in build-<arch>-<DSM-version>.log

[th0ma7]

Follow to this, my issue seems to be somewhat due to setuptools-rust not behaving properly using a crossenv.

Here's my finding using docker image above (always using a x86_64 host) and also reproduced on a Debian 11 VM. My first thing to try is, is it even cross-compiling using rust? Lets take ripgrep as an easy app to build:

# CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" cargo install ripgrep --target=aarch64-unknown-linux-gnu --force
# file rg
rg: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

It works! Now let's try that using CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu instead of enforcing --target:

# CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" cargo install ripgrep --force
# file rg
rg: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

Still working! So the theory is that I have a working rustc along with a proper recognition of my target gcc as default linker and CARGO* and RUSTUP* environment variables are also recognized and functional. That's a good start :)

More on this first testing can be found SynoCommunity/spksrc#5435 (comment)

Then I've resumed at trying to build cryptography using our spksrc framework. It's rather easy to trigger with PR above, once the build is complete (e.g. make arch-armv7-7.0) you simply have to first setup your PATH to point towards the crossenv first:

$ export PATH=/spksrc/spk/python310/work-aarch64-7.0/crossenv/cross/bin:/spksrc/spk/python310/work-aarch64-7.0/crossenv/cross/bin:$PATH

Then build cryptography from the extracted source:

$ CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" \
CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu \
CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" \
PYO3_CROSS_LIB_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/lib/ \
PYO3_CROSS_INCLUDE_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/include/ \
python3 setup.py bdist_wheel --py-limited-api=cp36

Interestingly enough it gets a little farther using RUSTFLAGS such as:

RUSTFLAGS="--target=aarch64-unknown-linux-gnu -Clinker=/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" \
CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" \
PYO3_CROSS_LIB_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/lib/ \
PYO3_CROSS_INCLUDE_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/include/ \
python3 setup.py bdist_wheel --py-limited-api=cp36

A call using crossenv pip will also result with the same outcome:

CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" \
CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu \
CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" \
PYO3_CROSS_LIB_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/lib/ \
PYO3_CROSS_INCLUDE_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/include/ \
pip install cryptography==38.0.1

My conclusion is that setuptools-rust is somehow mixing-up things between host & target and not always using the proper linker, often returning to /usr/bin/ld instead of the provided one.

[messense]

# CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" cargo install ripgrep --target=aarch64-unknown-linux-gnu --force
# file rg
rg: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

Does it still work if you also add all the CFLAGS/CXXFLAGS/LDFLAGS env vars?

[messense]

Looks like this circles back to an issue I posted before? benfogle/crossenv#65

[messense]

In our CI we cleared the LIBRARY_PATH crossenv set via --env LIBRARY_PATH=

setuptools-rust/.github/workflows/ci.yml

Line 221 in 96684af

python3.9 -m crossenv "/opt/python/cp39-cp39/bin/python3" --cc $TARGET_CC --cxx $TARGET_CXX --sysroot $TARGET_SYSROOT --env LIBRARY_PATH= --manylinux manylinux1 venv

[th0ma7]

I confirm, editing the crossenv/cross/bin/python3 and removing the LIBRARY_PATH from the extra_envs variable works!!!!

I got a few SSL deprecation warnings, mays have to find a way to pass -Wno-unused-function ... But it works:

# ll dist/cryptography-38.0.1-cp36-abi3-linux_aarch64.whl 
-rw-r--r-- 1 root root 1810579 Oct  1 08:41 dist/cryptography-38.0.1-cp36-abi3-linux_aarch64.whl

[th0ma7]

@messense is there an intent to include the --env LIBRARY_PATH= as built-in to setuptools-rust or should I apply a workaround on my end?

[messense]

is there an intent to include the --env LIBRARY_PATH= as built-in to setuptools-rust

No, we don't want to add too much assumption in setuptools-rust, bad assumption caused a trouble in the past. It's better to be dealt by user.

Closing as resolved.

[th0ma7]

@messense having a global option similar to --library-path= or --disable-library-path would be useful and allow not breaking the code while adding extra functionality