dhuseby/cargo-bootstrap

Bootstrapping script for Rust Cargo when cross-compilation is difficult or impossible.

About

This python script is designed to do the bare minimum to compile and link the Cargo binary for the purposes of bootstrapping itself on a new platform for which cross-compiling isn't possible. I wrote this specifically to bootstrap Cargo on Bitrig. Bitrig is a fork of OpenBSD that uses clang/clang++ and other BSD licensed tools instead of GNU licensed software. Cross compiling from another platform is extremely difficult because of the alternative toolchain Bitrig uses.

With this script, all that should be necessary to run this is a working Rust toolchain, Python, and Git.

This script will not set up a full cargo cache or anything. It works by cloning the cargo index and then starting with the cargo dependencies, it recursively builds the dependency tree. Once it has the dependency tree, it starts with the leaves of the tree, doing a breadth first traversal and for each dependency, it clones the repo, sets the repo's head to the correct revision and then executes the build command specified in the cargo config.

This bootstrap script uses a temporary directory to store the built dependency libraries and uses that as a link path when linking dependencies and the cargo binary. The goal is to create a statically linked cargo binary that is capable of being used as a "local cargo" when running the main cargo Makefiles.

Dependencies

• pytoml -- used for parsing toml files. https://github.com/avakar/pytoml

• dulwich -- used for working with git repos. https://git.samba.org/?p=jelmer/dulwich.git;a=summary

• requests -- used to download crates. http://python-requests.org

These can be installed via the pip tool:

sudo pip install pytoml dulwich requests

Command Line Options

--cargo-root <path>    specify the path to the cargo repo root.
--target-dir <path>    specify the location to store build results.
--crate-index <path>   path to where crates.io index shoudl be cloned
--target <triple>      build target: e.g. x86_64-unknown-bitrig
--host <triple>        host machine: e.g. x86_64-unknown-linux-gnu
--no-clone             don't clone crates.io index, --crate-index must point to existing clone.
--no-git               don't assume that the crates index and cargo root are git repos; implies --no-clone
--no-clean             don't remove the folders created during bootstrapping.
--download             only download the crates needed to bootstrap cargo.
--no-download          don't download any crates (fail if any do not exist)
--graph                output dot format graph of dependencies.
--urls-file <file>     file to write crate URLs to
--blacklist <crates>   list of blacklisted crates to skip
--patchdir <dir>       directory containing patches to apply to crates after fetching them

The --cargo-root option defaults to the current directory if unspecified. The target directory defaults to Python equivilent of mktemp -d if unspecified. The --crate-index option specifies where the crates.io index will be cloned. Or, if you already have a clone of the index, the crates index should point there and you should also specify --no-clone. The --target option is used to specify which platform you are bootstrapping for. The --host option defaults to the value of the --target option when not specified.

Examples

To bootstrap Cargo on Bitrig I followed these steps:

• Cloned this bootstrap script repo to /tmp/bootstrap.
• Cloned the crates.io index to /tmp/index.
• Created a target folder, /tmp/out, for the output.
• Cloned the Cargo repo to /tmp/cargo.
• Copied the bootstrap.py script to the cargo repo root.
• Ran the bootstrap.py script like so:

./bootstrap.py --crate-index /tmp/index --target-dir /tmp/out --no-clone --no-clean --target x86_64-unknown-bitrig

After the script completes, there is a Cargo executable named cargo-0_5_0 in /tmp/out. That executable can then be used to bootstrap Cargo from source by specifying it as the --local-cargo option to Cargo's ./configure script.

./configure --local-cargo=/tmp/out/cargo-0_5_0

Notes

FreeBSD

Make sure you do the following:

• Install py27-pip package
• Use pip to install pytoml and dulwich python modules
• Install ca_root_nss package
• Run: ln -s /usr/local/share/certs/ca-root-nss.crt /etc/ssl/cert.pem
• Install cmake, openssl, libssh2, libgit2, and pkgconf packages
• Install gmake for building cargo once you've bootstrapped a local cargo with this script.

cargo-bootstrap's Issues

Build scripts expect to be run in their respective crate directories

If for some reason (missing libs or pkg-config) a sys crate build script decides to build its library from source, it fails to run configure because of assuming (example) that the current directory is the one where build.rs is located but bootstrap.py doesn't seem to chdir there before running the script.

Found two different crates with name…

...
cargo:crates-io: Building crates-io-0.2.0 (needed by: cargo-0.11.0)
cargo:crates-io:  PROFILE="release" TARGET="x86_64-unknown-linux-musl" CARGO_MANIFEST_DIR="/tmp/cargo/crates-io-0.2.0" OUT_DIR="/tmp/cargo" CARGO_PKG_VERSION_MAJOR="0" CARGO_PKG_VERSION="0.2.0" CARGO_PKG_VERSION_PATCH="0" HOST="x86_64-unknown-linux-musl" PATH="/home/jirutjak/rustc/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" DEBUG="0" OPT_LEVEL="0" CARGO_PKG_VERSION_MINOR="2" NUM_JOBS="1" CARGO_PKG_VERSION_PRE="" rustc /tmp/cargo/crates-io-0.2.0/lib.rs --crate-name crates_io --crate-type lib -C extra-filename=-0_2_0 --out-dir /tmp/cargo --emit=dep-info,link --target x86_64-unknown-linux-musl -L /tmp/cargo -L /tmp/cargo/lib -l dylib=ssl -l dylib=crypto -L /tmp/cargo/lib -l static=curl --extern curl=/tmp/cargo/libcurl-0_2_19.rlib --extern rustc_serialize=/tmp/cargo/librustc_serialize-0_3_19.rlib --extern url=/tmp/cargo/liburl-0_2_38.rlib
cargo:crates-io: /tmp/cargo/crates-io-0.2.0/lib.rs:2:1: 2:18 error: found two different crates with name `url` that are not distinguished by differing `-C metadata`. This will result in symbol conflicts between the two. [E0523]
cargo:crates-io: /tmp/cargo/crates-io-0.2.0/lib.rs:2 extern crate url;
cargo:crates-io:                                     ^~~~~~~~~~~~~~~~~

Exception:
 from ./bootstrap.py, line 1002:
 build command failed: 101

ls /tmp/cargo/ | grep '^url-*'
url-0.2.38
url-0_2_38.d
url-1.0.0
url-1.1.1
url-1_1_1.d

Can't bootstrap on FreeBSD 10.1

Running like this: ./bootstrap.py --crate-index /tmp/index --target-dir /tmp/out --no-clone --target x86_64-unknown-freebsd

Rust: rustc 1.0.0-dev (built 2015-06-12)

cargo:threadpool: Building threadpool-0.1.4 (needed by: cargo-0.3.0)
cargo:threadpool: rustc /tmp/out/threadpool-0.1.4/src/lib.rs --crate-name threadpool --crate-type lib -C extra-filename=-0_1_4 --out-dir /tmp/out -L /tmp/out

cargo:filetime: Building filetime-0.1.4 (needed by: cargo-0.3.0)
cargo:filetime: rustc /tmp/out/filetime-0.1.4/src/lib.rs --crate-name filetime --crate-type lib -C extra-filename=-0_1_4 --out-dir /tmp/out -L /tmp/out
/tmp/out/filetime-0.1.4/src/lib.rs:110:29: 110:57 error: unresolved import `std::os::freebsd::fs::MetadataExt`. Could not find `freebsd` in `std::os`
/tmp/out/filetime-0.1.4/src/lib.rs:110                         use std::os::$i::fs::MetadataExt;
                                                                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/out/filetime-0.1.4/src/lib.rs:104:9: 123:10 note: in expansion of birthtim!
/tmp/out/filetime-0.1.4/src/lib.rs:125:9: 131:10 note: expansion site
error: aborting due to previous error

Exception:
 from ./bootstrap.py, line 897:
 build command failed: 101```

Exception >= 0.5, < 2.0 is not a valid semver range string

curl: Resolving dependencies for: curl-0.2.19

curl: Looking up info for openssl-sys ^0.7.0
curl: opening crate info: /tmp/index/op/en/openssl-sys
curl: best match is openssl-sys-0.7.11
curl: Found crate already satisfying openssl-sys =0.7.11

Exception:
 from ./bootstrap.py, line 337:
 >= 0.5, < 2.0 is not a valid semver range string

This bug is fixed in #17.

env dictionary with non-string value

Building openssl-sys, I am getting

Exception:
from /usr/lib/python2.7/subprocess.py, line 1335:
execve() arg 3 contains a non-string value

This is from an entry being added to the env dictionary with a non-string value. The mystery to me is why I am getting this and everyone else isn't it -- is my python setup unique, or did openssl-sys change? shrug

Anyways, the fix is easy, and a pull request is forthcoming.

Prepare files for stage1 building

First, thanks a lot for this, it's great!

I've voiced for it to get more consideration by upstream, and we plan to actively use it for bootstrap purposes in Debian. I'm trying to use it for offline builds of stage1, and I can currently do it in a hack-ish way.

In particular, I'm trying to lay down all crates in a way that can be later re-used by ./configure+make. The current approach is to use paths overrides, but a better way would be to synthesize a registry index and point entries to local packages.

As such, my wishlist here is to steal some of the functionalities of cargo-vendor, in order to:

• resolve dependencies and download crates (probably already ok as-is in cargo-bootstrap)
• arrange paths so that they could be used by both carg-bootstrap and stage1 cargo builds
• generate a dummy registry pointing to local crates

can't find crate for `openssl_sys` on FreeBSD10.1

Hi @dhuseby
Thanks for sharing cargo-bootstrap script.

I tried it on FreeBSD 10.1 and got the following error.

cargo:registry:curl:curl-sys: /tmp/out/build_script_curl_sys-0_1_22
cargo:registry:curl:curl-sys: cargo:rustc-link-search=native=/usr/local/lib
cargo:registry:curl:curl-sys: cargo:rustc-link-lib=curl
cargo:registry:curl:curl-sys: 
...
cargo:registry:curl:curl-sys: 
cargo:registry:curl:curl-sys:  cmd: ['-L', 'native=/usr/local/lib', '-l', 'curl']
cargo:registry:curl:curl-sys:  env: {}
cargo:registry:curl:curl-sys: denv: {}

cargo:registry:curl:curl-sys:  env: {}
cargo:registry:curl:curl-sys: rustc /tmp/out/curl-sys-0.1.22/lib.rs --crate-name curl_sys --crate-type lib -C extra-filename=-0_1_22 --out-dir /tmp/out -L /tmp/out --extern libz_sys=/tmp/out/liblibz_sys-0_1_3.rlib --extern libc=/tmp/out/liblibc-0_1_8.rlib --extern pkg_config=/tmp/out/libpkg_config-0_3_4.rlib -l curl -L native=/usr/local/lib -l curl
/tmp/out/curl-sys-0.1.22/lib.rs:5:14: 5:39 error: can't find crate for `openssl_sys`
/tmp/out/curl-sys-0.1.22/lib.rs:5 #[cfg(unix)] extern crate openssl_sys;
                                               ^~~~~~~~~~~~~~~~~~~~~~~~~
error: aborting due to previous error
...

Exception:
 from ./bootstrap.py, line 895:
 build command failed: 101

Here is steps I tried:

git clone https://github.com/dhuseby/cargo-bootstrap /tmp/bootstrap
git clone https://github.com/rust-lang/crates.io-index /tmp/index
mkdir /tmp/out
git clone https://github.com/rust-lang/cargo /tmp/cargo
cp bootstrap.py /tmp/cargo
cd /tmp/cargo
./bootstrap.py --crate-index /tmp/index --target-dir /tmp/out --no-clean --no-cl����������one --no-clean --target x86_64-unknown-freebsd

Could you tell me how to fix this error?
Thanks!

can't find crate for core

im building on ubuntu 18

my build line is per readme,

./bootstrap.py --crate-index /tmp/index --target-dir /tmp/out --no-clone --no-clean --target x86_64-unknown-bitrig

==========================
===== BUILDING CARGO =====
==========================

cargo:jobserver:log:cfg-if: Building cfg-if-0.1.4 (needed by: log-0.4.3)
cargo:jobserver:log:cfg-if:  PROFILE="release" TARGET="x86_64-unknown-bitrig" CARGO_MANIFEST_DIR="/tmp/out/cfg-if-0.1.4" OUT_DIR="/tmp/out" CARGO_PKG_VERSION_MAJOR="0" CARGO_PKG_VERSION="0.1.4" CARGO_PKG_VERSION_PATCH="4" HOST="x86_64-unknown-bitrig" PATH="/home/don/anaconda3/bin:/home/don/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin" DEBUG="0" OPT_LEVEL="0" CARGO_PKG_VERSION_MINOR="1" NUM_JOBS="1" CARGO_PKG_VERSION_PRE="" rustc /tmp/out/cfg-if-0.1.4/src/lib.rs --crate-name cfg_if --crate-type lib -C extra-filename=-0_1_4 --out-dir /tmp/out --emit=dep-info,link --target x86_64-unknown-bitrig -L /tmp/out -L /tmp/out/lib
cargo:jobserver:log:cfg-if: error[E0463]: can't find crate for `core`
cargo:jobserver:log:cfg-if:   |
cargo:jobserver:log:cfg-if:   = note: the `x86_64-unknown-bitrig` target may not be installed
cargo:jobserver:log:cfg-if: error: aborting due to previous error

Exception:
 from ./bootstrap.py, line 1002:
 build command failed: 101

https://www.google.com/search?num=50&safe=active&ei=QKVcW-FEgfbxBdXdlZgP&q=cant+find+crate+for+core&oq=cant+find+crate+for+core&gs_l=psy-ab.3..0i71k1l8.0.0.0.18675.0.0.0.0.0.0.0.0..0.0....0...1..64.psy-ab..0.0.0....0.w0LU31xRHl8

bootstrap.py fails with "pop from empty list" when trying to bootstrap cargo >= 0.11.0

Bootstrapping any cargo version >= 0.11.0 fails with

rand: Resolving dependencies for: rand-0.5.0-pre.2
rand: Skipping optional dep serde
rand: Skipping optional dep serde_derive
rand: Looking up info for rand_core 0.2.0-pre.0
rand: opening crate info: /tmp/index/ra/nd/rand_core

Exception:
 from ./bootstrap.py, line 1244:
 pop from empty list

Instructions (pip) use sudo

can't find toml file @ url crate

$ ./bootstrap.py --cargo-root /home/javan/apps/cargo --crate-index ./index --target-dir ./out --target x86_64-unknown-openbsd

cargo: Looking up info for url ^0.5.7
cargo: opening crate info: ./index/3/u/url
cargo: best match is url-0.5.7

/home/javan/apps/cargo-bootstrap/bootstrap.py(1148)crate_info_from_toml()
-> dbg('failed to load toml file for: %s (%s)' % (cdir, str(e)))
(Pdb)

pytoml could not parse tar-0.4.7/Cargo.toml

cargo: Looking up info for tar 0.4
cargo: opening crate info: /tmp/cargo/crates.io-index/3/t/tar
cargo: best match is tar-0.4.7
cargo: failed to load toml file for: /tmp/cargo/tar-0.4.7 (/tmp/cargo/tar-0.4.7/Cargo.toml(27, 1): msg)

The problematic line is: [target.'cfg(unix)'.dependencies]

Requiring a dependency with a non-default feature enabled doesn't seem to work

Building url-0.5.9 fails:

cargo:curl:url: Building url-0.5.9 (needed by: curl-0.2.18)
cargo:curl:url:  PROFILE="release" TARGET="x86_64-unknown-openbsd" CARGO_MANIFEST_DIR="/home/zofrex/cargo-out/url-0.5.9" OUT_DIR="/home/zofrex/cargo-out" CARGO_PKG_VERSION_MAJOR="0" CARGO_PKG_VERSION="0.5.9" CARGO_PKG_VERSION_PATCH="9" HOST="x86_64-unknown-openbsd" PATH="/home/zofrex/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin:/usr/games:." DEBUG="0" OPT_LEVEL="0" CARGO_PKG_VERSION_MINOR="5" NUM_JOBS="1" CARGO_PKG_VERSION_PRE="" /usr/local/bin/rustc /home/zofrex/cargo-out/url-0.5.9/src/lib.rs --crate-name url --crate-type lib -C extra-filename=-0_5_9 --out-dir /home/zofrex/cargo-out --emit=dep-info,link --target x86_64-unknown-openbsd -L /home/zofrex/cargo-out -L /home/zofrex/cargo-out/lib --extern unicode_bidi=/home/zofrex/cargo-out/libunicode_bidi-0_2_3.rlib --extern rustc_serialize=/home/zofrex/cargo-out/librustc_serialize-0_3_19.rlib --extern unicode_normalization=/home/zofrex/cargo-out/libunicode_normalization-0_1_2.rlib --extern matches=/home/zofrex/cargo-out/libmatches-0_1_2.rlib --extern uuid=/home/zofrex/cargo-out/libuuid-0_2_0.rlib
cargo:curl:url: /home/zofrex/cargo-out/url-0.5.9/src/lib.rs:219:22: 219:34 error: no associated item named `new_v4` found for type `uuid::Uuid` in the current scope
cargo:curl:url: /home/zofrex/cargo-out/url-0.5.9/src/lib.rs:219         OpaqueOrigin(Uuid::new_v4())
cargo:curl:url:                                                                      ^~~~~~~~~~~~
cargo:curl:url: error: aborting due to previous error

Exception:
 from ./bootstrap.py, line 974:
 build command failed: 101

The error message given is because the uuid library was not compiled with the v4 feature, which is optional - the uuid Cargo.toml is:

[features]
use_std = []
v4 = ["rand"]

So the v4 feature is not enabled by default. But it should have been enabled, because the url crate requires it:

[dependencies]
uuid = { version = "0.2", features = ["v4"] }

Looking at the feature handling code it kind of looks like only features from the RHS of declarations will be picked up (values, not keys)? Working on that theory I managed to get a little bit further with this patch:

diff --git a/bootstrap.py b/bootstrap.py
index 031a249..3d713ea 100755
--- a/bootstrap.py
+++ b/bootstrap.py
@@ -821,6 +821,8 @@ class Crate(object):
                     # and any features they depend on recursively
                     def add_features(f):
                         if ftrs.has_key(f):
+                            if not f in features:
+                                features.append(f)
                             for k in ftrs[f]:
                                 # guard against infinite recursion
                                 if not k in features:

With this patch in place, compilation of uuid itself fails, but I can see that the v4 feature gets enabled, which is possibly progress:

cargo:curl:url:uuid: Building uuid-0.2.0 (needed by: url-0.5.9)
cargo:curl:url:uuid:  CARGO_FEATURE_V4="1" CARGO_PKG_VERSION_PRE="" CARGO_MANIFEST_DIR="/home/zofrex/cargo-out/uuid-0.2.0" OUT_DIR="/home/zofrex/cargo-out" HOST="x86_64-unknown-openbsd" PATH="/home/zofrex/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin:/usr/games:." DEBUG="0" OPT_LEVEL="0" CARGO_PKG_VERSION_MINOR="2" PROFILE="release" TARGET="x86_64-unknown-openbsd" CARGO_PKG_VERSION_MAJOR="0" CARGO_PKG_VERSION="0.2.0" CARGO_PKG_VERSION_PATCH="0" CARGO_FEATURE_RAND="1" NUM_JOBS="1" /usr/local/bin/rustc /home/zofrex/cargo-out/uuid-0.2.0/src/lib.rs --crate-name uuid --crate-type lib --cfg feature="v4" --cfg feature="rand" -C extra-filename=-0_2_0 --out-dir /home/zofrex/cargo-out --emit=dep-info,link --target x86_64-unknown-openbsd -L /home/zofrex/cargo-out -L /home/zofrex/cargo-out/lib
cargo:curl:url:uuid: /home/zofrex/cargo-out/uuid-0.2.0/src/lib.rs:253:9: 253:25 error: unresolved name `rand::thread_rng` [E0425]
cargo:curl:url:uuid: /home/zofrex/cargo-out/uuid-0.2.0/src/lib.rs:253         rand::thread_rng().gen()
cargo:curl:url:uuid:                                                          ^~~~~~~~~~~~~~~~
cargo:curl:url:uuid: /home/zofrex/cargo-out/uuid-0.2.0/src/lib.rs:253:9: 253:25 help: run `rustc --explain E0425` to see a detailed explanation
cargo:curl:url:uuid: error: aborting due to previous error

Exception:
 from ./bootstrap.py, line 974:
 build command failed: 101

It picks up rand as a feature but doesn't seem to add it as a library requirement.

For others finding this issue, for the time being I am getting past this point by changing the rand dependency in uuid to be non-optional:

sed -i"" -e 's/^rand.*version\([ ]*=[ ]*"[0-9.]*"\).*/rand\1/' /path/to/output/dir/uuid-0.2.0/Cargo.toml

Dependencies with inline dictionaries seem to be ignored

Cargo build fails because of missing tar crate which doesn't get downloaded along with the others.

There're no
cargo: Looking up info for tar
or
cargo: Looking up info for git2
lines, but git2 gets pulled in because of git2-curl.

It might have something to do with this syntax: https://github.com/rust-lang/cargo/blob/master/Cargo.toml#L27