Feedback / Experience / Adventures meta-issue

[ryankurte]

Feedback / Experience / Adventures meta-issue

[fasterthanlime]

Discovered cargo-binstall through cargo-watch's readme, instantly worked, no complaints.

I've been trying to run cargo binstall some-random-package, and it's often failed due to some-random-package not having the right "releases layout" / not having binaries at all / not having any cargo-binstall config. I'm wondering if cargo-binstall should fall back to building from source in these cases (maybe after asking / only if a CLI flag is passed?). I'd love for cargo-binstall to be my one-stop-shop for installing Rust utilities. If there's a binary, yay! If not, I'll take the compile time hit.

[pinage404]

for the format, this could be done without breaking changes, supporting both the deprecated format and the new archive-format and binary-format

[passcod]

The usage report for quickinstall is because:

• the quickinstall project directly uses this to build more packages
• while it wasn't a requirement it was a pretty strong suggestion that it be implemented in order to support quickinstall

I don't personally want to collect tracking information; Ryan may have a different opinion. It's a service to keep up, potentially bills to pay, security to be looked after; there's privacy considerations. It's a very different ballgame to just providing a tool.

Additionally, I don't see the interest in maintaining two tracking services. The features you propose could very well be built into quickinstall's solution; we even have a different user agent so provenance can in theory be differentiated.

Also note that binstall pulls directly from the urls, it doesn't transit or cache, so if you do have binaries up you can get usage stats from those. For github release artifacts, that information is exposed in the github UI (and API).

[passcod]

It's great to see enthusiasm for this project! It's a big list and there's certainly a lot of potential. I think primarily what we're looking at moderating here is going too fast and not justifying complexity which will come back to bite us in the long run. One thing in particular you've touched on is this:

If cargo-binstall is being used across hundreds or thousands of ci-pipelines

That would be great success! It would also mean that a breaking change will break thousands of uses. I think we want to avoid that, and a good way to avoid this is to be careful on what we implement now so we are less likely to revisit it later. We can move a tad faster than Rust itself ;) but let's not go too far in the "move fast and break things" direction either.

One thing I see often in open source contributions vs open source maintainership is this idea that Ryan touched on a bit earlier: we want to fix problems that exist. But it's not enough that you have the issue. As maintainers, we're looking at the software as it is used by a lot of people. You having the issue is one person. Sometimes it's obvious that something will affect more people. Sometimes it's much less so. Other times, an issue might be better solved outside of this project, and "fixing" it here is a false herring.

We also want to consider the direction for the project. As a secondary maintainer, this is something that I'll often defer to Ryan about, as it's his project in the first place, right? It's things like, do we want to be a one-stop-shop where you hit cargo binstall package and it figures out the right thing to do, even if it has a bunch of "magic" (yes), and do we want to track users (no). Going forward, it might be do we want to focus on the human experience and discourage CI use or do we want to make both ways as ergonomic as possible or do we want to focus on automated use and near-automated use and perhaps accept some minor annoyances for human use. How much of a "drop in replacement for cargo install" do we want to be? Is that even a goal? That kind of thing.

So, let's take a step back, let's take a breath, let's talk some things out and see where we fall. I think also something to keep in mind is both Ryan and I are fairly busy and don't have a lot of time to dedicate to this project: so it's both great to see progress being made, and a bit overwhelming at the same time.

---

Over at diesel there is an issue to support binary installs

Can you expand more on this? I can't see how that would fit/work with what diesel is.

[NobodyXu]

I am working on adding cargo-binstall to taiki-e's install-action taiki-e/install-action#8 so that github action can use cargo-binstall.

[passcod]

I'm working on the "installing a suite of tools from manifest" thing btw

[yonkeltron]

I found out about cargo-binstall through this awesome introductory blog post about cargo-dist. I am admittedly late to the party. Unfortunately, when I went to try this out at work, the corporate proxy got in the way and I can't yet figure out how to specify a certificate bundle like I can with other apps. I already have the customary environment variables set properly such as CARGO_HTTP_CAINFO, SSL_CERT_FILE, and SSL_CERT_PATH. This is a bit frustrating but doesn't have to be the end. Excited to learn more.

[kekonn]

I found out about cargo-binstall through this awesome introductory blog post about cargo-dist. I am admittedly late to the party. Unfortunately, when I went to try this out at work, the corporate proxy got in the way and I can't yet figure out how to specify a certificate bundle like I can with other apps. I already have the customary environment variables set properly such as CARGO_HTTP_CAINFO, SSL_CERT_FILE, and SSL_CERT_PATH. This is a bit frustrating but doesn't have to be the end. Excited to learn more.

Can confirm this solved it for me as well.

[echoix]

It would be nice in the README to have an example of a one-liner to download and install (extract and move) from the links given. At each time I want to use binstall on a new setup without a desktop, I totally forget the syntax when it comes to tar, so I always end up searching how to do it. At least for the linux links

[NobodyXu]

@hawkw You could use --tarfers to specifically use the musl target instead, that will definitely work.
We can modify crates/detect-targets to disable gnu if the interpreter does not exist or is not executable.

[hawkw]

@hawkw You could usw the musl target instead, that will definitely work. We can modify crates/detect-targets to disable gnu if the interpreter does not exist or is not executable.

Oh, that's a great idea, installing statically-linked musl targets on NixOS would probably work fine.

[NobodyXu]

Could you consider changing the default priority to install musl first?

We could add another glibc version check, if it is lower than a certain threshold then we would fallback to musl instead.

We can also provide an environment variable for overriding targets.

[eitsupi]

@eitsupi After thinking this again, I think this is more of an issue with mdbook pre-built and it's a good idea to open an issue there.

Thanks for pointing that!
I'm sorry I don't know exactly what the problem is, but perhaps it has something to do with the fact that they are building here using old Ubuntu 20.04?
https://github.com/rust-lang/mdBook/blob/02f3823e4caae28cc0b1ceac284f81743f8fd29b/.github/workflows/deploy.yml#L26-L31

Edit: I found the issue about that. Thanks! rust-lang/mdBook#1954

In any case, a fallback mechanism to musl would definitely be useful.

[Fyko]

Howdy folks! Really enjoying cargo-binstall. I'm not sure if this already exists (because it's not noted in SUPPORT.md), but it would be lovely to support a bin name that's different from the package name. For a project I'm working on, scyllax-cli, I'd like the bin to be named scyllax, which can easily be done with Cargo.toml's [[bin]] feature, but it doesn't seem to work with binstall.

[passcod]

Yes, this should work as we do parse the Cargo.toml. Can you clarify and open an issue with what "doesn't work" in your case?

[ryankurte]

Discovered cargo-binstall through cargo-watch's readme, instantly worked, no complaints.

glad to hear it ^_^

I've been trying to run cargo binstall some-random-package, and it's often failed due to some-random-package not having the right "releases layout" / not having binaries at all / not having any cargo-binstall config.

ahh, yeah, i'm definitely open to improvements if you have ideas! have contemplated making the discovery process a bit more robust/overzealous if there's no config, or some kind of unofficial registry so where folks have CI builds we could link to their binaries without Cargo.toml config, but i think actually building them would move this from (hopefully) helpful tool to accidentally a package manager.

I'm wondering if cargo-binstall should fall back to building from source in these cases (maybe after asking / only if a CLI flag is passed?). I'd love for cargo-binstall to be my one-stop-shop for installing Rust utilities. If there's a binary, yay! If not, I'll take the compile time hit.

sounds like a great idea to me! given it's already interactive a prompt w/ ci flag to automagically do it would probably make sense?

[fasterthanlime]

given it's already interactive a prompt w/ ci flag to automagically do it would probably make sense?

That sounds great! I vaguely remembered seeing some interactive prompts already in cargo-binstall hence my suggestion.

[pinage404]

It seems that both terms are used in a confusing way :

• bin-dir https://github.com/ryankurte/cargo-binstall/blob/76a7fb00c90598150531cd4a2a91c841cc8df43b/README.md?plain=1#L63
• bin-path https://github.com/ryankurte/cargo-binstall/blob/76a7fb00c90598150531cd4a2a91c841cc8df43b/README.md?plain=1#L71

[pinage404]

It's hard to understand why format :

• sometimes need to be prefixed with a . https://github.com/ryankurte/cargo-binstall/blob/76a7fb00c90598150531cd4a2a91c841cc8df43b/README.md?plain=1#L63
• sometimes the . is included https://github.com/ryankurte/cargo-binstall/blob/76a7fb00c90598150531cd4a2a91c841cc8df43b/README.md?plain=1#L64

Does the format refer to the same concept ?

[ryankurte]

thanks for the reports!

wrt. bin-dir and bin-path, this appears to be a typo / should all be called bin-dir

wrt. format, those are actually two (unfortunately named) different concepts, the first refers to the archive format (eg. .tgz), the second to the binary format (which needs a .exe appended for windows).

it would be nice to rename these to highlight the difference, and it might make sense to include the . in both cases, but i would aim to avoid any breaking changes. it's probably worth looking at whether there's a compatible way to mitigate these and definitely able to improve the documentation to clarify both!

[auronandace]

Would it be possible to add a scan feature to see what you have installed via cargo install and list whether they are also able to be installed via cargo binstall? Sort of a binstall version of this:

https://github.com/nabijaczleweli/cargo-update

[somehowchris]

Something like "install from Cargo.toml" would be nice. We have a case where some people joined the team and didn't know the tools and just listing them one by one can be a long process on multiple machines.

Also this would give us the oportinity to profit from tooling like dependabot, renovate, synk,... which would give you a heads up if some tools have been updated, so you should update your tooling too (or the Cargo.toml which you then can just i.e. cargo binstall --manifest .). This could be included in things like git hooks or make files so all devs know they are running the same thing.

Also, tried to add cicd with cross and a lot more to just have binstall support. I think this is a bit hurdle for some which then just get frustrated. But I'm trying to get a github action running for something like this. (would also be lovely to have #1 with that 😄 )

But in general, just awesome, saves a lot of time

[passcod]

Oh, that's interesting! Are you looking for a list of crates or also something like having version requirements or a lockfile?

[somehowchris]

Yes, well would be nice.

Just has been a small pain point for me today. Me personally, I have a small repo with a Cargo.toml which is watched by renovate, so I know when a tool has been updated. For a) update the container image and b) actualy know what changed.

So lets say if v0.1.1 was released and I have v0.1.0 on my laptop, it would be lovely to just have something like cargo binstall update or as said cargo binstall --manifest . to update the binaries. (or as mentioned have them as githooks for something like git pull)

Edit: Also just had a thought about rust equivalent tooling to like npms husky, and tooling used alongside cargo like cargo-watch or wasm-bindgen-cli, specified in the Cargo.toml under something like package.metadata.binstall-tools to not get confused with other dependencies and to have them versioned per project as they are needed for it. This would not be included in usual tooling like renovate or dependabot but I know renovate can easily be configured to include these and they would be for sure be open to have a PR to include this (behind the scenes it would use cargos registries & functionality as usual)

[somehowchris]

Just saw that there is some usage report thing going on for cargo-quickinstall. Mind if I ask, why aren't you tracking too?

• You could notify maintainers to let them know, people would actually appreciate to use cargo-binstall (kinda like lobbying to get the word spread)
• You could actually have charts on how much something is downloaded, kinda like the stats crates.io itself presents
• Maintainers could know which platforms are used, to optimize or add targets to their cd for binary distribution

Over at cargo-all-features I know that there are about 8k recent downloads via crates.io but would there be interest in binaries (aka to be able to use cargo-binstall)? We don't know as we do not have that kind of inisght

[ryankurte]

yep we're on the same page @passcod, if we can feed better user-agent information (or anything else) to the APIs we're interacting with that could be neat, but definitely prefer no tracking here.

(thinking about feedback, we could have a if this doesn't work for you, please open an issue with the following information link in the help or something to help point people back here when things go wrong?)

[somehowchris]

So, during the last weekend I made some PRs, some intentionally (more or less planed) and some more spontaneousas I ran into some small issues or saw some potential. You @ryankurte engeuraged me to open issues about some things. Just to reiterate, I like this project, so I would like to contribute. Its up to you what you accept, ignore or reject. I'm motivated by learning and also by hearing/reading others opinions.

In order to proceed, I listed some of my wishes/motivations below, I would love to hear from you on which you would like to see PRs, issues or general discussions, or plainly reject. These may include some of the things already merged but may paint the bigger picture

Some ideas:

• As mentioned above I would love to have a command/flag to install from a Cargo.toml manifest -> This would open the oportunity to version tools for ci/cd or deployments. this would follow along the lines of gitops styles as you could version tooling in git. Specifing multiple crates in the cli would need a redo on how to specify versions.
• Remove tempdir and use specific extraction for known files #130 -> no need to make a disk round-trip if the file to extract (or generally an algorithem to compare paths) is known.
• Move away from openssl as better rust alternatives exist #126 #128
• Self updates or hints to version updates -> npm or pip have a nice message telling if one could update this tooling. This could be nice and easy to implement. Just comparing the version built with the latest version from crates.io. Updating itself would be dangerous but providing a command to do so would benefit dx
• Aarch64 support #123 Raspberry pi is going 64bit arm8, so are m1 chips
• Proper exit codes #127 -> Currently additional steps need to be added to scripts such as Dockerfiles to check if a binary has been installed. That's not nice and not as easy to use as I believe it should be.
• Restriction to https transports and package verification #124 #1 -> I see https + checksums as the most simple and viable solution. They would make cargo-binstall as secure as cargo-install
• Support features -> I have seen some comments about this and support for something like this doesn't sound huge. Over at diesel there is an issue to support binary installs, this would be a nice thing. Other rust/cargo cli tools also have feature gates for things like openssl vs rustls,... . A spontaneus idea would be to sort a list of features asc alph + len hash it and introduce a new option for the url templates
• Either a subcommand or new cli along the lines of cargo-bpackage -> Apply best bractises as I believe many developers of crago crates or rust binaries aren't that much into configuring ci & cd for multiple targets, features. actions-rs made clear that there is space for simple tooling with rust in ci&cd. Developing this as a subcommand or its own binary would give the oportunity to also use this on dev devices as some choose to do releases manually but a gh-action would be nice too. This also would also be a move in the direction of 0 config deploy and use binaries but wouldn't harm existing cd
• I believe a gh-action to install and run cargo-binstall would be nice and aide some with their multi platform cis to make use of the prebuilt binaries of cargo-binstall
• Support for third party registries https://doc.rust-lang.org/cargo/reference/registries.html -> This would reintroduce crates-index and would need some investigation
• Support for authentication against the binary host -> Sometimes not every binary should be public i.e. some binaries for some integration tests
• Support for targeting git repositories for the Cargo.toml metadata -> I believe this is what #3 references, this would currently reintroduce openssl-sys via git2-rs as gitoxide doesn't seem ready for a bare-clone

More spontaneous ones:

• Flags to opt out of fallbacks #136 -> In ci, I would like to make sure that its either installed correctly or fails, so I know something is wrong
• More explicit information about versions #116 #112 #125 -> Just be more transparent
• Retries on network failures #131 -> shitty wifi and proxies are frustrating and oversubscribed infra has its things too
• Use parallelism for sorting, mapping and filtering #132 -> Just as many projects join and enter a more "agile" release flow metadata fetched by cargo-binstall keeps on growing. Many environments like gh-actions have more than 1 core, so why not use that? rayon would max out at a 1:1 ratio for threads per core.
• Build optimizations #121 -> If you're offering binaries to download, why not optimize them? If cargo-binstall is being used across hunders or thousands of ci-pipelines then sure 1 or 2 minutes more builtime doesn't hurt
• Shell/CMD scripts to download cargo-binstall -> Currently there are wget instructions, though they just download the tgz or zip files. A script file for CMD/Shell would be nice to a) figure out what the user is running b) unpackage and install the binary, yes this would mean curl ... | wget or something like it but some initial install has to happen. This could be used inside a gh-action to harden its use
• Talking #1 the file from crates.io is not verified, this could be done by using the checksum from crates-index, yes this would mean after removing it, adding it again

If you feel like discussions or comment threads could be too big, feel free to email me

[hawkw]

Binaries installed using cargo-binstall will not execute on NixOS, because they have an interpreter path that points to:

interpreter /lib64/ld-linux-x86-64.so.2

but for locally-built executables, the interpreter path is to the ld in the Nix store:

eliza@noctis:~ $ file /home/eliza/.cargo/bin/cargo-binstall
/home/eliza/.cargo/bin/cargo-binstall: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/ibp4camsx1mlllwzh32yyqcq2r2xsy1a-glibc-2.37-8/lib/ld-linux-x86-64.so.2, for GNU/Linux 3.10.0, with debug_info, not stripped

note that the interpreter path is:

interpreter /nix/store/ibp4camsx1mlllwzh32yyqcq2r2xsy1a-glibc-2.37-8/lib/ld-linux-x86-64.so.2

I first encountered this issue while trying to install Oranda using cargo-binstall (axodotdev/oranda#606) , but I'd imagine this is not an Oranda-specific issue and instead occurs whenever using cargo-binstall to install binaries on NixOS.

I'm not sure if there's really anything that cargo-binstall can do to fix this. It's probably better for NixOS users to just install binaries using Nix (I reached for cargo binstall in the hopes that I could install a newer version than what's packaged for Nix). If I'm wrong and there is a way to make it play nice with NixOS, that would be cool, but I imagine the best solution is probably just to document that cargo-binstall doesn't work on NixOS, so that other people don't run into similar problems.

[eitsupi]

You could use --tarfers to specifically use the musl target instead, that will definitely work.
We can modify crates/detect-targets to disable gnu if the interpreter does not exist or is not executable.

Related to this, I recently ran into a problem that I believe was caused by gnu being installed instead of musl. (PRQL/prql#3336)
Could you consider changing the default priority to install musl first?

Thanks for wonderful tool!

[NobodyXu]

@eitsupi After thinking this again, I think this is more of an issue with mdbook pre-built and it's a good idea to open an issue there.

You could also try using quickinstall build, it uses older glibc.

The best way forward is to check the glibc version used by the binary, if it is newer than the system-installed glibc, then fallback to musl

[Sculas]

I'm not sure why, but cargo-binstall is really slow for me. I am authenticated to GitHub using gh, so it's not GitHub that's rate-limiting me here. This was me installing cargo-nextest for the first time:

$ cargo binstall cargo-nextest -y
 INFO resolve: Resolving package: 'cargo-nextest'
 WARN The package cargo-nextest v0.9.57 will be downloaded from github.com
 INFO This will install the following binaries:
 INFO   - cargo-nextest.exe (cargo-nextest.exe -> C:\Users\$USER\.cargo\bin\cargo-nextest.exe)
 INFO Installing binaries...
 INFO Done in 196.7039375s

Then, I ran the same command again to check how long it took when cargo-nextest was already installed:

$ cargo binstall cargo-nextest -y
 INFO resolve: Resolving package: 'cargo-nextest'
 INFO resolve: cargo-nextest v0.9.57 is already installed, use --force to override
 INFO Done in 45.3914203s

Taking a look at other issues in this repo, I'm definitely sure that this shouldn't take that long.

[echoix]

It looks like it is on Windows, can you rule out any antivirus interference in that test?

[Sculas]

It looks like it is on Windows, can you rule out any antivirus interference in that test?

I first ran cargo uninstall cargo-nextest, then I disabled "Real-time protection" in Windows Defender (I don't have any other AV), and then ran cargo binstall cargo-nextest -y again. These are the results:

$ cargo binstall cargo-nextest -y
 INFO resolve: Resolving package: 'cargo-nextest'
 WARN The package cargo-nextest v0.9.57 will be downloaded from github.com
 INFO This will install the following binaries:
 INFO   - cargo-nextest.exe (cargo-nextest.exe -> C:\Users\$USER\.cargo\bin\cargo-nextest.exe)
 INFO Installing binaries...
 INFO Done in 197.1092732s

No dice, unfortunately. I've reran the same commands, but now with debug logging enabled:

$ cargo binstall cargo-nextest -y --log-level debug

Log: contents.log

[passcod]

Can you provide that log again but with --json-output and at trace level so we can see where the time spent is? thanks

[Sculas]

Can you provide that log again but with --json-output and at trace level so we can see where the time spent is? thanks

I've done the same steps as before (uninstall, turn off AV), here's the command I ran:

$ cargo binstall cargo-nextest -y --log-level trace --json-output

Log: contents.log

---

Looking at the log myself, the 2 things that take the longest are the download from crates.io and the download from GitHub. Why is that? I have a gigabit fiber connection (and a speed test proves that), so it's not my internet being slow. I'm clueless.

[NobodyXu]

That's indeed a usecase we yet to support: We assume that the release name contains package name, not binary name.

I'm working on dist-manifest support which might help this, and maybe we can also support this directly in existing fetchers.

[Fyko]

I just discovered, it seems to work for taplo-cli.

$ cargo binstall taplo-cli -y
$ which taplo-cli
taplo-cli not found
$ which taplo
/home/carter/.cargo/bin/taplo