Feature: install from manifest

[cargo-bins]

Feature: install from manifest

[ryankurte]

ooh, i like the idea of this! would it be too self important to call this Tools.toml instead of CargoTools.toml?

[NobodyXu]

Right, well, that's not a great experience. I wonder if we could rig it such that it defaults to two binaries, but produces a multicall given a feature. That way we can generate prebuilds with the smaller multicall setup, and cargo install still works.

I think we can create bins/cargo-tools.rs and bins/cargo-binstall.rs which simply call the multicall main.

Since the binary is named cargo-tools, multicall would resolve to call the right "main".
Same for cargo-binstall.

[passcod]

Hmm, we could compromise by writing only the simpler toml/v1 file which doesn't have any of these things, so that cargo-update can be used to update via cargo-install if the user so desires but not via us. Drop the json/v2 support and eventually have our own manifest/lockfile that can capture the nuances we care about and work with our own tooling, though we should make it nice enough that other stuff can consume/write it too.

[passcod]

I also kinda want to avoid the proliferation of many different files that we write to. ...or maybe we say, let's go and write as many files as we want to, but I'd say let's make our own config folder for all this. there's other stuff we might want to start doing, like keeping caches e.g. for hsts. and having an actual config file, I think that came up once or twice recently

[NobodyXu]

Now that we have the global $CARGO_HOME/binstall/crates-v1.json, I think it is time to support upgrading installed bins.

I propose something like:

cargo binstall --upgrade

to upgrade all installed binaries and

cargo binstall --uprade $crate1 $crate2:$ver2

to upgrade specific binaries only and also enable them to provide a specification for upgrading (it cannot be used to downgrade though).

[NobodyXu]

Now with #270 and cargo-update adding support for cargo-binstall, perhaps we don't need --upgrade anymore?

[NobodyXu]

Perhaps we can add that as a subcommand and make cargo-binstall a multi-call binary?

[passcod]

Are you concerned about size? I was just thinking of having two binaries.

[NobodyXu]

Are you concerned about size?

Yes.

I don't think we can simply invoke cargo-binstall in cargo-tools since we need to check for the latest release.

[passcod]

I don't mind a multi-call, but on windows we'll still need to distribute two (identical) binaries, I think. Probably the best we can do without going down the path of shipping a dynlib along. Also not entirely sure how multicall binaries work in the cargo context, especially with the cargo install cargo-binstall pathway?

[NobodyXu]

Also not entirely sure how multicall binaries work in the cargo context, especially with the cargo install cargo-binstall pathway?

It seems that the user would have to create the symlink themselves in these cases.

cargo-install does not support post installation script.

[passcod]

Right, well, that's not a great experience. I wonder if we could rig it such that it defaults to two binaries, but produces a multicall given a feature. That way we can generate prebuilds with the smaller multicall setup, and cargo install still works.

[NobodyXu]

From #222 (comment):

@passcod Found it!

common_for_install_and_uninstall defines all the metadata for this.

Though I think it might be a good idea to create a format for our own, since the format that make sense to cargo-install but not cargo-binstall, e.g. features, profile, rustc, etc.

It is also not appendable, I will like to format to be appendable for installation though the cargo-tools can remove the duplicate entries when updating installed crates.

Co-op with cargo-install means we have to keep track of the upstream, otherwise we might accidentally break it

[passcod]

We still want to cooperate with the standard files for global installs.

Not entirely sure what to do with them, though, regarding file locking and proper handling and such. I don't really want to embed the cargo lib crate... but maybe that's the lesser evil...

[NobodyXu]

We still want to cooperate with the standard files for global installs.

Other than using cargo install-update -a, I can't think of any other scenario.
Also, since users explicitly request the crate to be installed using pre-built binary, they might not want cargo install-update -a to compile them from source.

Not entirely sure what to do with them, though, regarding file locking and proper handling and such.

I'm doing file locking, but it would take a lot of time to get other part right.

I don't really want to embed the cargo lib crate... but maybe that's the lesser evil...

Neither do I, since the CI already takes >5m, adding cargo lib would at least add 1m or maybe 3-4m.

I watched jonhoo's youtube video on trying to speedup compilation when having cargo as a dependency, the conclusion is that there is not much to do except for waiting for the upstream to add more feature flags and perform more optimization.

The problem seems to be mostly about toml_edit taking too long, but I remembered cargo itself also took quite some time, so I really don't want it to further slow down the CI.

[passcod]

The idea for meta files support, separate from this issue, was for cargo-update to be modified to pull from cargo-binstall when available, originally.

[passcod]

Hmm, toml_edit taking a while is annoying, as I wanted to use that for our own manifest, at least the human-writable bit

[NobodyXu]

The idea for meta files support, separate from this issue, was for cargo-update to be modified to pull from cargo-binstall when available, originally.

As a user, I really don't want cargo install-update -a to update binaries installed via cargo-binstall.
Since I have a mixed set of binary/compiled crates installed locally, I would like two systems to be handled separately.

[NobodyXu]

Hmm, toml_edit taking a while is annoying, as I wanted to use that for our own manifest, at least the human-writable bit

cargo b --timings on M1 (Macbook Air 2020):

cargo b --release --timings with profile.release set to be the same as cargo-binstall:

Since the CI is usually overloaded and I think we only have two cores, IMO it is going to be a lot slower than my M1, taking at least twice or 3x time to build.

Looks like I was wrong about toml_edit, it is taking a lot of time, but its cost relatively small and acceptable compared to cargo.

Though I still don't want to add toml_edit as our CI is already quite slow.

[NobodyXu]

The idea for meta files support, separate from this issue, was for cargo-update to be modified to pull from cargo-binstall when available, originally.

@passcod IMHO this co-op might not work since cargo-binstall cannot provide information.

InstallInfo contains fields that it is unknown to cargo-binstall, such as features, all_features, no_default_features and rustc (currently unused though).

Even profile is unknown to cargo-binstall though assuming release is reasonable.

Thus I think cargo-install's metafiles are designed to serve cargo-install only and not suitable for cargo-binstall.

[NobodyXu]

I propose that we can have a manifest like this:

{
    "name": "cargo-binstall",
    "version_req": none,
    "current_version": "0.10.0"
    "source": {
        "type": "Registry",
        "url": "https://crates.io",
     },
     "target": none,
     "bins": [
         "cargo-binstall",
      ],
}
{
    "name": "..."
}

Before installing a crate, we can perform an optional check to see if it is already present.
After installation, we can simply append to the file.

When checking for upgrade, we would clean up the file by removing entries with duplicate "name", keeping only the latest one.
Then we check each of them one by one for upgrade.

[passcod]

So, the reason I had both tool and package sections in the OP is because there's two concepts of "source": the metadata source, and where the particular file came from (and what target it was installed for etc). I'm not particularly attached to format (toml v json v whatever), but I think that distinction is important, and recording both types of info is useful.

Having the split also meant that a single metafile could be e.g. saved to a repo like the Cargo.lock is, and be used across different hosts, e.g. mac and linux, without conflicting every time it's used on alternating hosts.

Additionally it makes it possible to install from metafile without querying the registry at all, as long as all the required targets are already saved.

[NobodyXu]

Sounds reasonable, I will update the PR after I fixed the error.

[NobodyXu]

@passcod Does something like this sound good to you?

{
    "name": "cargo-binstall",
    "version_req": "*",
    "source": {
        "type": "Registry",
        "url": "https://crates.io",
     },
    "packages": {
         "current_version": "0.10.0",
         "target": "x86_64-unkown-linux-musl",
         "bins": [
             "cargo-binstall",
          ],
      },
}
{
    "name": "..."
}

Having the split also meant that a single metafile could be e.g. saved to a repo like the Cargo.lock is, and be used across different hosts, e.g. mac and linux, without conflicting every time it's used on alternating hosts.

Not sure that is doable, two different hosts can easily have conflict with each other.
E.g. they might have different version_req, one might be * and other might be >0.11.1.

The install-into can also be different.

The [[package]] in your spec is where I really don't understand.

If we want to share that single lockfile across different OSes, then I presume that we must have multiple versions of the same package in [[package]].

[[package]]
tool = "cargo-binstall"
version = "0.9.1"
target = "x86_64-unknown-linux-musl"
source = "https://github.com/ryankurte/cargo-binstall/releases/latest/download/cargo-binstall-x86_64-unknown-linux-musl.tgz"
format = "tar+gzip"
checksum = "b3:c5cd314cea5aca5391427bf9885d11f1fbc86d79679353a36d1b89480422149d"
files = ["cargo-binstall"]

[[package]]
tool = "cargo-binstall"
version = "0.9.1"
target = "aarch64-apple-darwin"
source = "https://github.com/ryankurte/cargo-binstall/releases/latest/download/cargo-binstall-aarch64-apple-darwin.zip"
format = "zip"
checksum = "b3:qwed314cea5aca5391427bf9885d11f1fbc86d79679353a36d1b89480422149d"
files = ["cargo-binstall"]

But then how do we differentiate between two nearly identical records for different OSes?
Using target seems like a very fragile way to differentiate between them.

This also won't work if you are sharing the lockfile across multiple linux installations.
It probably won't even work if you use it in WSL and a native linux installation.

I would rather propose, that the user should share the manifest instead:

version = 1

[defaults.'cfg(target = "x86_64-unkown-linux-musl")']
# Where to install binaries, defaults to `./bin` for projects and to `$HOME/.cargo/bin` for global
install-into = "./vendor"

[tools]
cargo-binstall = "0.9.1"
cargo-watch = "8.1.1"
watchexec = "1.19.0"

This is much easier to make portable and you could even put it in a portable HDD/SDD.

Additionally it makes it possible to install from metafile without querying the registry at all, as long as all the required targets are already saved.

I don't think this is a good idea, we should have a separate file for this and it can only be a best effort since the upstream can change this at anytime.

[passcod]

well, the original thing is to have both the manifest and the lockfile in tandem, modelled like Cargo. Cargo.lock has a fully resolved tree, so if you dev on linux and CI runs on windows, the lockfile need not change, and similarly across targets. however a lockfile is useless on its own, it's only paired with the manifest that it makes sense.

similarly here, the op design is for these two files together. however we can't really reasonably resolve the entire thing so that's why there's provisions for updating the lockfile with new targets even when doing a "non updating" install. the design is also specifically for a project-local manifest and lockfile, with the global use a special case, rather than being made for the global use first. there's surely edge cases and flaws in this first draft as you point out that would need to be worked out but that's the intent.

now, I think perhaps what would be better at this point is we can define something completely different for our own .crates.toml-like global metafile with separate considerations, something more like your original. I'd want to then:

• split it from this issue to avoid derailing further ;P
• define what the goals for the metafile are. what problems does it solve, what the use will be now and in future
• explore a bit the interaction between the metafile and the theoretical manifest+lock described here. are they complementary? conflicting?
• consider the external use. if an auxillary project like cargo-update wants to use it, could they? if a project like quickinstall, which has its own cli, wants to use it, could they? is that something we want to enable or to discourage?

[NobodyXu]

now, I think perhaps what would be better at this point is we can define something completely different for our own .crates.toml-like global metafile with separate considerations, something more like your original.

Since we have agreed to use a different format for global installations, let's keep the discussion in #252 !

[ryankurte]

hey thinking about this a bit more, how do you manage multiple projects with different tool requirements? it feels like a global manifest is a different problem to a per-project manifest (whether that's at a crate or workspace level), particularly given projects could have different tool version requirements.

if you want a -project- to be able to specify required tools it might be plausible to add a tag in Cargo.toml then use the existing build-dependencies and lockfile... as well as necessary to locate the binaries -within- the project / get that on the path somehow?

[passcod]

Yeah, the way I was thinking is that in project mode, we install to a directory inside the project, so it's completely separate from the global installs. There's no convention in rust to have a project-local bin folder that the tooling uses like in Node (node_modules/.bin), which makes this a bit unfortunate to use (vendor/tool args rather than tool args, for example), but plenty of projects have a "scripts" folder, which this would just be an extension of, I guess.

[passcod]

There's also the cargo xtask pattern, but I'm not sure if we could reasonably use that or hook into it for anything.

[passcod]

Implemented by (third party project) cargo-run-bin, see #1514.