axodotdev/cargo-dist

📦 shippable application packaging

cargo-dist

cargo-dist distributes your binaries

The TL;DR is that with cargo-dist setup, just doing this:

git commit -am "release: 0.2.0"
git tag "v0.2.0"
git push
git push --tags

Will make this Github Release:

Or if you're using oranda, you'll get this website.

Plan, Build, Host, Publish, Announce

Cutting releases of your apps and distributing binaries for them has a lot of steps, and cargo-dist is quickly growing to try to cover them all!

To accomplish this, cargo-dist functionality can be broken up into two parts:

• building (planning the release; building binaries and installers)
• distributing (hosting artifacts; publishing packages; announcing releases)

The build functionality can be used on its own if you just want some tarballs and installers, but everything really comes together when you use the distribution functionality too.

Building

As a build tool, cargo-dist can do the following:

• Pick good build flags for "shippable binaries"
• Make tarballs and installers for the resulting binaries
• Generate machine-readable manifests so other tools can understand the results

That's a short list because "we make installers" is doing a lot of heavy lifting. Each installer could be (and sometimes is!) an entire standalone tool with its own documentation and ecosystem.

Distributing

As a distribution tool, cargo-dist gets to flex its biggest superpower: it generates its own CI scripts. For instance, enabling GitHub CI with cargo dist init will generate release.yml, which implements the full pipeline of plan, build, host, publish, announce:

• Plan
  • Waits for you to push a git tag for a new version (v1.0.0, my-app-v1.0.0, my-app/1.0.0, ...)
  • Selects what apps in your workspace to announce new releases for based on that tag
  • Generates a machine-readable manifest with changelogs and build plans
• Build
  • Spins up machines for each platform you support
  • Builds your binaries and tarballs
  • Builds installers for your binaries
• Publish:
  • Uploads to package managers
• Host + Announce:
  • Creates (or edits) a GitHub Release
  • Uploads build artifacts to the Release
  • Adds relevant release notes from your RELEASES/CHANGELOG

Read The Book!

We've got all the docs you need over at the cargo-dist book!

• Introduction
• Install
• Way-Too-Quickstart
• Workspaces Guide
• Reference

Contributing

Updating Snapshots

cargo-dist's tests rely on cargo-insta for snapshot testing various outputs. This allows us to both catch regressions and also more easily review UI/output changes. If a snapshot test fails, you will need to use the cargo insta CLI tool to update them:

cargo install cargo-insta

Once installed, you can review and accept the changes with:

cargo insta review

If you know you like the changes, just use cargo insta accept to auto-apply all changes.

(If you introduced brand-new snapshot tests you will also have to git add them!)

NOTE: when it succeeds, cargo-dist-schema's emit test will actually commit the results back to disk to cargo-dist-schema/cargo-dist-schema.json as a side-effect. This is a janky hack to make sure we have that stored and up to date at all times (the test also uses an insta snapshot but insta snapshots include an extra gunk header so it's not something we'd want to link end users). The file isn't even used for anything yet, I just want it to Exist because it seems useful and important. In the future we might properly host it and have our outputs link it via a $schema field.

Cutting Releases

cargo-dist is self-hosting, so you just need to push a git-tag with the right format to "do" a release. Of course there's lots of other tedious tasks that come with updating a release, and we use cargo-release to handle all those mechanical details of updating versions/headings/tags. See these sections of the docs for the release workflow we use.

TL;DR:

• Update CHANGELOG.md's "Unreleased" section to include all the release notes you want
• run cargo-release as described in the docs
• ..you're done!

Note that we've wired up cargo-dist and cargo-release to understand the "Unreleased" heading so you should never edit that name, the tools will update it as needed.

If that releases succeeds, we recommend updating the bootstrap version of cargo-dist as a follow up:

• install the version of cargo-dist you just released on your system
• run cargo dist init --yes
• commit "chore: update bootstrap dist to ..."

Note that as a consequence of the way we self-host, cargo-dist's published artifacts will always be built/generated by a previous version of itself. This can be problematic if you make breaking changes to cargo-dist-schema's format... so don't! Many things in the schema are intentionally optional to enable forward and backward compatibility, so this should hopefully work well!

cargo-dist's Issues

better installer behaviour

Various improvements:

• cargo home
  • respect CARGO_HOME
  • Update ~/.cargo/.crates.toml to register the binary?
• other install dirs
  • #235
  • Look for other big likely-on-PATH dirs?
• better installs
  • #314
  • Run an "install" subcommand of the binary itself?

Centralize curl-sh expression generation

We generate these expressions as "installer hints" in the main code but the github CI code also recomputes it -- we should have a single source of truth here.

(Slightly complicated by the current hints including comments saying they're dangerous)

add snapshot(?) tests for release note generation

This probably necessitates flags for specifying the version to assume we're releasing, and to output just this part to a file/stdout. Or it should be a unit test of the library (not wired up).

v1.0 settings

We read in [metadata.workspace.dist] and [metadata.$package.dist] but no keys are currently supported. This is a metabug for dist-specific settings that should be added.

• #17
• #18
• #19
• #22
• #28
• #33
• #34

better templating for generating installers/ci

Currently I just have a bunch of HUGE string literals that start/end between spots where I need to inject values and... that sucks lol. We should be using some proper rust string templating library. I just hacked it up like this to get an MVP working.

asdsadas

"run" and "install" subcommands

These would be useful for testing that cargo-dist builds Actually Work.

Ideally by default both would make a zip and then unpack the binary to try to best emulate end-user usage.

Similarly it would be nice if they made sure debuggers could "see" the split-debuginfo files, if at all possible.

Dependency License Gathering

Something Something Software Build Of Materials Something Something Summary File? Not sure if there's a Good tool for this yet we should just use. Not sure if there's a Standard Format to produce (iirc linux distros have some tooling around this we should interop with).

linux flatpak support?

This one is gui-specific aiui so might be out of scope for our particular focus.

Regardless out of scope for the MVP.

Better downloads listings in release notes

The current impl is a linear table with not-super-useful fields/labels.

In the future when things get more complex it might be nice to have this kind of layout (where the lhs is labels and the rhs is links):

Although I believe markdown tables don't support cell merging, and github strips manual table html :(

This isn't a huge deal because "better UX for downloads listings" is just "use oranda" at the end of the day.

Don't generate github urls in auto-generated release notes unless using github

Most of the code is fairly rigorous about only talking about github releases if you've explicitly asked it to, but the release notes code was cludged up by me for the MVP to unconditionally assume it.

Probably it should look for --ci=github being set..? This may necessitate changing generate-ci to make it pass that flag for the command where we compute The One True dist-manifest.json (which is then used for the release notes).

path / private info sanitization?

I vaguely recall there's a cargo/rustc flag for stripping build-machine paths from the binary or something. Not terribly important for CI builds, but important if anyone opts to use this manually. Good Defaults!

adsadasd

release note parsing?

It would be nice to automatically look at RELEASES.md/CHANGELOG.md and parse out the changes for the current version (if they exist).

https://github.com/taiki-e/parse-changelog works pretty well but will complain and die on releases having duplicate entries (cargo-release I believe has a few of these kinds of entries lolsob).

Add dist-manifest.json to itself?

There's a stubbed out artifact type that's just "dist-manifest.json itself" because well, it is an artifact of cargo-dist.

Currently it's a bit weird though because it's not produced in the target/distrib/ dir at all, but is just something that shows up on stdout (if you specify --output-format=json). It's possible we can/should just add a flag to say "hey make this thing a proper artifact too for this run". You don't want it to be an unconditional artifact because you only want one instance to produce it (same problem as installer=github-shell).

Or maybe it being considered an artifact is "too cute" and I should just drop it!

pipe "interesting" cargo.toml fields into the output manifest:

https://doc.rust-lang.org/cargo/reference/manifest.html#the-manifest-format

Currently with "1 binary = 1 release" this is a pretty straight-forward mapping. It will be more complicated when we support grouping things.

rationalize --no-builds

This is a flag I hackily added to support invocations that just generate the "platform agnostic" shell installers, but it's not terribly well thought out or implemented.

For one, we still think we should produce build-dependent artifacts like executable-zips. I believe we actually produce zips but just without the binaries. This necessitates the CI script filtering out anything that isn't an "installer".

I'm not really sure if --no-builds should just be made to work Better, or if it should be replaced with more affirmative flags to say "I want to build these exact artifacts kthx".

test

Way to specify desired targets

Building the host target is obvious and easy, but you might want to cross-compile for ARM64 or i386 or even other OSes.

At a minimum milestone 1 should support building:

• aarch64-apple-darwin.tar.gz
• aarch64-unknown-linux-gnu.tar.gz
• x86_64-apple-darwin.tar.gz
• x86_64-pc-windows-msvc.zip
• x86_64-unknown-linux-gnu.tar.gz

The two aarch64 cases require cross-compilation and are a bit of a "killer app" in the sense that it's non-obvious how to generate them for someone just setting up basic CI by hand. The rest can be handled by different github runners. Probably need to get cargo-cross involved?

os-vendor code signing

Some platforms/app-stores complain if you run unsigned code, and want you to do some fiddly junk with them (...potentially giving them money to be part of their developer program).

This could be relevant to:

• apple platforms (mac app store, ios app store, "Gatekeeper")
• microsoft platforms (windows app store, "SmartScreen")
• android?

Not sure if this is in scope for dist or a different tool altogether, but I sure think about it a lot.

properly detect if "init" hasn't been run

The only reason you "must" run init is to ensure --profile=dist works. I don't think there's any way to ask cargo if that profile is defined, so there are three options:

• parse the Cargo.tomls and try to find the definition (will erroneously fail if the profile is specified totally globally... but also that won't work when checked in anyway so they're still wrong?)
• ??? is there some info in cargo-metadata about defined profiles ???
• parse the error message from rustc/cargo for the "hey the dist profile doesn't exist" to provide a better diagnostic

test

Handle Workspaces

There are a few complicated situations that we need to consider in a workspace:

• Multiple packages (some having no binaries, but potentially newly published!)
• Multiple packages that have binaries
• Multiple binaries for a single package
• Platform-specific packages/binaries?
• Non-application binary artifacts? (e.g. a dll the application loads)

I can imagine situations where you'd want all the binaries in one package, and one where you'd want each one to have its own package. Probably we will need to support some configuration for this, but we should also have a good default. How to source README/LICENSE/RELEASES/CHANGELOG files in these situations is also more complicated.

My assumed default here is that you want one package per binary.

terminal autocomplete stuff..?

Some terminals/clis have explicit support for tab-completion integration. Is that something cargo-dist can/should help with in some way? Generating it? Packaging it? Installing it?

Out of scope for MVP I think.

Make the auto-generated release note tables/sections configurable

At very least you should be able to turn them off.

hint-only installers

It would be nice to have:

• --installer=cargo(-install?)
• --installer=cargo-binstall

Which produce installer hints without any need for any physical artifact. This requires some way to tell the reader of the dist-manifest that no such artifact exists, only the hint/description.

gitlab support

All the places we support github-specific things, ideally we could support gitlab equivalents.

Doing this would be a nice proof-of-concept of the internal architecture and the premise of "migrate to a new platform with just one command".

msi (un)installer support

This would be another potential killer app, but should be out of scope for the MVP.

Make cargo-dist-json-schema.json useful

• We should add a "$schema": "https://..." key to dist-manifest.json
• We should store a versioned copy of the schema somewhere more stable
  • Teach cargo-dist to have adhoc static artifacts and just upload it to releases?
  • This may have wonky behaviour for prereleases which might link the wrong/non-existent versions

freeform dist.rs file?

In the same vein as build.rs, but for dist tasks. This is always evil but it seems inevitable?

unhardcode compression schemes for executables

Currently the installer scripts (and probably some other stuff) hardcode .tar.xz in installer.sh and .zip in installer.ps1.

We should either make the installer scripts more intelligent and able to sniff the release and/or make the installer-script-generator more aware of the current config so it can bake the right values into the installer script.

generate ci subcommand

This can be cut from the milestone but I think it would be Killer if you could do something like:

cargo dist init github-ci

And it just creates .github/ci/cargo-dist.yml or whatever.

However this raises questions about the Scope of cargo-dist's functionality and what the action would cover:

• What would the action react to? (presumably: pushing a tag with vBlah ala)
• Would it create a github release? (https://github.com/taiki-e/create-gh-release-action)
• Would it push packages to a github release (or elsewhere?) (https://github.com/taiki-e/upload-rust-binary-action)

Or is this stuff more in scope for "the axo cli"?

One possible reason to want cargo-dist to be in the loop for ci declaration is for #19. One could image declaring "hey I want these targets built, your figure it out" and dist figures out the ideal matrix of platforms/cross-compilation to do this Best (building apple stuff on non-apple platforms is a pain in the ass that apple doesn't want you doing).

provide manual instructions in addition to cargo-release

i get a gnarly error cargo-installing cargo-release, to be on the safe side i think including manual docs would be a good call

nix support?

Nix is its own wild beast that really doesn't like the idea of a "general linux distro binary". If you try the binaries will just crash with "file not found". In reality you probably should just give up and generate a nix flake or whatever so people can build it themselves and use the nix binary cache.

The precise details of this are beyond me though I'm vaguely aware there's tools like cargo2nix.

("Just use musl" isn't an answer here, right?)

Allow user to pick packaging scheme

Currently we just use .zip when targetting windows and .tar.xz everywhere else, but we should allow the user to configure this in the settings.

https://github.com/taiki-e/upload-rust-binary-action has some examples for inspiration.

properly gathering info from subtasks?

When building things like the dist-manifest.json, we currently avoid getting any feedback from the actual subtasks that build the linux binaries. This is largely good and desirable, because we want the tool to know what should happen before it does it, and for every platform to be able to know what all the other platforms would do.

However some facts like build ids (for symbols/debuginfo/sourcemaps) and build environment info (cpu, os, toolchains...) aren't predictable and may want to be recorded. There are two ways to do this:

• Give each subtask its own artifact where it can just dump a bunch of "computed info" (messy, spammy)
• Have the dist-manifest task block on all other tasks and actually communicate with them (complicated, unreliable).

I hate both of these options, but we'll probably need to do this eventually!

readme for crates.io

currently the README for cargo-dist sits at the workspace root so that it is displayed in github, however, this prevents it from being displayed on crates.io as it is not in the default position (at the root of the crate project).

based on the cargo manifest docs, the readme path has to be relative to the project root, so i think we could fix this using ../README.md.

however we could also fix this by changing the organization from a "flat workspace" (workspace -> crates) to a "nested workspace" (app/workspace -> crates) where the cargo dist bin app sits on the top level and then has a nested workspace for the schema crate (and any other future workspace items).

curious what your pref is @Gankra - i think i sorta prefer the "nested workspace" because the workspace is in support of the main application, so it feels more semantically accurate.

don't mess up release note generation if there are multiple releases

really we just don't handle multiple releases properly at all, but I'm pretty sure something funky will happen to the release note generation code/scripts if there are multiple. Possibly they'll end up concatenated? Or one will get picked arbitrarily?

Ideally you workflow either looks like:

• I am cutting a unified release for everything in the workspace, merge the release notes harmoniously
• I am cutting N logically independent releases at once, create N github releases with their own ntoes
• I am cutting a release for only one the projects in the workspace

The last 2 may actually end up looking the same, as you may just push N commits with tags like "specific-app-vX.Y.Z". This kind of pattern isn't at all handled by the CI scripts though (but I'd like it to be!). But if we're adding that kind of complex parsing/detection of the git tag, I'd like that logic to somehow live in cargo-dist itself (no complex logic in CI scripts!!!).

Ability to specify feature flags / profile / etc

The code supports the notion but there's currently no way for the user to provide this information.

I think ideally for something like dist the user should prefer specifying them in the settings (#16) but they should also be allowed on the cli.

produce a machine-readable manifest

This will get fed into https://github.com/axodotdev/oranda for generating a page for the app, so it's very important.

• application name
• application version
• various things like repo/author/website
• list of packages (.zip)
  • features/variants, one may wish to have a "minimal" and "full" version of the app! (#22)
  • platform
    • OS
    • CPU
  • complete manifest of package contents..?
  • list of binaries? (.exes)
  • README text..?

add snapshot(?) tests for generate-ci

This probably necessitates adding a flag to specify what file it writes to, or to make it write to stdout.

cascading config

You should be able to specify things via cli, env, config file, ...

Probably something similar to what cargo-release supports: https://github.com/crate-ci/cargo-release/blob/master/docs/reference.md#configuration

macos .dmg/.app installer support?

This one is more gui app specific aiui so might not really make sense for what we care about?

Either way out of scope for MVP.

harden against shell injection attacks in the generated scripts

I'm like 80% sure we don't properly quote some things, like this $CHANGELOG_TITLE use is probably bad?

cargo-dist/.github/workflows/release.yml

Line 112 in 344c997

gh release edit ${{ needs.create-release.outputs.tag }} --title="$CHANGELOG_TITLE" --notes-file=new_dist_changelog.md

Not super concerning since these only run when a git tag is pushed to main but...

centralize git-tag computation

There's a ton of places where we compute "v{VERSION}" to use in things like URLs for fetching from github, but really we should be computing that in one place and passing it around so that in the future we can have more robust configuration of the pattern.

handle colliding targets properly

Under the current design, we:

1. Compute everything that needs to be done
2. Build all targets that need to be built
3. Copy artifacts from builds.
4. Bundle

I believe this will do the wrong thing if you told cargo-dist that you want builds that differ only in feature flags. i.e. if you have a "lightweight" version of your app and a "full" version that basically differs in --all-features being passed.

In this case we will successfully build both in step 2, but the final binary (and debuginfo) of the second one will overwrite the first, and so we'll use the same binary for both targets. To correctly handle this we need to change the build to be more responsive:

1. Compute everything that needs to be done
2. For each target (SERIAL)...
   a. Build
   b. Copy artifacts
3. Bundle

Note that this problem doesn't exist for --targets because each triple gets its own build sub-directory.

Ideally we would be able to detect when this would happen and still support non-conflicting build-parallelism, but tbh I'm not sure that cargo even would allow two copies of itself to run concurrently anyway. Not sure how granular its locking is.

Way to specify assets to include

Some files we should just auto-detect and add:

• README.*
• RELEASES.*
• CHANGELOG.*
• LICENSE-*

But also we should a way for the user to say "hey include these files/dirs" (possibly platform-specific). And to override auto-includes.

Splitting Debuginfo

Another future killer app: properly setting the flags and running the tools to produce proper symbol/debuginfo files (similar to source maps, but for native code).

This comment covers a lot of useful details on the state of this stuff in rust.

Some quick notes:

Each platform has different formats for this:

• windows pdb
• apple dsym
• linux dwp
• breakpad (platform-agnostic) sym

Different platforms have different symbol servers:

• windows symbol server (can fetch exe/dll/pdb)
  • breakpad generalizes this for .sym, see mozilla's tecken
• linux debuginfod
• apple: just ask xcode lolsob
• android: suck eggs

To generate symbol files for rust you want to:

• build a release binary
  • with opt-level=3 or w/e release is
  • with codegen-units=1
  • with LTO..?
• with debug info cranked to max
  • build with debug=2
  • enable split-debuginfo which is horribly platform specific
• if targetting breakpad: run dump_syms to get a .sym
• strip the binary?

Then the user will want to:

• grab the final debuginfo/symbol file
• get the unique build ids that map the debuginfo to the binary (platform-specific, hopefully embedded in the file, but we should still get them out?)
• upload it to a symbol server?
• package it?

github ci should support manual triggering

If you add workflow_dispatch as an triggering event, the user can manually trigger it with a nice builtin form. This would be great for debugging/testing without having to drown your repo in prerelease git tags!

https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow

Don't assume release notes are markdown?

Currently we'll detect and slurp up RELEASES.rst but then embed the contents in markdown. Oops!

I don't know if Github Releases supports not-markdown, or if we actually need to convert the user's notes to markdown.