cashapp/hermit-packages

Hermit manages isolated, self-bootstrapping sets of tools in software projects.

hermit-packages's Issues

node corepack is broken (yarn, pnpm, ...)

The preferred way to install tools like yarn and pnpm is through NodeJS corepack command:

• https://yarnpkg.com/getting-started/install

The preferred way to manage Yarn is through Corepack, a new binary shipped with all Node.js releases starting from 16.10. It acts as an intermediary between you and Yarn, and lets you use different package manager versions across multiple projects without having to check-in the Yarn binary anymore.

For projects using hermit, this can be done via:

hermit install node
corepack enable

Which adds new symlinks to the ./bin directory, like yarn, yarnpkg, pnpm, and pnpx. And now I can just invoke them using yarn install as usual, which is great for local development!

However, these symlinks are added relative to my local Hermit folder (MacOS laptop):

ls -la ./bin/yarn
./bin/yarn -> ../../../Library/Caches/hermit/pkg/node-19.2.0/lib/node_modules/corepack/dist/yarn.js

Which means it is broken if I try to use it in CI (ubuntu), or check it into the repository so that other developers use it, since that path is only valid on my machine.

1. Is there an alternative (Hermit-blessed) way of installing tools managed by corepack?
2. If not, can these symlinks be corrected somehow to point to a portable path, that can work across different machines, and in CI?

rust packages broken by wrong digests

Several (all?) Rust projects are now suddenly broken with the following error:

fatal:hermit: https://static.rust-lang.org/dist/rust-1.64.0-x86_64-unknown-linux-gnu.tar.xz: https://static.rust-lang.org/dist/rust-1.64.0-x86_64-unknown-linux-gnu.tar.xz: checksum bd69e42f6cfe3ba96d781ad0b4095ddac4f0fc31c1af445018edf6f0aba543e4 should have been b51d1e2f752f58d02d781566f3e3ac4da0a666896ebba49d70f404dbaf312d7c

I traced the issue to 900d521 (cc @alecthomas). I think there is a bug in whichever import script it is using, which made it put the same digest in rust.hcl, for both apple-darwin.tar.xz and unknown-linux-gnu.tar.xz, for all versions.

To unblock these projects, I tried locking the hermit-packages source version, which failed as well. I filed a separate bug for: cashapp/hermit#342

So, for now, I manually forked rust.hcl (as a private package) to unblock these projects.

Support Node 22

It's been out for a few months, and is required for Wasm GC support.

`protoc-gen-connect-go` has moved

see https://github.com/connectrpc/connect-go/releases/tag/v1.11.0

One path to support would be just to update the org

• build: https://github.com/cashapp/hermit-build/blob/ffb06bd1f5f6e675d9afcc44823bc0702636db0f/go-tools/Makefile#L13
• package: https://github.com/cashapp/hermit-packages/blob/master/protoc-gen-connect-go.hcl

However, the code doesn't appear to be backwards incompatible

cannot use greeter (variable of type *GreetServer) as ballonv1connect.BalloonServiceHandler value in argument to ballonv1connect.NewBalloonServiceHandler: *GreetServer does not implement ballonv1connect.BalloonServiceHandler (wrong type for method Greet)
		have Greet(context.Context, *"connectrpc.com/connect".Request[ballonv1.GreetRequest]) (*"connectrpc.com/connect".Response[ballonv1.GreetResponse], error)
		want Greet(context.Context, *"github.com/bufbuild/connect-go".Request[ballonv1.GreetRequest]) (*"github.com/bufbuild/connect-go".Response[ballonv1.GreetResponse], error)compiler[InvalidIfaceAssign](https://pkg.go.dev/golang.org/x/tools/internal/typesinternal#InvalidIfaceAssign)

which is super fun to fix...

Run tests for only changed packages in hermit-packages PRs

Enable Folks to contribute

Context: https://cashapp.github.io/hermit/packaging/tutorial/

At this point you can (and should!) contribute the package back to the community via a PR.
Lies! When I tried to I received:

• remote: No anonymous write access (and then after logging in)
• remote: Permission to cashapp/hermit-packages.git denied to seniledeveloper. fatal: unable to access 'https://github.com/cashapp/hermit-packages/': The requested URL returned error: 403

Validate Auto-Version

Auto-Version does not validate that newly added versions can be installed and validated.
See: #388

It might be nice if the auto-version github workflow could catch errors like this, for example by attempting to install any auto-versioned packages.

ffmpeg package hasn't picked up 6.0.0

I was just looking into adding ffprobe to the ffmpeg package and I noticed it hasn't added the most recent 6.0 release:

https://github.com/eugeneware/ffmpeg-static/releases

I wonder if this is because it was 6.0 vs 6.0.0?

npx doesn't work

I am using macOS 13.1 on an m1 mac.

In any typical directory using system node I am able to use npx:

❯ npx rimraf --help
Need to install the following packages:
  [email protected]
Ok to proceed? (y) y
rimraf version 4.3.1

Usage: rimraf <path> [<path> ...]
Deletes all files and folders at "path", recursively.

Options:
  --                   Treat all subsequent arguments as paths
  -h --help            Display this usage info
  --preserve-root      Do not remove '/' recursively (default)
  --no-preserve-root   Do not treat '/' specially
  -G --no-glob         Treat arguments as literal paths, not globs (default)
  -g --glob            Treat arguments as glob patterns
  -v --verbose         Be verbose when deleting files, showing them as
                       they are removed. Not compatible with --impl=native
  -V --no-verbose      Be silent when deleting files, showing nothing as
                       they are removed (default)
  -i --interactive     Ask for confirmation before deleting anything
                       Not compatible with --impl=native
  -I --no-interactive  Do not ask for confirmation before deleting

  --impl=<type>        Specify the implementation to use:
                       rimraf: choose the best option (default)
                       native: the built-in implementation in Node.js
                       manual: the platform-specific JS implementation
                       posix: the Posix JS implementation
                       windows: the Windows JS implementation (falls back to
                                move-remove on ENOTEMPTY)
                       move-remove: a slow reliable Windows fallback

Implementation-specific options:
  --tmp=<path>        Temp file folder for 'move-remove' implementation
  --max-retries=<n>   maxRetries for 'native' and 'windows' implementations
  --retry-delay=<n>   retryDelay for 'native' implementation, default 100
  --backoff=<n>       Exponential backoff factor for retries (default: 1.2)

However, if I initialize a hermit project and install node, npx doesn't work:

~/Downloads
❯ mkcd hermit-npx-bug

~/Downloads/hermit-npx-bug
❯ hermit init
info: Creating new Hermit environment in /Users/knpwrs/Downloads/hermit-npx-bug
info:   -> /Users/knpwrs/Downloads/hermit-npx-bug/bin/README.hermit.md
info:   -> /Users/knpwrs/Downloads/hermit-npx-bug/bin/activate-hermit
info:   -> /Users/knpwrs/Downloads/hermit-npx-bug/bin/hermit
info:   -> /Users/knpwrs/Downloads/hermit-npx-bug/bin/hermit.hcl
info:

Hermit environment initialised in /Users/knpwrs/Downloads/hermit-npx-bug

To activate the environment run:

  . /Users/knpwrs/Downloads/hermit-npx-bug/bin/activate-hermit

Then run the following to list available commands:

  hermit --help

To deactivate the environment run:

  deactivate-hermit

For more information please refer to https://github.com/cashapp/hermit

~/Downloads/hermit-npx-bug
❯ . ./bin/activate-hermit
Hermit environment /Users/knpwrs/Downloads/hermit-npx-bug activated
hermit-npx-bug🐚

~/Downloads/hermit-npx-bug
❯ hermit install node
info:node-19.7.0:install: Installing node-19.7.0
hermit-npx-bug🐚

~/Downloads/hermit-npx-bug
❯ which npx
/Users/knpwrs/Downloads/hermit-npx-bug/bin/npx
hermit-npx-bug🐚

~/Downloads/hermit-npx-bug
❯ npx rimraf --help
npm notice
npm notice New minor version of npm available! 9.5.0 -> 9.6.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v9.6.0
npm notice Run npm install -g [email protected] to update!
npm notice
npm ERR! code ENOENT
npm ERR! syscall lstat
npm ERR! path /Users/knpwrs/Downloads/hermit-npx-bug/.hermit/node/lib
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, lstat '/Users/knpwrs/Downloads/hermit-npx-bug/.hermit/node/lib'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/knpwrs/Downloads/hermit-npx-bug/.hermit/node/cache/_logs/2023-03-08T15_10_58_975Z-debug-0.log
zsh: exit 254   npx rimraf --help
hermit-npx-bug🐚

This output in particular looks to be of interest:

npm ERR! code ENOENT
npm ERR! syscall lstat
npm ERR! path /Users/knpwrs/Downloads/hermit-npx-bug/.hermit/node/lib
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, lstat '/Users/knpwrs/Downloads/hermit-npx-bug/.hermit/node/lib'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

ruby uses homebrew's "package" of ruby which is not intended to be used in this fashion

Homebrew/homebrew-portable-ruby#185 (comment)

Adding new package to hermit - Crane cli tool

Hi everyone, I have a pr pending review to add a new package to hermit. Would appreciate a cr
#268

gcloud package forces Python2

The gcloud package force an environment variable to use Python2 causing the following error in systems that do not have python2

.cache/hermit/pkg/gcloud-352.0.0/bin/gcloud: 194: exec: python2: not found

Python 2 is out of support since Jan 1st 2020 so it doesn't have security updates because of that we have removed it from all Linux machines.

We have the CLOUDSDK_PYTHON=python3 exported on the shells but it is overwritten by the hermit value.

hermit test openjdk failing - Zulu OpenJDK no source provided for darwin-amd64 arch

Running hermit test openjdk produces:

info:validate: Validating openjdk-11.0.8_10
info:validate: Validating openjdk-11.0.9_11
info:validate: Validating openjdk-11.0.10_9
info:validate: Validating openjdk-11.0.11_9-zulu11.48.21
info:validate: Validating openjdk-17.0.1_12
info:validate: Validating openjdk-17.0.2_8
info:validate: Validating openjdk@stable
info:validate: Validating openjdk@zulu
info:validate: Validating openjdk@latest
info:validate: Validating [email protected]
info:validate: Validating openjdk@17
info:validate: Validating [email protected]
info:validate: Validating openjdk@11
warn: openjdk: darwin-amd64: https://github.com/cashapp/hermit-packages.git/openjdk.hcl: openjdk-11.0.11_9-zulu11.48.21: no source provided
warn: openjdk: darwin-amd64: https://github.com/cashapp/hermit-packages.git/openjdk.hcl: openjdk-11.0.11_9-zulu11.48.21@zulu: no source provided

Problems installing 'gcloud' components

This stinks of something wrong on my machine, but after wasting far too much time already I am turning here as some sort of sanity check. I am successfully able to install and use gcloud, but trying to install a component like gke-gcloud-auth-plugin using gcloud components install ... fails with a permission error:

$ gcloud components install gke-gcloud-auth-plugin

ERROR: (gcloud.components.install) You cannot perform this action because you do not have permission
to modify the Google Cloud SDK installation directory [/home/usrme/.cache/hermit/pkg/gcloud-405.0.1].

Re-run the command with sudo: sudo /home/usrme/.cache/hermit/pkg/gcloud-405.0.1/bin/gcloud ...

Looking at the listed directory I can see that the gcloud-405.0.1 directory does not have the correct permissions set despite other packages installed through Hermit having them:

$ ls -la /home/usrme/.cache/hermit/pkg
drwx------.  3 usrme usrme   40  6. okt   10:52 azcopy-10.16.1
drwx------.  2 usrme usrme   50  6. okt   13:20 dive-0.10.0
dr-x------.  9 usrme usrme 4096 17. okt   17:41 gcloud-405.0.1
...

I can, of course, adjust this manually (chmod -R u+rwx /home/usrme/.cache/hermit/pkg/gcloud-405.0.1) which then allows for the installation to finish successfully, but this still begs the question: why are the initial permissions different for gcloud?

Also, removing gcloud with hermit uninstall ... does not remove the underlying directory, which is understandable given the lack of write permissions.

Just in case:

• my umask is 0022
• it does not matter if I install an older version of gcloud as the permissions are still the same

Problem with gcloud and python3

Using the python3 and gcloud packages via Hermit, trying to run gcloud yields this error:

$ gcloud
ERROR: gcloud failed to load: module 'collections' has no attribute 'Mapping'
    gcloud_main = _import_gcloud_main()
    import googlecloudsdk.gcloud_main
    from googlecloudsdk.calliope import cli
    from googlecloudsdk.calliope import actions
    from googlecloudsdk.calliope import markdown
    from googlecloudsdk.calliope import usage_text
    from googlecloudsdk.calliope import parser_arguments
    from googlecloudsdk.calliope import parser_completer
    from googlecloudsdk.core.console import progress_tracker
    class _BaseStagedProgressTracker(collections.Mapping):

This usually indicates corruption in your gcloud installation or problems with your Python interpreter.

Please verify that the following is the path to a working Python 2.7 or 3.5+ executable:
    /path-to-dir-with-hermit-environment/bin/python3

If it is not, please set the CLOUDSDK_PYTHON environment variable to point to a working Python 2.7 or 3.5+ executable.

If you are still experiencing problems, please reinstall the Cloud SDK using the instructions here:
    https://cloud.google.com/sdk/

This is with gcloud 360.0.0 and python3 3.10.3

Add Maestro package

https://maestro.mobile.dev/getting-started/installing-maestro