metricmike/asdf-awscli

asdf-awscli's Issues

Add m1/apple silicon to testing matrix

Is your feature request related to a problem? Please describe.

We're not testing on M1s, which led to a lot of frustration and new issues during the first half of this year.

Describe the solution you'd like

GitHub just released M1 runners.
github/roadmap#528

We should add those to the os matrix in actions.

Use bundled releases for v2

Is your feature request related to a problem? Please describe.

asdf-awscli currently builds from source and requires an external Python.

awscli v2 bundles all dependencies, and provides packages for linux, macos, and windows. This should be the preferred method.

Describe the solution you'd like

asdf install awscli 2.x.y uses the prebuilt releases, and does not require an external Python for:

• linux (thanks @iamhsa!)
• macos (thanks @elementalvoid!)
• windows

See discussion at asdf-vm/asdf-plugins#222

Add unzip

To install applications ASDF needs unzip. Add unzip to the package list.

How to install plugins with pip?

With awscli v1 installed with pip, you can simply pip install the plugin.
With awscli v2 installed with homebrew, you can install a plugin with the bundled pip:

/usr/local/opt/awscli/libexec/bin/pip install awscli-plugin-endpoint

However, is it possible to install a plugin for awscli installed with this asdf plugin? I don't see any bundled pip.

MacOS: package expansion to the working directory can cause subsequent failures

Describe the bug

MacOS
In

asdf-awscli/lib/utils.bash

Line 71 in ec4ccdc

pkgutil --expand-full "${release_file}" ./AWSCLIV2 || fail "Could not extract ${release_file}"

package expansion is performed into the current working directory to a fixed directory.

Should a previous install have failed for any reason, as did on one of our systems, this directory is not cleaned up and pkgutil will refuse to expand the package into an existing directory with a slightly cryptic (but perfectly valid) error message:

Could not unarchive /Users/<elided>/.asdf/installs/awscli/2.4.6/awscli-2.4.6.pkg (The operation couldn't be completed. File exists)

The problem is the directory it is actually talking about is in the working directory where the package will be expanded, not in the installs subdir of asdf.

Suggestion - perhaps expand to a temp directory that is deleted via trap if there's a failure?

Steps to reproduce

1. in the current directory, create a directory called AWSCLIV2 - this is to simulate the failure of a prior installation
   mkdir AWSCLIV2
2. attempt to install using asdf (any v2 should have the same behaviour):
   ```asdf install awscli 2.4.6``

Expected behavior

Installs as expected

Screenshots

Errors as described above

Additional context
N/A

GitHub Token is sent to amazonaws.com on MacOS and Linux

Describe the bug

A single array CURL_OPTS is built and used for all curl operations and all URLs.

This means that even amazonaws.com receives that token, see macOS PKG download and Linux ZIP download.

On other platforms, such as Windows, there's no problem, since the source file is actually downloaded from GitHub, so putting the tokens in header is correct and also useful, to overcome API rate limits for anonymous API calls.

Steps to reproduce

• run asdf install awscli latest on macOS or on Linux
• as long as the download is still running, enter command ps auw | grep -i curl | grep -i amazonaws.com
• you should see the GH token as a curl parameter sent to amazonaws.com, in cleartext

Expected behavior

GitHub Token must be used only when calling GitHub APIs.

This could be fixed by moving the code which adds the GH token to CURL_OPTS in the download_release function before this line and working on a different, local variable.

Improve local development/testing workflow

Is your feature request related to a problem? Please describe.

Kinda stuck in error-driven development - make a change, commit, wait for Github Actions, guess at fix for whatever build is failing. Developing this plugin could be much easier!

Describe the solution you'd like

(split these into other tickets)

• Examples/workflow for a more robust debugger like BashDB
• Examples/workflow for making a change, and testing locally (point to git repo on disk instead of github? BATS?)
• Setup codespaces so a collaborator can just . on this page, apply fix, and publish a PR in ~10 minutes (arbitrary number) with nothing but a browser
• Document dependencies because it is so annoying to use something like long flags on uname and then have macos blow up in GithubActions (see: https://github.com/MetricMike/asdf-awscli/actions/runs/4794362683/jobs/8527665463#step:3:45 and asdf-vm/asdf#1435)
• Implement caching/workflow reuse to speed up GithubActions? asdf-vm/actions#235 asdf-vm/actions#442
• Hook into ASDF BATS to ensure we're keeping up with upstream's requirements/best practices? asdf-vm/actions#514 asdf-vm/asdf#1512
  • also asdf-vm/asdf#1318 https://github.com/bash-bastion and
• Implement all scripts (the helper/dependencies ones are what I'm most excited about) asdf-vm/asdf-plugin-template#61 https://asdf-vm.com/plugins/create.html#scripts-overview
• Implement benchmarking ala asdf-vm/asdf#290 http://stratus3d.com/blog/2022/08/11/asdf-performance/
• enforce semantic commits before GH Actions yell at me

Support more configuration options/levers when building from source

Is your feature request related to a problem? Please describe.

Plugin currently hard codes most options when building from source. We can be friendlier by exposing these options to the plugin users, a la https://github.com/asdf-vm/asdf-nodejs#configuration

Describe the solution you'd like
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-source-install.html#source-getting-started-install-config

• review all options available at ./configure --help
• --with-download-deps
  • some users (especially in an airgapped setting) may want to bring their dependencies separately. with-download-deps should probably be the default though
• --with-install-type=system-sandbox
  • this couples the installation to the python selected at build time, similar to the v1 bundled installers. It's super quick and lightweight, but will break often if the python updates frequently (like if it's the ASDF python). Maybe also include a warning if the python isn't "stable" / installed-from-yum/apt?

Out of scope, will not support options

• --prefix --libdir --bindir
  • ASDF requires these to be set to specific locations so it doesn't make sense to expose them

Bundled installs fail with coreutils `cp`

Describe the bug

If coreutils cp has been set up as the preferred command for cp, asdf installs fail. This maybe is the problem of the user who chose to do this, but possibly there's an agnostic use of cp that could work for both cases, or the script added in #21 could try to detect which cp is in use.

Steps to reproduce

❯ which -a cp
/usr/local/opt/coreutils/libexec/gnubin/cp
/bin/cp

❯ asdf install awscli 2.11.18
asdf-awscli: Expected /Users/me/.asdf/installs/awscli/2.11.18/bin/aws to be executable.
asdf-awscli: An error ocurred while installing awscli 2.11.18.

❯ export PATH=/bin:$PATH
❯ which -a cp
/bin/cp
/usr/local/opt/coreutils/libexec/gnubin/cp
/bin/cp

❯ asdf install awscli 2.11.18
mkdir: /Users/me/.asdf/downloads/awscli/2.11.18: File exists
asdf-awscli: asdf-awscli 2.11.18 installation was successful!

Expected behavior

Expected this to work.

Additional context

The issue is with different behaviour of cp -a. With coreutils, the directory b/ is preserved in the destination when doing a cp -a a/b/ c, with the system cp on MacOS, it is not. Example.

❯ ls -1 bar/foo/test.txt 
bar/foo/test.txt

❯ mkdir baz quux
❯ which -a cp
/usr/local/opt/coreutils/libexec/gnubin/cp
/bin/cp

❯ cp -a bar/foo/ baz
❯ /bin/cp -a bar/foo/ quux

❯ ls -1 baz/
foo
❯ ls -1 baz/foo/
test.txt

❯ ls -1 quux/
test.txt

Complete github community standards checklist

https://github.com/MetricMike/asdf-awscli/community

• Description
• Code of conduct (consider opening issue on https://github.com/asdf-vm/asdf-plugin-template/issues?q=code+of+conduct ?)
• Security policy (see: asdf-vm/asdf-plugin-template#3)
• Pull request template (https://github.com/MetricMike/asdf-awscli/blob/main/.github/PULL_REQUEST/pull_request_template.md doesn't count?)

asdf install awscli is assuming current working directory is $HOME

Describe the bug

I got this error when install awscli, here is the command and error:

jenkins@c236aff3753b:/$ asdf install awscli 2.7.24
checkdir:  cannot create extraction directory: ./AWSCLIV2
           Permission denied
asdf-awscli: Could not extract /home/jenkins/.asdf/installs/awscli/2.7.24/awscli-2.7.24.zip
asdf-awscli: An error ocurred while installing awscli 2.7.24.

Steps to reproduce

# install plugin.
asdf plugin add awscli
# go to a directory that the current user doesn't have permission.
cd /
# install awscli and error happen.
asdf install awscli 2.7.24

Expected behavior

We should not use current directory for extracting temporary files.

Improve security of downloading the binary by checking the fingerprint

Is your feature request related to a problem? Please describe.

As described on the asdf issues asdf-vm/asdf#1320 , it would be safer if the plugins would check a fingerprint for the binaries it downloads. Additionnally (for the specific case of awscli) gpg checking could be done.

The documentation

(for example for linux) gives a number of commands to check the downloaded binary https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

A quick search doesn't show me any sha256 fingerprints of the zip file, maybe you know where to find this.

Describe the solution you'd like

Add some checks in the bash scripts.

Describe alternatives you've considered

Have another plugin / program that checks after asdf has downloaded and installed things, but this is not ideal.

Additional context

This can be an important security risk for users how have corporate certificates installed on their work stations and thus https/ssl cannot but trusted.

Support for arm64 / apple m1 chips?

Is your feature request related to a problem? Please describe.

We would love to be able to use asdf to distribute specific versions of awscli to our users, however, we're now in a mix of architectures including the new m1 chips. I'd rather not have my users need a mix of homebrew and asdf to set up dependencies.

Describe the solution you'd like

Seamless support for m1/arm64 chips

Thanks for your consideration!

Add logging levels / hide pip messages

Is your feature request related to a problem? Please describe.

asdf install awscli X is loud and includes messages like:

~/.asdf/installs/awscli/2.1.25 ~/projects/dev-environment
Collecting pip
  Using cached pip-21.0.1-py3-none-any.whl (1.5 MB)
Collecting setuptools
  Using cached setuptools-53.0.0-py3-none-any.whl (784 kB)
Collecting wheel
  Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Installing collected packages: pip, setuptools, wheel
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3

Those should be hidden.

Describe the solution you'd like

Unless ASDF_AWSCLI_LOG=DEBUG is present, don't print those messages.

use `asdf` local dependencies when running install

Describe the bug

I tried to do an asdf install awscli latest and got an error relating to python not being installed.

I have a version of python installed via asdf install python latest I think asdf-awscli should be smart enough to use local dependencies when running an install.

Steps to reproduce

install local version of python into a folder
install awscli and watch it fail (must remove global versions of python)

Expected behavior

awscli should install

Screenshots

Additional context

Documentation to install lastest v1 not working

Describe the bug
The install of awscli v1 as describe in README.md, do not work and try to install v2

Steps to reproduce

Try to install the latest awscli v1 as describe in [README.md]

$ asdf install awscli latest 1 # 1.19.33
awscli 2.1.31 is already installed

Expected behavior

Installation of the latest version of awscli v1

Additional context
Test only on linux.
Install specific version is ok :

$ asdf install awscli 1.19.33
* Downloading awscli release 1.19.33...
~/.asdf/installs/awscli/1.19.33 ~/.asdf/plugins/awscli
Collecting pip
  [...]
awscli 1.19.33 installation was successful!