aidan-/httpie-aws-authv4

AWS Auth v4 (API Gateway, Elasticsearch, etc) plugin for HTTPie

httpie-aws-authv4's Issues

TypeError: object supporting the buffer API required with multipart requests

First of all, thank you very much for this plugin! It works awesome, but not in the following use case:

When I'm trying to make a multipart/form-data request with http --multipart and AWS Auth, like this:

http --auth-type aws4 --auth REDACTED.execute-api.us-west-2.amazonaws.com \
     --multipart POST http://redacted.dpmain.tld/endpoint field=value file@/path/to/file

I'm getting following error:

http: error: TypeError: object supporting the buffer API required

Also this error is thrown when files are included in x-www-urlencoded requests with http --form url file@/path/to/file (in case of multipart requests it fails with any fields)

http --debug output with versions and traceback is here

HTTPie 2.3.0
Requests 2.25.1
Pygments 2.7.4
Python 3.8.2 (default, Oct 22 2020, 20:12:25) 
[GCC 5.5.0]
/home/envek/.asdf/installs/python/3.8.2/bin/python3.8
Linux 5.4.0-64-generic

<Environment {'colors': 256,
 'config': {'default_options': []},
 'config_dir': PosixPath('/home/envek/.config/httpie'),
 'devnull': <property object at 0x7f7e9cce4130>,
 'is_windows': False,
 'log_error': <function Environment.log_error at 0x7f7e9cce2820>,
 'program_name': 'http',
 'stderr': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>,
 'stderr_isatty': True,
 'stdin': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>,
 'stdin_encoding': 'utf-8',
 'stdin_isatty': True,
 'stdout': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>,
 'stdout_encoding': 'utf-8',
 'stdout_isatty': True}>

>>> requests.request(**{'auth': <httpie_aws_authv4.AWSAuth object at 0x7f7e9e7e52b0>,
 'data': <MultipartEncoder: <generator object MultiValueOrderedDict.items at 0x7f7e9c69c970>>,
 'headers': {'User-Agent': b'HTTPie/2.3.0', 'Content-Type': 'multipart/form-data; boundary=cb266ae3fb8a4cf89c8c9c8a40ba7f85'},
 'method': 'post',
 'params': <generator object MultiValueOrderedDict.items at 0x7f7e9c69c9e0>,
 'url': 'http://localhost:8080/fileAttachments'})


http: error: TypeError: object supporting the buffer API required

Traceback (most recent call last):
  File "/home/envek/.asdf/installs/python/3.8.2/bin/http", line 8, in <module>
    sys.exit(main())
  File "/home/envek/.asdf/installs/python/3.8.2/lib/python3.8/site-packages/httpie/__main__.py", line 11, in main
    exit_status = main()
  File "/home/envek/.asdf/installs/python/3.8.2/lib/python3.8/site-packages/httpie/core.py", line 79, in main
    exit_status = program(
  File "/home/envek/.asdf/installs/python/3.8.2/lib/python3.8/site-packages/httpie/core.py", line 196, in program
    for message in messages:
  File "/home/envek/.asdf/installs/python/3.8.2/lib/python3.8/site-packages/httpie/client.py", line 80, in collect_messages
    prepared_request = requests_session.prepare_request(request)
  File "/home/envek/.asdf/installs/python/3.8.2/lib/python3.8/site-packages/requests/sessions.py", line 456, in prepare_request
    p.prepare(
  File "/home/envek/.asdf/installs/python/3.8.2/lib/python3.8/site-packages/requests/models.py", line 320, in prepare
    self.prepare_auth(auth, url)
  File "/home/envek/.asdf/installs/python/3.8.2/lib/python3.8/site-packages/requests/models.py", line 551, in prepare_auth
    r = auth(self)
  File "/home/envek/.asdf/installs/python/3.8.2/lib/python3.8/site-packages/httpie_aws_authv4.py", line 60, in __call__
    return aws_request.__call__(r)
  File "/home/envek/.asdf/installs/python/3.8.2/lib/python3.8/site-packages/aws_requests_auth/aws_auth.py", line 76, in __call__
    aws_headers = self.get_aws_request_headers_handler(r)
  File "/home/envek/.asdf/installs/python/3.8.2/lib/python3.8/site-packages/aws_requests_auth/aws_auth.py", line 88, in get_aws_request_headers_handler
    return self.get_aws_request_headers(r=r,
  File "/home/envek/.asdf/installs/python/3.8.2/lib/python3.8/site-packages/aws_requests_auth/aws_auth.py", line 146, in get_aws_request_headers
    payload_hash = hashlib.sha256(body).hexdigest()
TypeError: object supporting the buffer API required

Specifying credentials profile not working for me

I am using and loving httpie-aws-authv4 and it works great using either credentials from the default profile without the --auth flag or explicitly specifying credentials using --auth ACCESSKEYXXX:AWSSECRETKEYXXX.

However, when I try using --auth profile:<profile_name>, it doesn't work and I get the error: "AWS was not able to validate the provided access credentials".

I have multiple profiles in my ~/.aws/credentials file and they all work using the aws CLI.

Example call that works:

$ http -f  -A aws4 ec2.us-east-1.amazonaws.com Action=DescribeVpcs Version=2015-10-01

Example call that fails:

$ http -f  -A aws4 -a profile:tleonhardt ec2.us-east-1.amazonaws.com Action=DescribeVpcs Version=2015-10-01

Installation breaks HTTPie: ModuleNotFoundError: No module named 'urllib3.exceptions'

httpie-aws-authv4 uninstalls urllib3 from /usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages/. Other plugins don't. See:

$ brew install httpie
Warning: Treating httpie as a formula. For the cask, use homebrew/cask/httpie or specify the `--cask` flag.
==> Downloading https://ghcr.io/v2/homebrew/core/httpie/manifests/3.2.2_4-1
Already downloaded: /Users/czeluk1/Library/Caches/Homebrew/downloads/8326a367c9785f69ce68be2ce71f72b4900f1c95783e03f78d95af5a295befc4--httpie-3.2.2_4-1.bottle_manifest.json
==> Fetching httpie
==> Downloading https://ghcr.io/v2/homebrew/core/httpie/blobs/sha256:86dce0667f668caba805363925b4a4e5a36ce6cfae95b95227c890bca4897e0c
Already downloaded: /Users/czeluk1/Library/Caches/Homebrew/downloads/bf37360a12228f4fcf8ded877915e556783a0aa6d35a6022cc4da461da7c085e--httpie--3.2.2_4.sonoma.bottle.1.tar.gz
==> Pouring httpie--3.2.2_4.sonoma.bottle.1.tar.gz
🍺  /usr/local/Cellar/httpie/3.2.2_4: 1,076 files, 9.7MB
==> Running `brew cleanup httpie`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

$ httpie
usage: httpie [-h] [--debug] [--traceback] [--version] {cli,plugins} ...
httpie: error: Please specify one of these: 'cli', 'plugins'

This command is only for managing HTTPie plugins.
To send a request, please use the http/https commands:

  $ http POST pie.dev/post hello=world

  $ https POST pie.dev/post hello=world

$ httpie cli plugins install httpie-aws-authv4
Installing httpie-aws-authv4...
Collecting httpie-aws-authv4
  Using cached httpie_aws_authv4-0.3.0-py3-none-any.whl.metadata (5.3 kB)
Requirement already satisfied: httpie>=1.0.0 in /usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages (from httpie-aws-authv4) (3.2.2)
Collecting aws-requests-auth>=0.4.0 (from httpie-aws-authv4)
  Using cached aws_requests_auth-0.4.3-py2.py3-none-any.whl.metadata (567 bytes)
Collecting boto3>=1.9.0 (from httpie-aws-authv4)
  Using cached boto3-1.34.71-py3-none-any.whl.metadata (6.6 kB)
Requirement already satisfied: urllib3 in /usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages (from httpie-aws-authv4) (2.2.0)
Requirement already satisfied: requests>=0.14.0 in /usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages (from aws-requests-auth>=0.4.0->httpie-aws-authv4) (2.31.0)
Collecting botocore<1.35.0,>=1.34.71 (from boto3>=1.9.0->httpie-aws-authv4)
  Using cached botocore-1.34.71-py3-none-any.whl.metadata (5.7 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from boto3>=1.9.0->httpie-aws-authv4)
  Using cached jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting s3transfer<0.11.0,>=0.10.0 (from boto3>=1.9.0->httpie-aws-authv4)
  Using cached s3transfer-0.10.1-py3-none-any.whl.metadata (1.7 kB)
Requirement already satisfied: pip in /usr/local/lib/python3.12/site-packages (from httpie>=1.0.0->httpie-aws-authv4) (24.0)
Requirement already satisfied: charset-normalizer>=2.0.0 in /usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages (from httpie>=1.0.0->httpie-aws-authv4) (3.3.2)
Requirement already satisfied: defusedxml>=0.6.0 in /usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages (from httpie>=1.0.0->httpie-aws-authv4) (0.7.1)
Requirement already satisfied: Pygments>=2.5.2 in /usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages (from httpie>=1.0.0->httpie-aws-authv4) (2.17.2)
Requirement already satisfied: requests-toolbelt>=0.9.1 in /usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages (from httpie>=1.0.0->httpie-aws-authv4) (1.0.0)
Requirement already satisfied: multidict>=4.7.0 in /usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages (from httpie>=1.0.0->httpie-aws-authv4) (6.0.5)
Requirement already satisfied: setuptools in /usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages (from httpie>=1.0.0->httpie-aws-authv4) (69.1.0)
Requirement already satisfied: rich>=9.10.0 in /usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages (from httpie>=1.0.0->httpie-aws-authv4) (13.7.0)
Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.35.0,>=1.34.71->boto3>=1.9.0->httpie-aws-authv4)
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting urllib3 (from httpie-aws-authv4)
  Using cached urllib3-2.2.1-py3-none-any.whl.metadata (6.4 kB)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages (from requests>=0.14.0->aws-requests-auth>=0.4.0->httpie-aws-authv4) (3.6)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/opt/python-certifi/lib/python3.12/site-packages (from requests>=0.14.0->aws-requests-auth>=0.4.0->httpie-aws-authv4) (2024.2.2)
Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages (from requests[socks]>=2.22.0->httpie>=1.0.0->httpie-aws-authv4) (1.7.1)
Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages (from rich>=9.10.0->httpie>=1.0.0->httpie-aws-authv4) (3.0.0)
Requirement already satisfied: mdurl~=0.1 in /usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages (from markdown-it-py>=2.2.0->rich>=9.10.0->httpie>=1.0.0->httpie-aws-authv4) (0.1.2)
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.35.0,>=1.34.71->boto3>=1.9.0->httpie-aws-authv4)
  Using cached six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Using cached httpie_aws_authv4-0.3.0-py3-none-any.whl (6.0 kB)
Using cached aws_requests_auth-0.4.3-py2.py3-none-any.whl (6.8 kB)
Using cached boto3-1.34.71-py3-none-any.whl (139 kB)
Using cached botocore-1.34.71-py3-none-any.whl (12.0 MB)
Using cached urllib3-2.2.1-py3-none-any.whl (121 kB)
Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
Using cached s3transfer-0.10.1-py3-none-any.whl (82 kB)
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: urllib3, six, jmespath, python-dateutil, botocore, aws-requests-auth, s3transfer, boto3, httpie-aws-authv4
  Attempting uninstall: urllib3
    Found existing installation: urllib3 2.2.0
    Uninstalling urllib3-2.2.0:
      Successfully uninstalled urllib3-2.2.0
Successfully installed aws-requests-auth-0.4.3 boto3-1.34.71 botocore-1.34.71 httpie-aws-authv4-0.3.0 jmespath-1.0.1 python-dateutil-2.9.0.post0 s3transfer-0.10.1 six-1.16.0 urllib3-2.2.1

$ httpie
Traceback (most recent call last):
  File "/usr/local/bin/httpie", line 5, in <module>
    from httpie.manager.__main__ import main
  File "/usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages/httpie/manager/__main__.py", line 6, in <module>
    from httpie.context import Environment
  File "/usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages/httpie/context.py", line 20, in <module>
    from .utils import repr_dict
  File "/usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages/httpie/utils.py", line 19, in <module>
    import requests.auth
  File "/usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages/requests/__init__.py", line 45, in <module>
    from .exceptions import RequestsDependencyWarning
  File "/usr/local/Cellar/httpie/3.2.2_4/libexec/lib/python3.12/site-packages/requests/exceptions.py", line 7, in <module>
    from urllib3.exceptions import HTTPError as BaseHTTPError
ModuleNotFoundError: No module named 'urllib3.exceptions'

Only works with some services

I was able to successfully use this plugin with S3, but was unable to get it to work correctly with EC2. Whenever I tried to use it with an EC2 endpoint, all I was able to get were return code 500 (Internal Server Error).

Here is an example:

11:25 $ http -A aws4 https://ec2.us-east-1.amazonaws.com Action=DescribeVpcs
HTTP/1.1 500 Internal Server Error
Connection: close
Content-Language: en
Content-Length: 1003
Content-Type: text/html;charset=utf-8
Date: Sat, 16 Mar 2019 15:29:04 GMT
Server: AmazonEC2

<html><head><title>Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Encountered an Internal Error</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>Encountered an Internal Error</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><HR size="1" noshade="noshade"></body></html>

The server is there and if I don't pass it an Action , I get a bad request complaining about no action being supplied:

11:29 $ http -A aws4 https://ec2.us-east-1.amazonaws.com
HTTP/1.1 400 Bad Request
Connection: close
Date: Sat, 16 Mar 2019 15:29:50 GMT
Server: AmazonEC2
Transfer-Encoding: chunked

<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>MissingAction</Code><Message>No action was supplied with this request.</Message></Error></Errors><RequestID>a76908bb-f2b7-4f2a-9790-08707e9af301</RequestID></Response>

httpie-aws-authv4-0.1.1 Isn't compatible with Python 3.6

httpie-aws-authv4-0.1.1 fail with python 3.6

Traceback (most recent call last):
File "c:\applications\python\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
(......)
module = import(self.module_name, fromlist=['name'], level=0)
File "c:\applications\python\lib\site-packages\httpie_aws_authv4.py", line 43
print "ERROR: Could not parse neccessary information from URL."

A patch is available on #1, could you release a new version on PIP ?

Start cutting releases?

Hey,

I know that this is a small project but it's possible to start cutting releases and tags here on GitHub.

The reason for this request is because I want to package this httpie plugin to Debian/Ubuntu and having releases and tags helps me when writing the package configuration.

Cheers!

hotfix - support AWS China regions

Hello, I am using this plugin to execute some queries on AWS elasticsearch. Just bumped into issue, that httpie-aws-authv4 could not determine AWS region when being assumed to aws-cn partition

$ http -A aws4 https://XXXXXXXXXXXXXXX.cn-northwest-1.es.amazonaws.com.cn/_snapshot/_all                                            
ERROR: Could not parse neccessary information from URL.

http: error: ValueError: Could not determine AWS region or service from domain name.

aws-cli requires AWS_DEFAULT_REGION env variable to be set, maybe this is a solution?

AWS STS support?

I used httpie aws auth4 successfully with my access key & secret.
I tried to use it also with the access key & secret returned by AWS STS service. But then this fails.
Is it supposed to fail?

(I've a feeling one should also use the session token returned by the STS service).

It seems David Muller's implementation has support for the session token in one of the examples in the README.md

Thanks

httpie-aws-authv4-0.1.3 fails to pip install on Python 2.7

httpie-aws-authv4-0.1.3 fails to pip install on Python 2.7 with the following error:

Collecting httpie-aws-authv4
  Downloading https://files.pythonhosted.org/packages/cb/60/22fc40c01915d6d9debf0179aebeb4e189300475f351e34c678d16b2301c/httpie-aws-authv4-0.1.3.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/fb/hljqhyrn053_rb1ggp48frw80000gn/T/pip-install-97JIhr/httpie-aws-authv4/setup.py", line 6, in <module>
        with open('README.md', 'rt', encoding='utf8') as f:
    TypeError: 'encoding' is an invalid keyword argument for this function

It appears to install correctly on Python 3.7.

TypeError: object supporting the buffer API required with BufferedReader requests

Very similar to Issue 10, we are seeing an issue when POSTing some piped content, e.g. via here file:

http -A aws4 POST $AWS_ES_URL/some_index/doc/TEST:1 <<<'{"SOME":"text"}'
...
http: error: TypeError: object supporting the buffer API required

In this case, the body appears to be a BufferedReader, with BufferedIOBase superclass defining the read() method.
Materializing the reader using read() works around that problem, at the cost of loading the body into memory.