Giter Site home page Giter Site logo

aws-sigv4-proxy's Introduction

AWS SigV4 Proxy

The AWS SigV4 Proxy will sign incoming HTTP requests and forward them to the host specified in the Host header.

You can strip out arbirtary headers from the incoming request by using the -s option.

Getting Started

Build and run the Proxy

The proxy uses the default AWS SDK for Go credential search path:

* Environment variables.
* Shared credentials file.
* IAM role for Amazon EC2 or ECS task role

More information can be found in the [developer guide](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html)

docker build -t aws-sigv4-proxy .

# Env vars
docker run --rm -ti \
  -e 'AWS_ACCESS_KEY_ID=<YOUR ACCESS KEY ID>' \
  -e 'AWS_SECRET_ACCESS_KEY=<YOUR SECRET ACCESS KEY>' \
  -p 8080:8080 \
  aws-sigv4-proxy -v

# Shared Credentials
docker run --rm -ti \
  -v ~/.aws:/root/.aws \
  -p 8080:8080 \
  -e 'AWS_SDK_LOAD_CONFIG=true' \
  -e 'AWS_PROFILE=<SOME PROFILE>' \
  aws-sigv4-proxy -v

Configuration

When running the Proxy, the following flags can be used (none are required) : s", "

Flag (or short form) Type Description Default
verbose or v Boolean Enable additional logging, implies all the log-* options False
log-failed-requests Boolean Log 4xx and 5xx response body False
log-signing-process Boolean Log sigv4 signing process False
unsigned-payload Boolean Prevent signing of the payload" False
port String Port to serve http on 8080
strip or s String Headers to strip from incoming request None
custom-headers String Comma-separated list of custom headers in key=value format None
duplicate-headers String Duplicate headers to an X-Original- prefix name None
role-arn String Amazon Resource Name (ARN) of the role to assume None
name String AWS Service to sign for None
sign-host String Host to sign for None
host String Host to proxy to None
region String AWS region to sign for None
upstream-url-scheme String Protocol to proxy with https
no-verify-ssl Boolean Disable peer SSL certificate validation False
transport.idle-conn-timeout Duration Idle timeout to the upstream service 40s

Examples

S3

# us-east-1
curl -s -H 'host: s3.amazonaws.com' http://localhost:8080/<BUCKET_NAME>

# other region
curl -s -H 'host: s3.<BUCKET_REGION>.amazonaws.com' http://localhost:8080/<BUCKET_NAME>

SQS

curl -s -H 'host: sqs.<AWS_REGION>.amazonaws.com' 'http://localhost:8080/<AWS_ACCOUNT_ID>/<QUEUE_NAME>?Action=SendMessage&MessageBody=example'

API Gateway

curl -H 'host: <REST_API_ID>.execute-api.<AWS_REGION>.amazonaws.com' http://localhost:8080/<STAGE>/<PATH>

Running the service and stripping out sigv2 authorization headers

docker run --rm -ti \
  -v ~/.aws:/root/.aws \
  -p 8080:8080 \
  -e 'AWS_SDK_LOAD_CONFIG=true' \
  -e 'AWS_PROFILE=<SOME PROFILE>' \
  aws-sigv4-proxy -v -s Authorization

Running the service and preserving the original Authorization header as X-Original-Authorization (useful because Authorization header will be overwritten.)

docker run --rm -ti \
  -v ~/.aws:/root/.aws \
  -p 8080:8080 \
  -e 'AWS_SDK_LOAD_CONFIG=true' \
  -e 'AWS_PROFILE=<SOME PROFILE>' \
  aws-sigv4-proxy -v --duplicate-headers Authorization

Running the service with Assume Role to use temporary credentials

docker run --rm -ti \
  -v ~/.aws:/root/.aws \
  -p 8080:8080 \
  -e 'AWS_SDK_LOAD_CONFIG=true' \
  -e 'AWS_PROFILE=<SOME PROFILE>' \
  aws-sigv4-proxy -v --role-arn <ARN OF ROLE TO ASSUME>

Include service name & region overrides when you notice errors like unable to determine service from host for API gateway, for example.

docker run --rm -ti \
  -v ~/.aws:/root/.aws \
  -p 8080:8080 \
  -e 'AWS_SDK_LOAD_CONFIG=true' \
  -e 'AWS_PROFILE=<SOME PROFILE>' \
  aws-sigv4-proxy -v --name execute-api --region us-east-1

OpenSearch

  • Access AWS OpenSearch domain, hosted in private subnet of AWS VPC, with access policy restricted to IAM role.

    Prepare connection (assume role, export AWS_PROFILE, run ssh tunnel):

    aws sts assume-role \
     --role-arn "arn:aws:iam::123456789012:role/example-role" \
     --role-session-name role-profile
    export AWS_PROFILE=role-profile
    
    ssh \
     -4 \
     -o BatchMode="yes" \
     -o StrictHostKeyChecking="no" \
     -o ProxyCommand="aws ssm start-session --target %h --region eu-west-1 --document-name AWS-StartSSHSession --parameters portNumber=%p" \
     -i /Users/user/.ssh/id_rsa ubuntu@i-bastion-host \
     -L 4443:vpc-private-domain-name.eu-west-1.es.amazonaws.com:443 \
     -N

    Finally, run proxy:

    docker run --rm -ti \
         -v ~/.aws:/root/.aws \
         --network=bridge \
         -p 8080:8080 \
         -e "AWS_SDK_LOAD_CONFIG=true" \
         -e "AWS_PROFILE=role-profile" \
         aws-sigv4-proxy \
                 --verbose --log-failed-requests --log-signing-process --no-verify-ssl \
                 --name es --region eu-west-1 \
                 --host host.docker.internal:4443 \
                 --sign-host eu-west-1.es.amazonaws.com

    Access dashboard via http://localhost:8080/_dashboards/app/home#/tutorial_directory

Reference

License

This library is licensed under the Apache 2.0 License.

aws-sigv4-proxy's People

Contributors

a7i avatar alvinlin123 avatar anand99 avatar brianbianco avatar chadbean avatar danielblando avatar dekimsey avatar denish-m avatar dependabot[bot] avatar eqemccudden avatar jafferli avatar jfautley avatar kagahehk01 avatar luisgerhorst avatar njegosrailic avatar pduleba avatar prudhvigodithi avatar rapphil avatar rohang98 avatar roystchiang avatar sheetaljoshi avatar tantona avatar vijayansarathy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aws-sigv4-proxy's Issues

Question: AWS Elastic Search Example

Hi Team,
I have a EKS cluster with fluentd deamonsets to collect logs, all these logs are sending to elastic search cluster.

Natively fluentd doesnt have support for AWS ElasticSearch and I need move from elasticsearch to AWS Elastic Search.

Can you provide an example how to configure this proxy with AWS ElasticSearch ?

Thanks in advance.

Support ALB health check endpoint

I'm trying to run the aws-sigv4-proxy behind an ALB as an ECS Fargate service and when the load balancer hits the service it connects to HTTP / and the proxy returns a 502 Bad Gateway. ALB only supports HTTP status codes 200-499 so there's no way for me to add a health check to the proxy for "signs of life".

Could we add a /aws-sigv4-proxy-healthcheck endpoint or something like that that returns 200 if the proxy is up and listening?

Support TCP proxy

There are some teams out there that have a custom implementation of the proxy to support other protocols, like TCP.

For example, OTEL has an internal fork of the proxy that uses TCP, see open-telemetry/opentelemetry-collector-contrib#774.

It'd be nice if this repo could support TCP (and other protocols) and then in the configurations, specify which protocol to use. For this specific issue, I'd like TCP to be supported.

Make idle connection timeout adjustable

Some user were seeing error 10.0.0.0:1234->10.0.0.1:5678: write: broken pipe. The issue seems to be the idle connection timeout configured in the client is longer than the configuration on the server and RST from the server is lost from time to time.

We should make idle connection timeout configurable so user's log can be cleaner. Also I propose we also se the default idle connection time out to something lower than 60s because that is the default for AWS ALB, and a lot of AWS probably use ALB under the hood.

Issue with chunk-signature when load testing with minio warp

Version of sigv4 proxy is 1.6.

We're currently load testing throughput capabilities of the proxy and ran into this issue whilst testing with minio warp.

warp: <ERROR> unexpected download size. want:10485760, got:10500246                                                                                                                       
warp: <ERROR> unexpected download size. want:10485760, got:10500246                                                                                                                       
warp: <ERROR> unexpected download size. want:10485760, got:10500246  

Looks like chunk-signature is ending up in the file:

10000;chunk-signature=7cc8934a07a23a0b069244e0f418202c055291dffeb9cf8bfa3b92df546b4aeb

If we go direct to the S3 endpoint we don't get this issue so aws-sigv4-proxy seems to be uploading with that chunk-signature in the file when using the tool?

Seems like this has been an issue before with other "servers". See: minio/warp#164

Ive also raised an issue here: minio/warp#232

Is it possible to use it as an HTTPs Proxy?

Hi Folks,

I'm evaluating the aws-sigv4-proxy to access some opensearch instances and for that I made the following setup:

docker-compose.yaml

version: '3'

services:
  proxy:
    image: public.ecr.aws/aws-observability/aws-sigv4-proxy:1.2
    environment:
      AWS_ACCESS_KEY_ID:
      AWS_SECRET_ACCESS_KEY:
      AWS_DEFAULT_REGION: us-east-1
      AWS_SESSION_TOKEN:
    command:
      - --host=s3.amazonaws.com
    ports:
      - 8080:8080

After that, I just ran the curl -s -H 'host: s3.amazonaws.com' http://localhost:8080/ and it worked flawlessly! Nice!

But, when I try to use it with awscli for example by setting HTTPS_PROXY variable, the following error occurs:

image

Based on PR #16, I tried to configure name and host as well, but still I got no luck.

Is there a way to make it work using HTTPS_PROXY env var?

Can'b build image on master branch.

I am trying to build a docker image and have following error:

#5 [build  2/11] RUN yum -y update && rm -rf /var/cache/yum/*
#5 sha256:a18e915006ef0c9b3ca52110f578e7d6a2824fae1220d77ed1b98e97047de39f
#5 0.286 Loaded plugins: ovl, priorities
#5 1.897 https://cdn.amazonlinux.com/2/core/2.0/x86_64/0c4b5094bba8d46b07c60e3d85cd8baac5f75d07af6a33086b6d0cd9eb2e13f1/repodata/repomd.xml?instance_id=URLError&region=unknown: [Errno 14] curl#60 - "SSL certificate problem: unable to get local issuer certificate"
#5 1.898 Trying other mirror.
#5 1.898
#5 1.898
#5 1.898  One of the configured repositories failed (Amazon Linux 2 core repository),
#5 1.898  and yum doesn't have enough cached data to continue. At this point the only
#5 1.898  safe thing yum can do is fail. There are a few ways to work "fix" this:
#5 1.898
#5 1.898      1. Contact the upstream for the repository and get them to fix the problem.
#5 1.898
#5 1.898      2. Reconfigure the baseurl/etc. for the repository, to point to a working
#5 1.898         upstream. This is most often useful if you are using a newer
#5 1.898         distribution release than is supported by the repository (and the
#5 1.898         packages for the previous distribution release still work).
#5 1.898
#5 1.898      3. Run the command with the repository temporarily disabled
#5 1.898             yum --disablerepo=amzn2-core ...
#5 1.898
#5 1.898      4. Disable the repository permanently, so yum won't use it by default. Yum
#5 1.898         will then just ignore the repository until you permanently enable it
#5 1.898         again or use --enablerepo for temporary usage:
#5 1.898
#5 1.898             yum-config-manager --disable amzn2-core
#5 1.898         or
#5 1.898             subscription-manager repos --disable=amzn2-core
#5 1.898
#5 1.898      5. Configure the failing repository to be skipped, if it is unavailable.
#5 1.898         Note that yum will try to contact the repo. when it runs most commands,
#5 1.898         so will have to try and fail each time (and thus. yum will be be much
#5 1.898         slower). If it is a very temporary problem though, this is often a nice
#5 1.898         compromise:
#5 1.898
#5 1.898             yum-config-manager --save --setopt=amzn2-core.skip_if_unavailable=true
#5 1.898
#5 1.898 failure: repodata/repomd.xml from amzn2-core: [Errno 256] No more mirrors to try.
#5 1.898 https://cdn.amazonlinux.com/2/core/2.0/x86_64/0c4b5094bba8d46b07c60e3d85cd8baac5f75d07af6a33086b6d0cd9eb2e13f1/repodata/repomd.xml?instance_id=URLError&region=unknown: [Errno 14] curl#60 - "SSL certificate problem: unable to get local issuer certificate"
#5 ERROR: executor failed running [/bin/sh -c yum -y update && rm -rf /var/cache/yum/*]: exit code: 1
------
 > [build  2/11] RUN yum -y update && rm -rf /var/cache/yum/*:
------
executor failed running [/bin/sh -c yum -y update && rm -rf /var/cache/yum/*]: exit code: 1

Getting incomplete signature exception

Trying with host: *.vpc-lattice-svcs.us-west-2.on. and name vpc-lattice-svcs and I get error IncompleteSignatureException: Missing the 'x-amz-content-sha256' in canonical headers. Can you please help.

ARM builds

ARM64 is popping up more frequently including the M1 Macs. More AWS services are offering a Graviton2 option.

Please provide an ARM64 build. I did a quick test and the proxy compiles ok for arm64.

Support for VPCE?

Hi!

While the proxy does a great job to access 'regular' API gateways, it fails with VPCEs as in aws.go around line 26 there is no lookup defined for vpce.amazonaws.com:

	for region := range endpoints.AwsPartition().Regions() {
		host := fmt.Sprintf("execute-api.%s.amazonaws.com", region)
		services[host] = endpoints.ResolvedEndpoint{URL: fmt.Sprintf("https://%s", host), SigningMethod: "v4", SigningRegion: region, SigningName: "execute-api"}

		// something missing here for VPCE?

Request is sent to RDS endpoint, but signed by docdb.

I have set up sig-v4-proxy with docker-compose.

version: "3"

services:
  sig-v4-proxy:
    image: public.ecr.aws/aws-observability/aws-sigv4-proxy:1.5
    volumes:
      - ./.aws:/root/.aws
    ports:
      - "8080:8080"
    environment:
      - AWS_SDK_LOAD_CONFIG=true
    command: ['-v']

I request rds.ap-northeast-1.amazonaws.com, but error due to signing by docdb service
(Sometimes it would be Neptune.)

curl -s -H 'host: rds.ap-northeast-1.amazonaws.com' 'http://localhost:8080/?Action=DescribeDBInstances&Version=2014-10-31'
<ErrorResponse xmlns="http://rds.amazonaws.com/doc/2014-10-31/">
  <Error>
    <Type>Sender</Type>
    <Code>SignatureDoesNotMatch</Code>
    <Message>Credential should be scoped to correct service: 'rds'. </Message>
  </Error>
  <RequestId>2c8dcc75-87c9-421a-8c5d-4818489f5d0f</RequestId>
</ErrorResponse>

Log

2205-sig-v4-proxy-1  | time="2022-05-15T13:58:38Z" level=debug msg="Initial request dump:" request="GET /?Action=DescribeDBInstances&Version=2014-10-31 HTTP/1.1\r\nHost: rds.ap-northeast-1.amazonaws.com\r\nAccept: */*\r\nUser-Agent: curl/7.68.0\r\n\r\n"
2205-sig-v4-proxy-1  | time="2022-05-15T13:58:38Z" level=info msg="DEBUG: Request Signature:\n---[ CANONICAL STRING  ]-----------------------------\nGET\n/\nAction=DescribeDBInstances&Version=2014-10-31\nhost:rds.ap-northeast-1.amazonaws.com\nx-amz-date:20220515T135838Z\n\nhost;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\n---[ STRING TO SIGN ]--------------------------------\nAWS4-HMAC-SHA256\n20220515T135838Z\n20220515/ap-northeast-1/docdb/aws4_request\n44422fd5808526e3aa2b9afc325b6b21c4cdcec2ce713f3144291252639facf9\n-----------------------------------------------------"
2205-sig-v4-proxy-1  | time="2022-05-15T13:58:38Z" level=debug msg="signed request" region=ap-northeast-1 service=docdb
2205-sig-v4-proxy-1  | time="2022-05-15T13:58:38Z" level=debug msg="proxying request" request="GET /?Action=DescribeDBInstances&Version=2014-10-31 HTTP/1.1\r\nHost: rds.ap-northeast-1.amazonaws.com\r\nAccept: */*\r\nAuthorization: AWS4-HMAC-SHA256 Credential=AKIA3SSOAQIFEARL35XE/20220515/ap-northeast-1/docdb/aws4_request, SignedHeaders=host;x-amz-date, Signature=785a279627141c870d8d6db25ab3a4d67f1191949ba067cb1f78187a2fa01d5c\r\nUser-Agent: curl/7.68.0\r\nX-Amz-Date: 20220515T135838Z\r\n\r\n"
2205-sig-v4-proxy-1  | time="2022-05-15T13:58:39Z" level=error msg="error proxying request" message="<ErrorResponse xmlns=\"http://rds.amazonaws.com/doc/2014-10-31/\">\n  <Error>\n    <Type>Sender</Type>\n    <Code>SignatureDoesNotMatch</Code>\n    <Message>Credential should be scoped to correct service: 'rds'. </Message>\n  </Error>\n  <RequestId>a717e22d-0016-4c7c-b1c2-cebbb01ff9b3</RequestId>\n</ErrorResponse>\n" request="GET https://rds.ap-northeast-1.amazonaws.com/?Action=DescribeDBInstances&Version=2014-10-31" status_code=403

Request to ec2 endpoint, work fine.

curl -s -H 'host: ec2.ap-northeast-1.amazonaws.com' 'http://localhost:8080/?Action=DescribeInstances&Version=2016-11-15'

On Upload Object Fail

Hi all,

When I try to upload an object I get the following error:
    <?xml version=\"1.0\" encoding=\"UTF-8\"?>
    <Error>
        <Code>AccessDenied</Code>
        <Message>There were headers present in the request which were not signed</Message>
        <HeadersNotSigned>content-md5</HeadersNotSigned>
        <RequestId>E6ZWRP6AH7X7KW1K</RequestId>
        <HostId>zKDm2Xhp9D3kyUlDYq/YjLFNFYIElsDwOrCuKVs23jH8t/bZlR+vlpDwhvL3J2HMdNei3WoCKSY=</HostId>
    </Error>

I already tried to add the headers before and after the signature, without success

You can support me because this happens I will appreciate it very much, this is the fragment of the log when the call is made:

time="2021-08-16T19:26:17Z" level=info msg="Stripping headers []" StripHeaders="[]"
time="2021-08-16T19:26:17Z" level=info msg="Listening on :8080" port=":8080"
time="2021-08-16T19:26:30Z" level=debug msg="Initial request dump:" request="PUT /edgwork-my-bucket/edgwork-ac-upload.txt HTTP/1.1\r\nHost: localhost:8080\r\nAccept-Encoding: identity\r\nAuthorization: AWS4-HMAC-SHA256 Credential=ASIAVLCN4DG7MYCHQTG5/20210816/us-east-1/s3/aws4_request, SignedHeaders=content-md5;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=5349eaa0fb81aefae7e1c774da8762dab01deb4839c1b5c46d468b875ac3b248\r\nContent-Length: 161\r\nContent-Md5: BEfrRIi1xGJS1dGfkeBzkA==\r\nContent-Type: text/plain\r\nExpect: 100-continue\r\nUser-Agent: aws-cli/1.18.147 Python/2.7.18 Linux/4.14.238-182.422.amzn2.x86_64 botocore/1.18.6\r\nX-Amz-Content-Sha256: 369d88d309b4d1184784ac0f60490faf168e75a2747095e0bfc74f307f43b8c6\r\nX-Amz-Date: 20210816T192630Z\r\nX-Amz-Security-Token: IQoJb3JpZ2luX2VjEDwaCXVzLWVhc3QtMSJHMEUCIQC9EyElXnmyk/UxC7R++d/v6XfG3fsk6U+nkIe+LzM+lAIgHuBYJF/MJPQQ9HEuWI9pCPA6TyAA2VOom7Mqa5N3B/Aq+gMIZBAAGgwzNjczODMwMjYxMTAiDHC4yMKBE1SN56ff2yrXAyluPo4aSYuza77stxvUOJssXDHZJq96KnFJXJbPzmCIkwxgmV/HFdwrkAE5/hmFe7J7HGEdrctokXsUfbRr+/ifos41gY1dLr1ZSg0fZ5kKWK9CgQNYGKiyXwwlTsWQATIag2T5VvV1r7i88Ek64sx+cdBB/E7Vfrmos6yZvneeUiWVu6R81OB0UlozdsdG9bxenN0lxyrH3MAVQW5UsKGCDQ3i89JjMl3KDOgJTNOI6kslaBxgm+zpb8pCRcUdsCjL7ZxCLLGTww0HXAMDCVqszHHwaTWA5RXHsY4/DHSngAEiqwYl9dpQb1LMrK9kcjLLKLNvVz3FX2Wxc5x/96VJdF2ORjQ7emT3J8cZqPzAbcv22TP4rBJnmxy18//brz4jkRikNuB/PzgSRLBQ3XqXBzJ1uLnXAySX9gy6xN84OZjt4OAADqb+CUSx2EKQl4XblQvQSfILGPC1j4tO2e1g3Bb5tflpbAonjuy7YAUEz98HCRo2N3YrkL+4XvqtIOk0PNQo3ikLLjyCyiC2NrqpP7BiF0loqXpI1983f+1BM2sRZ3dCqt2koP+51TQIEqLJ34ghNSGOqWv7MjExBUqCvDbZghmFTBMx/M9PfDbskqBgsxgMPTDI8eqIBjqlARcK2wPRKIqIW10qJ8daq6ZxE94xVDYtLV304qyPE9CVaqFNdWO7hfxPL+JmwktIVYpE8efJy8tIog7/W7F5wxH1KKddV//VoTy8h/AHnI8nTxDq/PXcU2tKBhX8sTmZtWvVs1TRhVaNj0XpANptYQ6zEN+2CnAsqhmiEg6V3xAGFugbcZEpGJ0vEf9JH9OjhMjwT8yNTqPq14HChTe83euhqPp9aA==\r\n\r\n0 Hola Edgwork!\n1 Hola Edgwork!\n2 Hola Edgwork!\n3 Hola Edgwork!\n4 Hola Edgwork!\n5 Hola Edgwork!\n6 Hola Edgwork!\n7 Hola Edgwork!\n8 Hola Edgwork!\n9 Hola Edgwork!\n\n"
time="2021-08-16T19:26:30Z" level=debug msg="signed request" region=us-east-1 service=s3
time="2021-08-16T19:26:30Z" level=debug msg="proxying request" request="PUT /edgwork-my-bucket/edgwork-ac-upload.txt HTTP/1.1\r\nHost: s3.amazonaws.com\r\nAccept-Encoding: identity\r\nAuthorization: AWS4-HMAC-SHA256 Credential=ASIAVLCN4DG7MYCHQTG5/20210816/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=6c0423f517c7667ed7a2badbd6af05ffe7e887f870d130e8211ea3bbbe13f912\r\nContent-Length: 161\r\nContent-Md5: BEfrRIi1xGJS1dGfkeBzkA==\r\nContent-Type: text/plain\r\nExpect: 100-continue\r\nUser-Agent: aws-cli/1.18.147 Python/2.7.18 Linux/4.14.238-182.422.amzn2.x86_64 botocore/1.18.6\r\nX-Amz-Content-Sha256: 369d88d309b4d1184784ac0f60490faf168e75a2747095e0bfc74f307f43b8c6\r\nX-Amz-Date: 20210816T192630Z\r\nX-Amz-Security-Token: IQoJb3JpZ2luX2VjEDwaCXVzLWVhc3QtMSJHMEUCIQC9EyElXnmyk/UxC7R++d/v6XfG3fsk6U+nkIe+LzM+lAIgHuBYJF/MJPQQ9HEuWI9pCPA6TyAA2VOom7Mqa5N3B/Aq+gMIZBAAGgwzNjczODMwMjYxMTAiDHC4yMKBE1SN56ff2yrXAyluPo4aSYuza77stxvUOJssXDHZJq96KnFJXJbPzmCIkwxgmV/HFdwrkAE5/hmFe7J7HGEdrctokXsUfbRr+/ifos41gY1dLr1ZSg0fZ5kKWK9CgQNYGKiyXwwlTsWQATIag2T5VvV1r7i88Ek64sx+cdBB/E7Vfrmos6yZvneeUiWVu6R81OB0UlozdsdG9bxenN0lxyrH3MAVQW5UsKGCDQ3i89JjMl3KDOgJTNOI6kslaBxgm+zpb8pCRcUdsCjL7ZxCLLGTww0HXAMDCVqszHHwaTWA5RXHsY4/DHSngAEiqwYl9dpQb1LMrK9kcjLLKLNvVz3FX2Wxc5x/96VJdF2ORjQ7emT3J8cZqPzAbcv22TP4rBJnmxy18//brz4jkRikNuB/PzgSRLBQ3XqXBzJ1uLnXAySX9gy6xN84OZjt4OAADqb+CUSx2EKQl4XblQvQSfILGPC1j4tO2e1g3Bb5tflpbAonjuy7YAUEz98HCRo2N3YrkL+4XvqtIOk0PNQo3ikLLjyCyiC2NrqpP7BiF0loqXpI1983f+1BM2sRZ3dCqt2koP+51TQIEqLJ34ghNSGOqWv7MjExBUqCvDbZghmFTBMx/M9PfDbskqBgsxgMPTDI8eqIBjqlARcK2wPRKIqIW10qJ8daq6ZxE94xVDYtLV304qyPE9CVaqFNdWO7hfxPL+JmwktIVYpE8efJy8tIog7/W7F5wxH1KKddV//VoTy8h/AHnI8nTxDq/PXcU2tKBhX8sTmZtWvVs1TRhVaNj0XpANptYQ6zEN+2CnAsqhmiEg6V3xAGFugbcZEpGJ0vEf9JH9OjhMjwT8yNTqPq14HChTe83euhqPp9aA==\r\n\r\n0 Hola Edgwork!\n1 Hola Edgwork!\n2 Hola Edgwork!\n3 Hola Edgwork!\n4 Hola Edgwork!\n5 Hola Edgwork!\n6 Hola Edgwork!\n7 Hola Edgwork!\n8 Hola Edgwork!\n9 Hola Edgwork!\n\n"
time="2021-08-16T19:26:30Z" level=error msg="error proxying request" message="<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>There were headers present in the request which were not signed</Message><HeadersNotSigned>content-md5</HeadersNotSigned><RequestId>E6ZWRP6AH7X7KW1K</RequestId><HostId>zKDm2Xhp9D3kyUlDYq/YjLFNFYIElsDwOrCuKVs23jH8t/bZlR+vlpDwhvL3J2HMdNei3WoCKSY=</HostId></Error>"

Greetings

Name prefix "aws-sigv4-proxy" to the prometheus server pod causing failure when remotewrite to AMP

Setup:
kube-prometheus-stack, prometheus-operator, aws-sigv4-proxy for remoteWrite to AMP.

When using the kube-prometheus-stack and deploying the prometheus server with the below overrides

kube-prometheus-stack:
enabled: true
prometheus:
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam:::role/EKS-AMP-ServiceAccount-Role
prometheusSpec:
containers:
- name: aws-sigv4-proxy-sidecar
image: .dkr.ecr.us-east-1.amazonaws.com/aws-observability/aws-sigv4-proxy:1.0
args:
- --name
- aps
- --region
- us-east-1
- --role-arn
- arn:aws:iam:::role/EKS-AMP-Central-Role
- --host
- aps-workspaces.us-east-1.amazonaws.com
- --port
- :8005
ports:
- name: aws-sigv4-proxy
containerPort: 8005
remoteWrite:
- url: http://localhost:8005/workspaces//api/v1/remote_write

Hitting an issue where the prefix "aws-sigv4-proxy" is adding to the already existing calculated length of kube-prometheus-stack and causing the roleArn to go over 64 chars and failing.

ts=2022-02-22T10:59:04.797Z caller=dedupe.go:112 component=remote level=warn remote_name=ec6d05 url=http://localhost:8005/workspaces//api/v1/remote_write msg="Failed to send batch, retrying" err="server returned HTTP status 502 Bad Gateway: unable to proxy request - ValidationError: 1 validation error detected: Value 'aws-sigv4-proxy-prometheus-AAA-monitoring-kube-promet-prometheus-0' at 'roleSessionName' failed to satisfy constraint: Member must have length less than or equal to 64"

Here the roleSessionName - aws-sigv4-proxy-prometheus-AAA-monitoring-kube-promet-prometheus-0

"AAA-monitoring" is the release name.
"prometheus-AAA-monitoring-kube-promet-prometheus-0" - server pod name

If I shorten the release name, the template code in kube-prometheus-stack adds characters to pod name based on the logic it has, so that doesnt help.

Is there a significant reason to have the hardcoded prefix (aws-sigv4-proxy) ?
Anyway to configure or override the prefix?

Support for EKS Pod IDentity

I would like to use Sigv4 proxy, but not using IRSA, which is already working, but relying on new Pod Identity.

Seems it needs an upgrade for the SDK to let it work natively

CVE-2020-8911 and CVE-2020-8912

When I run trivy on this repo, it reports two vulnerabilities in the aws-sdk-go module. When I upgrade go.mod to point to the latest release of aws-sdk-go (1.44.221 at the time of this writing), I get the same two vulnerabilities:

Library Vulnerability Severity Installed Version Fixed Version Title
github.com/aws/aws-sdk-go CVE-2020-8911 MEDIUM 1.44.221 aws/aw-sdk-go: CBC padding oracle issue in AWS S3 Crypto (SDK for golang)
github.com/aws/aws-sdk-go CVE-2020-8912 LOW 1.44.221 aws-sdk-go: In-band key negotiation issue in AWS S3 Crypto (SDK for golang)

Getting "Signature doesn't match error" if the folder or file name contains special character ":"

Names of folders and files in our s3 buckets have timestamps suffixes. For example, s3://dev-moderation.dev.viooh.net.cn/2023-10-25/14:12:16.980077/54df88c2-35a8-4e55-a464-29aa0e5

I am able to download the files using the proxy if they don't have :. But not able to download the files with the timestamps.

Example:-

Downloading the file with aws s3 cp command:-

root@ip-10-30-11-182:~# aws s3 cp s3://dev-moderation.dev.viooh.net.cn/test-file-31-10-2023/test-file-11:23/testfile testfile
download: s3://dev-moderation.dev.viooh.net.cn/test-file-31-10-2023/test-file-11:23/testfile to ./testfile

But when the HTTP request is made using the proxy :-

root@ip-10-30-11-182:~# curl -H 'host: s3.cn-northwest-1.amazonaws.com.cn' 'http://localhost:8080/dev-moderation.dev.viooh.net.cn/test-file-31-10-2023/test-file-11:23/testfile'
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>ASIA4UMEKOSNWSYVO4CM</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
20231122T083730Z
20231122/cn-northwest-1/s3/aws4_request
a51eb9eb19a9a41b62d8a334a1861e4e31043661c0147bc042214d1469f0c0a1</StringToSign><SignatureProvided>03c59f7d44cfaa82023fe8bebba609fc0b52c40ceef80507c9b50c849283afc1</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 33 31 31 32 32 54 30 38 33 37 33 30 5a 0a 32 30 32 33 31 31 32 32 2f 63 6e 2d 6e 6f 72 74 68 77 65 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 61 35 31 65 62 39 65 62 31 39 61 39 61 34 31 62 36 32 64 38 61 33 33 34 61 31 38 36 31 65 34 65 33 31 30 34 33 36 36 31 63 30 31 34 37 62 63 30 34 32 32 31 34 64 31 34 36 39 66 30 63 30 61 31</StringToSignBytes><CanonicalRequest>GET
/dev-moderation.dev.viooh.net.cn/test-file-31-10-2023/test-file-11%3A23/testfile

host:s3.cn-northwest-1.amazonaws.com.cn
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20231122T083730Z
x-amz-security-token:FwoDYXdzEAEaDHRQSAYiecr5h5NGsiLoBJk10x0FsINcgUgkDe3lklJ5H/rsvPo3z94ACzbDsH5rkQuCx18T/JUeMvJUVvoF/K6JA9XP2VhKPcTdVrOQTBVOEWk8AT9OiMtHDGbYqIyppTKdtaOVwXpLuq+E8A6XkHJbv9vNtN1aHW0ffF5H+RlYL/Qq92p7qnhUoIfAy0mAC0XWgRGay5zHs4SzfyLVMAvmimfcbVESy9P6pWUWCB6ib6B6eKP3j2VC0iLLU9AKkyTC+zc+IDNH3gt8fWDxpPTYSIbtdviQIh70ldAKLwP73l1yZ+R2q0BgHpXQ9mVSlOg+mo/h9ChBOVwDGQa6oDHSdxCQ30WDcchvDKufspvm6lpAc9HCqaGCiCKXYIx/0JLm3+mILmVI/R5Oh54FfAthZCHyrf78Pg8ENpwBOg9Vxqz3S7cfQL03MYdNDu2NL+muVh/rAAmr5H5GNdGySDqe3fgmeuKMx6yg9mR8nX2m2/1heZlYLMH5AEr/3Nxuu3cPEZ8iSz5Ae/nDwI8Ime6fFw799cBtvzTAaZ5C+Wecg69CaeMpcUrGHeZ4e6DMNB+S5a52j/B10O6TZwTgCn35lupIVObSLj8OqKYRNxq19grNVIYDxA7xHCaF2OazmKvCDqhje3rUPo/ufURtET1Yhok9ml5cLYEyAN/siOqibk9JYwAsWlHhAfcVoMYjzOn48P9j+GfjKFUHTFQnmqoFUYpfX+1JOjPvFw997+wNuRnHyQOosc2dZ16wkD1fYQsRIGdfKQcPvyLKVwAGVJrEH6DbLJFnR2bbcp7DBlzDToRN6y8jI/CDpbhH3Lg4Q76Lh4q1ROIoqfb2qgYyQUi6MW9KimqF5IetSUgn4h/fE1FznlkGwtlXj+jpzrYtzxeWH8ie/YxQLErp5rhueapSC+CY/gwf417h37Lg4Ddj

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</CanonicalRequest><CanonicalRequestBytes>47 45 54 0a 2f 64 65 76 2d 6d 6f 64 65 72 61 74 69 6f 6e 2e 64 65 76 2e 76 69 6f 6f 68 2e 6e 65 74 2e 63 6e 2f 74 65 73 74 2d 66 69 6c 65 2d 33 31 2d 31 30 2d 32 30 32 33 2f 74 65 73 74 2d 66 69 6c 65 2d 31 31 25 33 41 32 33 2f 74 65 73 74 66 69 6c 65 0a 0a 68 6f 73 74 3a 73 33 2e 63 6e 2d 6e 6f 72 74 68 77 65 73 74 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 2e 63 6e 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 32 33 31 31 32 32 54 30 38 33 37 33 30 5a 0a 78 2d 61 6d 7a 2d 73 65 63 75 72 69 74 79 2d 74 6f 6b 65 6e 3a 46 77 6f 44 59 58 64 7a 45 41 45 61 44 48 52 51 53 41 59 69 65 63 72 35 68 35 4e 47 73 69 4c 6f 42 4a 6b 31 30 78 30 46 73 49 4e 63 67 55 67 6b 44 65 33 6c 6b 6c 4a 35 48 2f 72 73 76 50 6f 33 7a 39 34 41 43 7a 62 44 73 48 35 72 6b 51 75 43 78 31 38 54 2f 4a 55 65 4d 76 4a 55 56 76 6f 46 2f 4b 36 4a 41 39 58 50 32 56 68 4b 50 63 54 64 56 72 4f 51 54 42 56 4f 45 57 6b 38 41 54 39 4f 69 4d 74 48 44 47 62 59 71 49 79 70 70 54 4b 64 74 61 4f 56 77 58 70 4c 75 71 2b 45 38 41 36 58 6b 48 4a 62 76 39 76 4e 74 4e 31 61 48 57 30 66 66 46 35 48 2b 52 6c 59 4c 2f 51 71 39 32 70 37 71 6e 68 55 6f 49 66 41 79 30 6d 41 43 30 58 57 67 52 47 61 79 35 7a 48 73 34 53 7a 66 79 4c 56 4d 41 76 6d 69 6d 66 63 62 56 45 53 79 39 50 36 70 57 55 57 43 42 36 69 62 36 42 36 65 4b 50 33 6a 32 56 43 30 69 4c 4c 55 39 41 4b 6b 79 54 43 2b 7a 63 2b 49 44 4e 48 33 67 74 38 66 57 44 78 70 50 54 59 53 49 62 74 64 76 69 51 49 68 37 30 6c 64 41 4b 4c 77 50 37 33 6c 31 79 5a 2b 52 32 71 30 42 67 48 70 58 51 39 6d 56 53 6c 4f 67 2b 6d 6f 2f 68 39 43 68 42 4f 56 77 44 47 51 61 36 6f 44 48 53 64 78 43 51 33 30 57 44 63 63 68 76 44 4b 75 66 73 70 76 6d 36 6c 70 41 63 39 48 43 71 61 47 43 69 43 4b 58 59 49 78 2f 30 4a 4c 6d 33 2b 6d 49 4c 6d 56 49 2f 52 35 4f 68 35 34 46 66 41 74 68 5a 43 48 79 72 66 37 38 50 67 38 45 4e 70 77 42 4f 67 39 56 78 71 7a 33 53 37 63 66 51 4c 30 33 4d 59 64 4e 44 75 32 4e 4c 2b 6d 75 56 68 2f 72 41 41 6d 72 35 48 35 47 4e 64 47 79 53 44 71 65 33 66 67 6d 65 75 4b 4d 78 36 79 67 39 6d 52 38 6e 58 32 6d 32 2f 31 68 65 5a 6c 59 4c 4d 48 35 41 45 72 2f 33 4e 78 75 75 33 63 50 45 5a 38 69 53 7a 35 41 65 2f 6e 44 77 49 38 49 6d 65 36 66 46 77 37 39 39 63 42 74 76 7a 54 41 61 5a 35 43 2b 57 65 63 67 36 39 43 61 65 4d 70 63 55 72 47 48 65 5a 34 65 36 44 4d 4e 42 2b 53 35 61 35 32 6a 2f 42 31 30 4f 36 54 5a 77 54 67 43 6e 33 35 6c 75 70 49 56 4f 62 53 4c 6a 38 4f 71 4b 59 52 4e 78 71 31 39 67 72 4e 56 49 59 44 78 41 37 78 48 43 61 46 32 4f 61 7a 6d 4b 76 43 44 71 68 6a 65 33 72 55 50 6f 2f 75 66 55 52 74 45 54 31 59 68 6f 6b 39 6d 6c 35 63 4c 59 45 79 41 4e 2f 73 69 4f 71 69 62 6b 39 4a 59 77 41 73 57 6c 48 68 41 66 63 56 6f 4d 59 6a 7a 4f 6e 34 38 50 39 6a 2b 47 66 6a 4b 46 55 48 54 46 51 6e 6d 71 6f 46 55 59 70 66 58 2b 31 4a 4f 6a 50 76 46 77 39 39 37 2b 77 4e 75 52 6e 48 79 51 4f 6f 73 63 32 64 5a 31 36 77 6b 44 31 66 59 51 73 52 49 47 64 66 4b 51 63 50 76 79 4c 4b 56 77 41 47 56 4a 72 45 48 36 44 62 4c 4a 46 6e 52 32 62 62 63 70 37 44 42 6c 7a 44 54 6f 52 4e 36 79 38 6a 49 2f 43 44 70 62 68 48 33 4c 67 34 51 37 36 4c 68 34 71 31 52 4f 49 6f 71 66 62 32 71 67 59 79 51 55 69 36 4d 57 39 4b 69 6d 71 46 35 49 65 74 53 55 67 6e 34 68 2f 66 45 31 46 7a 6e 6c 6b 47 77 74 6c 58 6a 2b 6a 70 7a 72 59 74 7a 78 65 57 48 38 69 65 2f 59 78 51 4c 45 72 70 35 72 68 75 65 61 70 53 43 2b 43 59 2f 67 77 66 34 31 37 68 33 37 4c 67 34 44 64 6a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 3b 78 2d 61 6d 7a 2d 73 65 63 75 72 69 74 79 2d 74 6f 6b 65 6e 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35</CanonicalRequestBytes><RequestId>9DXNPP7VHGZNXGQA</RequestId><HostId>sQ1DKWl7iFcL3b+U0WbGiuchoKRB/6SRUBXje+T93FTUHdZ9D/kdNvz0FlGwCzOPGBlmiuRiwQA=</HostId></Error>

But I am able to download other files without having ":" in their names using the proxy :-

root@ip-10-30-11-182:~# curl -v -H 'host: s3.cn-northwest-1.amazonaws.com.cn' 'http://localhost:8080/dev-moderation.dev.viooh.net.cn/test-file-31-10-2023/testfile'
*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /dev-moderation.dev.viooh.net.cn/test-file-31-10-2023/testfile HTTP/1.1
> Host: s3.cn-northwest-1.amazonaws.com.cn
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 19
< Content-Type: binary/octet-stream
< Date: Wed, 22 Nov 2023 08:35:57 GMT
< Etag: "32068f20af575b3703c4cfcb2078dfe5"
< Last-Modified: Tue, 31 Oct 2023 07:14:05 GMT
< Server: AmazonS3
< X-Amz-Id-2: 5jtlLE79R3+dm9DF6Ax8+1aisA98qWUHEZwHg8dE2mMnDY5VuGtsY9VRqz3OUuG9TuZK6Kf/Y5g=
< X-Amz-Request-Id: JBG5W7DQ2505X3WA
< X-Amz-Server-Side-Encryption: AES256
< X-Amz-Tagging-Count: 1
< 
This is test file 

HTTP body is emptied for chunked requests

if proxyReq.ContentLength == 0 {

This removes the body if the content length is zero. However, for requests with Transfer-Encoding: chunked (happens with gRPC) the content length might not be set (i.e., zero) and therefore this removes the existing body (thereby corrupting the request). Shouldn't this also check if the request is already chunked and not remove the body if it is?

Add extra headers into the envelope

Is there a way to add an extra header into the envelope to be signed?

Selling Partner API uses x-amz-access-token header, which is also required to be signed.

https://developer-docs.amazon.com/amazon-shipping/docs/connecting-to-the-selling-partner-api#authorization-header

Authorization: AWS4-HMAC-SHA256
Credential=AKIAIHV6HIXXXXXXX/20201022/us-east-1/execute-api/aws4_request
SignedHeaders=host;user-agent;x-amz-access-token;x-amz-date
Signature=5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924aEXAMPLE

(I split it into lines for better readability)

elastic search: request signature we calculated does not match the signature you provided

Example of logs we got:

time="2022-08-11T11:14:37Z" level=debug msg="Initial request dump:" request="GET /// HTTP/1.1\r\nHost: es\r\nAccept-Encoding: gzip\r\nContent-Type: application/json\r\nUser-Agent: Go-http-client/1.1\r\n\r\n"
time="2022-08-11T11:14:37Z" level=info msg="DEBUG: Request Signature:\n---[ CANONICAL STRING ]-----------------------------\nGET\n///\n\nhost:##url##\nx-amz-date:20220811T111437Z\nx-amz-security-token:##token##\n\nhost;x-amz-date;x-amz-security-token\n##Token##\n---[ STRING TO SIGN ]--------------------------------\nAWS4-HMAC-SHA256\n20220811T111437Z\n20220811/ap-south-1/es/aws4_request\n33f78ea95c2ddcfdd4012d605b9319cbeccf397ac351e39ad4b5d8b6e85e47bc\n-----------------------------------------------------"
time="2022-08-11T11:14:37Z" level=debug msg="signed request" region=ap-south-1 service=es
time="2022-08-11T11:14:37Z" level=debug msg="proxying request" request="GET /// HTTP/1.1\r\nHost:##url##\r\nAccept-Encoding: gzip\r\nAuthorization: AWS4-HMAC-SHA256 Credential=ASIAY63QBOGXGV7L7YVF/20220811/ap-south-1/es/aws4_request, SignedHeaders=host;x-amz-date;x-amz-security-token, Signature=55aea73d7d56110bcd420a946c324f360a0516dd8f54daf5090045ce4e85c6c2\r\nContent-Type: application/json\r\nUser-Agent: Go-http-client/1.1\r\nX-Amz-Date: 20220811T111437Z\r\nX-Amz-Security-Token: ##token##\r\n\r\n"
time="2022-08-11T11:14:37Z" level=error msg="error proxying request" message="{\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\"}" request="GET https://##url##///" status_code=403

I can see the initial request is like

GET /// HTTP/1.1
Host: es
Accept-Encoding: gzip
Content-Type: application/json
User-Agent: Go-http-client/1.1

Given stackover question, #15, #13, I belive it's an issue with some extra "///" in the url or some redirecting issue.

Can someone help, Im not able to send post request to AWS certificate manager.

Hi, all I just started working with amazon and I want automate some thing for which I need to make api calls to AWS ACM to issue certificate, list, get etc.. I found aws-sigv4-proxy a solution for my workflow which will add aws signature to my calls but Im not able make the call to acm using post request as I am new to this tool.

Can some one provide examples with curl cmd to send request to acm api, just like we have some examples in this repo read me file related to SQS, S3... I am trying to use this api: https://docs.aws.amazon.com/acm/latest/APIReference/Welcome.htmlhttps://docs.aws.amazon.com/acm/latest/APIReference/Welcome.html

I will really appreciate all the help.

Documentation improvement suggestions

I am trying to run aws-sigv4-proxy in front of AWS-hosted prometheus (aps). After a lot of experimentation, I found something that works; it would be great if the documentation were clearer about the places where I went wrong.

  • The signature mechanism appears not to support session tokens. I took the documentation that suggested mounting ~/.aws into the container and setting AWS_PROFILE to mean that the proxy was using the standard AWS tool chain for interacting with credentials. Since our setup leans heavily on role assumption and SSO, the path of least resistance (and best security) involves setting AWS_SESSION_TOKEN and this does not work. I only made progress once I created an ad hoc AWS user in the target account and created a temporary access token.
  • The proxy does not do anything smart with HTTP headers provided by the user agent. In my case, I was using a CLI http client that set Accept, Accept-Heading, Connection, Host, and User-Agent. My signature failed until I explicitly skipped all of these headers. (It's possible that I only needed to skip a subset... I don't know the signature protocol very well, which is why I was trying to use the proxy in the first place)

In any case, my working configuration ended up using:

docker run --rm -it \
  -p 8080:8080 \
  --env-file .env \
  public.ecr.aws/aws-observability/aws-sigv4-proxy \
  --region us-west-2 \
  --name aps \
  --host aps-workspaces.us-west-2.amazonaws.com \
  --strip Accept \
  --strip Accept-Encoding \
  --strip Connection \
  --strip Host \
  --strip User-Agent \
  --verbose --log-failed-requests --log-signing-process

where .env contained:

AWS_ACCESS_KEY_ID=<REDACTED>
AWS_SECRET_ACCESS_KEY=<REDACTED>

Preserving Authorization header

This proxy sets the Authorization header but if the original request also had an Authorization header, then it will be overwritten. I'm using this to front a private Kubernetes (EKS) endpoint with API Gateway (using IAM authorization) and kubectl passes the bearer token in the Authorization header and I need a way to preserve that original Authorization header so it can make its way to EKS.

aws-sigv4-proxy sidecar running OOM on EKS EC2

I'm running an aws-sigv4-proxy sidecar along with opencost.io to proxy Prometheus metrics from an AMP workspace endpoint.

I've granted the aws-sigv4-proxy container an obscene level of resources, and it's continuing to burn through them and run OOM as though no GC is active.

I'm running this pod on EKS EC2 workers provisioned with Karpenter.

apiVersion: v1
kind: Pod
metadata:
  namespace: opencost
  labels:
    app: opencost
spec:
  containers:
  - env:
    - name: CLUSTER_ID
      value: <cluster id>
    - name: PROMETHEUS_SERVER_ENDPOINT
      value: http://localhost:8080
    image: gcr.io/kubecost1/opencost
    imagePullPolicy: Always
    name: opencost
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
  - args:
    - --verbose
    - --name
    - aps
    - --region
    - us-east-1
    - --host
    - localhost:8080
    - --sign-host
    - https://aps-workspaces.us-east-1.amazonaws.com/workspaces/<workspace id>/
    - --upstream-url-scheme
    - http
    image: public.ecr.aws/aws-observability/aws-sigv4-proxy
    imagePullPolicy: Always
    name: aws-sigv4-proxy
    resources:
      limits:
        cpu: "3"
        memory: 5Gi
      requests:
        cpu: "3"
        memory: 5Gi
  restartPolicy: Always
  serviceAccount: opencost

Support unsigned payload

Hi there,

We have a use case that we may send large request payload, and we'd like to leverage the unsigned payload option to avoid the expensive payload hash operations in the signing process.

aws-sdk-go supports unsigned payload option at https://github.com/aws/aws-sdk-go/blob/main/aws/signer/v4/v4.go#L691 but seems it is not configurable in this signing proxy. So we would like to propose to add an option to the proxy to configure the go sdk to use unsigned payload.

Repeated query string variables are sorted by value

We're using aws-sigv4-proxy to sign requests to send to an API gateway. We have some APIs that we're using which are sensitive to the order of the query string parameters.

e.g. ?a=1&a=2 is not equivalent to ?a=2&a=1. This seems uncommon but not unreasonable.

The proxy sorts query string variables by key and value, so if we send ?a=2&a=1, the service we're calling receives ?a=1&a=2 and responds accordingly.

I believe this comes from aws/aws-sdk-go#1495. https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html is now explicit that for the canonical request, the parameters must be sorted by value.

What I don't know is if the canonical request used to create the signature must exactly match the request we send, or would the API gateway recalculate the canonical signature according to those rules from the actual request it receives? If the latter, aws-sigv4-proxy could use the original request with the signature generated from the canonical request it creates.

We have some workarounds available, but it seems surprising that the proxy would mutate the request more than necessary.

[Question] AWS IAM Roles for Service Accounts (IRSA)

Hello and thanks for your hard work!

I would like to deploy the proxy in a Kubernetes cluster which leverages on service accounts.

This means that the proxy will run in a pod in which the following AWS env variables will be injected:

  • AWS_ROLE_ARN
  • AWS_WEB_IDENTITY_TOKEN_FILE
  • AWS_REGION

Does the proxy support somehow such access control method?

I created this simple project to play around with OpenSearch and the OS client should be created extracting the AWS_ACCESS_KEY_ID and the AWS_SECRET_ACCESS_KEY starting from the above env variables.

This can be achieved using a little npm package @aws-sdk/credential-provider-node

import { fromTokenFile } from '@aws-sdk/credential-providers';
...
interface AwsCredential {
  accessKeyId: string;
  secretAccessKey: string;
  sessionToken?: string;
}

const credentials: AwsCredential = await fromTokenFile({
  roleArn: process.env.AWS_ROLE_ARN as string,
  webIdentityTokenFile: process.env.AWS_WEB_IDENTITY_TOKEN_FILE as string,
})();

Is it something possible? If not, would it be hard to add as feature? I'm available even if I never used Go ๐Ÿ˜„ (@alvinlin123)

go binary doesn't build

[root@ip-10-192-162-61 sigv4-proxy]# go build -v
get "gopkg.in/gemnasium/logrus-airbrake-hook.v2": found meta tag get.metaImport{Prefix:"gopkg.in/gemnasium/logrus-airbrake-hook.v2", VCS:"git", RepoRoot:"https://gopkg.in/gemnasium/logrus-airbrake-hook.v2"} at //gopkg.in/gemnasium/logrus-airbrake-hook.v2?go-get=1
get "gopkg.in/airbrake/gobrake.v2": found meta tag get.metaImport{Prefix:"gopkg.in/airbrake/gobrake.v2", VCS:"git", RepoRoot:"https://gopkg.in/airbrake/gobrake.v2"} at //gopkg.in/airbrake/gobrake.v2?go-get=1
get "gopkg.in/alecthomas/kingpin.v2": found meta tag get.metaImport{Prefix:"gopkg.in/alecthomas/kingpin.v2", VCS:"git", RepoRoot:"https://gopkg.in/alecthomas/kingpin.v2"} at //gopkg.in/alecthomas/kingpin.v2?go-get=1
get "golang.org/x/net": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at //golang.org/x/net?go-get=1
get "golang.org/x/xerrors": found meta tag get.metaImport{Prefix:"golang.org/x/xerrors", VCS:"git", RepoRoot:"https://go.googlesource.com/xerrors"} at //golang.org/x/xerrors?go-get=1
get "gopkg.in/yaml.v2": found meta tag get.metaImport{Prefix:"gopkg.in/yaml.v2", VCS:"git", RepoRoot:"https://gopkg.in/yaml.v2"} at //gopkg.in/yaml.v2?go-get=1
get "golang.org/x/sys": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at //golang.org/x/sys?go-get=1
get "golang.org/x/text": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at //golang.org/x/text?go-get=1
get "golang.org/x/crypto": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at //golang.org/x/crypto?go-get=1
get "google.golang.org/protobuf": found meta tag get.metaImport{Prefix:"google.golang.org/protobuf", VCS:"git", RepoRoot:"https://go.googlesource.com/protobuf"} at //google.golang.org/protobuf?go-get=1
get "golang.org/x/tools": found meta tag get.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at //golang.org/x/tools?go-get=1
get "golang.org/x/sync": found meta tag get.metaImport{Prefix:"golang.org/x/sync", VCS:"git", RepoRoot:"https://go.googlesource.com/sync"} at //golang.org/x/sync?go-get=1
get "gopkg.in/check.v1": found meta tag get.metaImport{Prefix:"gopkg.in/check.v1", VCS:"git", RepoRoot:"https://gopkg.in/check.v1"} at //gopkg.in/check.v1?go-get=1
get "gopkg.in/tomb.v1": found meta tag get.metaImport{Prefix:"gopkg.in/tomb.v1", VCS:"git", RepoRoot:"https://gopkg.in/tomb.v1"} at //gopkg.in/tomb.v1?go-get=1
get "gopkg.in/fsnotify.v1": found meta tag get.metaImport{Prefix:"gopkg.in/fsnotify.v1", VCS:"git", RepoRoot:"https://gopkg.in/fsnotify.v1"} at //gopkg.in/fsnotify.v1?go-get=1
go: github.com/aws/[email protected] requires
        golang.org/x/[email protected] requires
        golang.org/x/[email protected]: invalid version: git fetch --unshallow -f origin in /root/go/pkg/mod/cache/vcs/f72f7a6f267d2fe79b1e174efab21a591b07c84ef53364f65b8281fda7000b49: exit status 128:
        fatal: git fetch-pack: expected shallow list

Releasing a versioned binary for this would be nice.

Problematic signature for S3 URLs requiring escaping

Uncovered while testing key existence checks from various clients (an old jets3t in an old hadoop distro, an old python boto release, a recent ruby AWS SDK release, various direct HEAD requests from curl): it appears that the proxy produces a problematic signature in cases where an S3 key contains keys requiring URL escaping.

Our initial exposure this came from hadoop's old S3 native filesystem abstraction, which ultimately looks for keys with the suffix _$folder$. The key existence check was getting a permission denied response, rather than a found/not-found response, despite being a resource to which the proxy's credentials clearly granted access.

To illustrate:

  • pick a bucket to which you have access; for this example, let's assume it's in us-east-1 and it's named "awesome-bucket"
  • pick an arbitrary key that has this _$folder$ suffix. It doesn't matter if the key refers to extant object, we can illustrate the problem whether the object is there or not.
  • launch your proxy container with creds appropriate for our "awesome-bucket". I'm assuming that the proxy is listening on port 8080 on localhost.

If we issue a HEAD request (as all the clients I mentioned above do in this scenario) without escaping the key, we should get the suitable found/not-found HTTP result (assuming your perms correct):

curl -s -H 'Host: awesome-bucket.s3.amazonaws.com' 'http://127.0.0.1:8080/foo_$folder$'

Supposing that foo_$folder$ doesn't exist in awesome-bucket, you ought to see something like this:

HTTP/1.1 404 Not Found
Content-Type: application/xml
Date: Fri, 17 May 2019 14:41:35 GMT
Server: AmazonS3
...

Now change the curl request to use the URI-escaped path instead (as the various clients mentioned above all do).

curl -s -H 'Host: awesome-bucket.s3.amazonaws.com' 'http://127.0.0.1:8080/foo_%24folder%24'

Now you'll get permission denied, even though this is actually the proper form of the request (at least according to my understanding).

HTTP/1.1 403 Forbidden
Content-Type: application/xml
Date: Fri, 17 May 2019 14:17:20 GMT
Server: AmazonS3
...

This makes me suspect that the signer is:

  • using an improper form of the path in its signature
  • possibly re-escaping the already escaped one.

It's very strange. If you play around with different escape-warranting characters in the path, you'll find multiple scenarios in which you get a 403 Forbidden response, which seems indicative of a consistent signature problem.

I'll continue digging into this but am somewhat flummoxed. I note further that the $ character seems to have slightly different behavior than others; if you use a key foo_$ it'll be okay, but the escaped form foo_%24 isn't. But try a different character, like foo_<, and neither the escaped nor unescaped form works. To be clear, I also tested these without the proxy, using the latest AWS golang SDK release, and there were no signature issues; the problem appears to originate within the proxy itself.

Host Head as Env Var

Hello,

Is it possible to optionally pass the host header as an environmental variable into the container, so no host head is needed from the command line, the normal browser can be used, and some client applications can be configured easily?

Tiger

Support lambda-url

Issue: With the host: 123456theid.lambda-url.ap-southeast-1.on.aws I got following error

ERRO[0237] unable to proxy request                       error="unable to determine service from host: 123456theid.lambda-url.ap-southeast-1.on.aws"

Expected: Support for lambda-url

Add support for TLS

I'm using this proxy in front of an API Gateway with IAM authorization which proxies to Nginx and finally to an EKS cluster. Kubectl only will send its bearer token (using the Authorization) header if the connection to the server is using TLS and I'd rather not add yet another proxy in front of this one to terminate TLS.

Would you be open to adding optional TLS support? I have it working locally and I was thinking about these flags:

+	enableTLS              = kingpin.Flag("enable-tls", "Enable TLS").Default("false").Bool()
+	tlsCertFile            = kingpin.Flag("tls-cert-file", "TLS certificate file path").String()
+	tlsKeyFile             = kingpin.Flag("tls-key-file", "TLS key file path").String()

If so, I'd be happy to create a PR to support this.

Add release tags

I found the releases, 1.5, 1.4, and 1.3 on public ECR, but cannot find the tags in this repository. Could you add the tags to here to understand the difference?
Thanks.

Ability to support retries

Issue
We are injecting aws-sigv4-proxy for services like AMP to add sigv4 headers. However, a lot of these services run into throttling issues and the client does not understand the "Throttling Exceptions" . It would be nice to have retry mechanism support in aws-sigv4-proxy when we run into throttling issues.

Unable to access kibana with /_plugin/kibana path

I'm trying to access an ES domain inside of a VPC. I'm running this proxy on one of hosts which has a public IP and is inside of the same VPC.

docker run --rm -p 5601:8080 --name aws-sigv4-proxy kaggggggga/aws-sigv4-proxy -v

following paths work,

  • /
  • /_cat/indices
  • /_plugin/kibana/app/kibana#/home?_g=()

But this path doesn't work (which is supposed to take you to _plugin/kibana/app/kibana#/home?_g=()

  • `/_plugin/kibana/

I get following error

time="2020-04-29T17:13:44Z" level=error msg="error proxying request" message="{\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\"}"

Any idea what could be the problem ?

Shared credentials file is not reloaded when credentials expire

As mentioned by https://github.com/aws/aws-sdk-go/blob/b75b2a7b3cb40ece5774ed07dde44903481a2d4d/example/aws/request/customRetryer/custom_retryer.go#L31, the default credentials search path does not reload the file even if it only contains temporary credentials. This is because it creates static credentials from the values read and does not use the provider: https://sourcegraph.com/github.com/aws/aws-sdk-go/-/blob/aws/session/credentials.go?L116

I'm working on a PR to fix that.

Note: Also, just using the provider is not sufficient. One also has to call Credentials.Expire() to trigger the reload (which does not happen automatically even if the refresh time is in the credentials file).

Option to change user in request

Hi,

I'm using the sigv4 proxy for ingesting metrics data to AWS Managed Prometheus.
I'm getting errors that I've hit "per-user series limit". Looking at the request, it seems the "user" with which the client is sending the data is hard set to {AWS-Account-ID}_{Prometheus-Workspace-ID}.

server returned HTTP status 400 Bad Request: user=2743877562_ws-4a37396a-a467-11eb-bcbc-0242ac130002: per-user series limit of 1000000 exceeded, please contact administrator to raise it.

Is there a way to overwrite the user in the request?

Publish a package

Hello,

Please publish a package to the GitHub registry for public usage.

Thanks!

Problematic signature for non-canonical, escaped URI paths

This is closely related to issues identified in #11 and may in fact describe much of what that issue identified. However, while S3 should indeed receive its own signer (based on my read of the SDK's v4.Signer), that's not the whole issue.

Part of the issue is this: if the URI path received by the proxy has a valid but non-canonical representation, it seems that the proxy will result in an incorrect signature.

By "valid but non-canonical": you can apply the %NN encoding to characters that do not strictly need them. For instance, the "/" character; you can have them appear as "/" in the URI path, or as "%2f". My understanding is that these should be regarded as equivalent to a given application.

So, let's suppose you have the proxy container running with proper creds for "awesome-bucket", listening on 127.0.0.1:8080. It doesn't matter if the keys we look for exist or not:

ethan@nopuppet:~/source$ curl -s --head -H 'Host: awesome-bucket.s3.amazonaws.com' 'http://127.0.0.1:8080/foo/bar'
HTTP/1.1 404 Not Found
Content-Type: application/xml
Date: Fri, 17 May 2019 19:53:50 GMT
Server: AmazonS3
...

A 404 indicates that we had authorization to look for that resource, but it simply doesn't exist. Fine.

ethan@nopuppet:~/source$ curl -s --head -H 'Host: awesome-bucket.s3.amazonaws.com' 'http://127.0.0.1:8080/foo%2fbar'
HTTP/1.1 403 Forbidden
Content-Type: application/xml
Date: Fri, 17 May 2019 20:01:30 GMT
Server: AmazonS3
...

403 indicates that we are not authorized to access the requested resource, yet it is the same resource as the one above to which we had permission.

My unproven suspicion:

  • The use of the URI.String method in copying the initial request will regard the non-canonical, escaped form of the URI path as a valid encoded form and copy it to the RawPath field. As described: "The Parse function sets both Path and RawPath in the URL it returns, and URL's String method uses RawPath if it is a valid encoding of Path, by calling the EscapedPath method."
  • The non-canonical form of path will inadvertently corrupt the downstream signing, because it is preserved through the copying logic.

I suspect that what the proxy ought to do is to unescape the original request URI path and disregard its "raw" form entirely; from there, it seems like the signer would be forced to calculate the canonical form.

OOM on AWS Fargate

Hi,

I am using this container in conjunction with NGINX to sign requests to S3.
I am using it on a fargate task with a total of 2GB of RAM (split between nginx and this container), but the memory behavior is strange, it seems that the garbage collector does not work and after a while it dies from out of memory.

What is the recommended memory value?
Is there a way to set the maximum amount of memory it can use?

Thanks

install the aws-sigv4-proxy on an EKS Cluster

Hi all,

there is a way to install the aws-sigv4-proxy on an EKS Cluster using ServiceAccounts instead of directly setting the IAM Role ?
all the examples provided show only how to install it using docker.

Thanks alot

API Gateway calls return "unable to determine service"

I followed instructions from README to call API Gateway execute-api endpoint.

curl -H 'Host: <rest-api-id>.execute-api.us-east-1.amazonaws.com' http://localhost:8080/stage/path

It prints following error:

unable to proxy request - unable to determine service from host: <rest-api-id>.execute-api.us-east-1.amazonaws.com

Further investigation suggested that function named determineAWSServiceFromHost in handler/aws.go fails to match host value because of dedicated API Gateway endpoint by resource.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.