pottava/aws-s3-proxy

Reverse proxy for AWS S3 with basic authentication.

aws-s3-proxy's Issues

Authentication Failure connecting to remote Azure store

I've set up the s3 proxy as mentioned in the documentation. When I try to access, I get the following error:

AuthenticationFailed: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:92bba39e-d01e-0128-7334-131fb6000000
Time:2020-04-15T14:44:45.2488458Z
	status code: 403, request id: , host id: 

Is this a known issue ?
From some of the search on google, it seems so.
https://visdup.blogspot.com/2019/03/azurestorage-blob-server-failed-to.html

Feature Request: Directory Listings

There's a good use case for straight directory listings. Prefixes without an immediate index.html key would just list the prefix.

Not able to use s3client using IAM roles for serviceaccount

We are trying to deploy this in an EKS cluster, using IAM roles for serviceaccounts, but it does not seem to be able to authenticate to s3 using that role and we get access denied errors for our requests.

Improvement request: Better usage of status codes

I'm testing out the proxy as a simple authentication proxy to get files from S3 regardless av tool used to download.

I do not know if it matters for my specific usecase, but I'd think several users would benefit from better http status code handling.

It now seems like every "error" is returned with 500.
Instead, it would be an improvement if for example "NoSuchKey" errors were returned as 404 back to the client.

NoSuchKey: The specified key does not exist. status code: 404, request id: XXXXXXXXX, host id: YyYYYYYYYYYYYYYYYYYYYYYYYYY

I have no experience with Go, but from the Go SDK documentation it's seems like something similar to this is the way to go:

result, err := svc.AbortMultipartUpload(params)
if err != nil {
    // Cast err to awserr.Error to handle specific error codes.
    aerr, ok := err.(awserr.Error)
    if ok && aerr.Code() == <error code to check for> {
        // Specific error code handling
    }
    return err
}

fmt.Println("AbortMultipartUpload result:")
fmt.Println(result)

Are there any gzip options?

Are there any gzip options?

Is adding the gzip option possible?

invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2

Hi there, thanks for creating the v2 release with the fix from PR 31, however I get an error when I try to use it with go get while building a custom Docker image:

$ go get -d -u github.com/pottava/[email protected]
go: finding github.com v2.0.1
go: finding github.com/pottava/aws-s3-proxy v2.0.1
go: finding github.com/pottava v2.0.1
go: finding github.com/pottava/aws-s3-proxy v2.0.1
go get github.com/pottava/[email protected]: github.com/pottava/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2

The previous version works fine with go get (v1.4.1). It seems to me based on some Googling that to fix this, Go wants you to modify the module name in go.mod like so:

- module github.com/pottava/aws-s3-proxy
+ module github.com/pottava/aws-s3-proxy/v2

What do you think? An alternative might be to create a v1.5.0 release, since there's not really enough code change to warrant a major version release.

PS, I originally asked this question in Issue 32, but the author closed it (rightfully, it's a new issue).

folders are not listed when DIRECTORY_LISTINGS=true

suppose that we have this structure in a the s3 bucket proxied by s3-proxy :

├── folder1
│   ├── f11
│   └── f12
└── folder2
    ├── f21
    └── f22

when DIRECTORY_LISTINGS=true
I expect that http://proxy-endpoint/ list folder1 and folder2, but it returns empty list.
however http://proxy-endpoint/folder1/ list files inside folder1

Thanks

Helm Chart

Hey,

are they any plans to provide a helm chart for using this in a Kubernetes cluster?

Best regards
Matthias

Add implicit '/' to directories

Feature or enhancement request

Given s3 bucket content:

├── folder
│   └── index.html
└── index.html

Desired result:
GET / returns /index.html content
GET /folder/ returns /folder/index.html content
GET /folder returns /folder/index.html content

Actual result:
GET / returns /index.html content
GET /folder/ returns /folder/index.html content
GET /folder returns 404

Support for HTTP Proxy for AWS SDK

Problem: AWS SDK HTTP client is customly constructed without Proxy, hence whole thing is not working behind proxy.
Solution:
Add proxy here
Like:

	transport := &http.Transport{
		Proxy:              http.ProxyFromEnvironment,
		MaxIdleConns:       config.Config.MaxIdleConns,
		IdleConnTimeout:    config.Config.IdleConnTimeout,
		DisableCompression: config.Config.DisableCompression,
		TLSClientConfig:    tlsCfg,
	}

Basic Auth problem

Hi!
All works well except basic auth. It no ask the pass and I can access to any resource without any creds.

This is my docker-compose.yml

proxy:
image: pottava/s3-proxy
ports:

• 80:80
  env_file: .env
  environment:
• AWS_REGION=$AWS_REGION
• AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
• AWS_S3_BUCKET=$AWS_S3_BUCKET
• BASIC_AUTH_USER=admin
• BASIC_AUTH_PASS=ttt
• ACCESS_LOG=$ACCESS_LOG
• DIRECTORY_LISTINGS=true
• DIRECTORY_LISTINGS_FORMAT=html
• GET_ALL_PAGES_IN_DIR=true
  container_name: proxy

Logs:

$ docker-compose up --build
Creating proxy ... done
Attaching to proxy
proxy | 2020/01/14 09:45:43 [config] Proxy to mifprojects-english-study-video
proxy | 2020/01/14 09:45:43 [config] AWS Region: eu-central-1
proxy | 2020/01/14 09:45:43 [config] Basic authentication: admin
proxy | 2020/01/14 09:45:43 [service] listening on :80
proxy | 2020/01/14 09:45:50 [myip:33520] 0.114 200 GET /

Caching S3

Can add object caching from S3 to reduce traffic with Amazon?

GET_ALL_PAGES_IN_DIR doesn't work for "dirs" full of "subdirs"

The fix in #31 doesn't seem to work if you attempt to list a directory (prefix) full of only subdirectories (longer prefixes) as page.Contents is 0 and it stops after the first page.

Various AWS SDK documentation and examples (e.g. here on GitHub just pass in a function that always returns true, apparently the SDK will stop calling it after the last page.

A better fix is something like this:

--- a/internal/service/amazon-s3.go
+++ b/internal/service/amazon-s3.go
@@ -37,7 +37,7 @@ func (c client) S3listObjects(bucket, prefix string) (*s3.ListObjectsOutput, err
                func(page *s3.ListObjectsOutput, lastPage bool) bool {
                        result.CommonPrefixes = append(result.CommonPrefixes, page.CommonPrefixes...)
                        result.Contents = append(result.Contents, page.Contents...)
-                       return len(page.Contents) == 1000
+                       return true
                })
        return result, err
 }

Auth doesn't work

I've added BASIC_AUTH_USER and BASIC_AUTH_PASS but when I go to http://localhost:8000 it doesn't ask for credentials - just show the directory listing.

My docker-compose

proxy:
  image: pottava/s3-proxy
  restart: always
  ports:
    - 8000:80
  environment:
    - AWS_REGION=us-east-1
    - AWS_ACCESS_KEY_ID=
    - AWS_SECRET_ACCESS_KEY=
    - AWS_S3_BUCKET=
    - ACCESS_LOG=true
    - DIRECTORY_LISTINGS=true
    - DIRECTORY_LISTINGS_FORMAT=html
    - BASIC_AUTH_USER=admin
    - BASIC_AUTH_PASS=pass
  container_name: proxy
``

I AM HAPPY TO HELP

Hey @pottava ,

It seems this repo is inactive and there are a lot of awesome PRs lined up. I am happy to help out! Let me know what you think.

ben

LDAP support

Hello
Very glad I found this project, thanks.
There is not enough LDAP support for base_auth authentication.

v2 don't build from dockerfile

Hi!
I'm trying to build aws-s3-proxy from original Dockerfile. I've got eroor on build process:
`Step 1/15 : FROM golang:1.13.7-alpine3.11 AS builder
1.13.7-alpine3.11: Pulling from library/golang
c9b1b535fdd9: Pull complete
cbb0d8da1b30: Pull complete
d909eff28200: Pull complete
f9e7c84be0a1: Pull complete
8a0c202c45df: Pull complete
Digest: sha256:7e8c9c559ca6cf6535f501ca5eec46cae69489b5b8ef9cdec5abcaa5b09e707b
Status: Downloaded newer image for golang:1.13.7-alpine3.11
---> 87eefb76f0a8
Step 2/15 : RUN apk --no-cache add gcc musl-dev git
---> Running in 72db009345b5
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/17) Upgrading musl (1.1.24-r0 -> 1.1.24-r3)
(2/17) Installing libgcc (9.3.0-r0)
(3/17) Installing libstdc++ (9.3.0-r0)
(4/17) Installing binutils (2.33.1-r1)
(5/17) Installing gmp (6.1.2-r1)
(6/17) Installing isl (0.18-r0)
(7/17) Installing libgomp (9.3.0-r0)
(8/17) Installing libatomic (9.3.0-r0)
(9/17) Installing mpfr4 (4.0.2-r1)
(10/17) Installing mpc1 (1.1.0-r1)
(11/17) Installing gcc (9.3.0-r0)
(12/17) Installing nghttp2-libs (1.40.0-r1)
(13/17) Installing libcurl (7.79.1-r0)
(14/17) Installing expat (2.2.9-r1)
(15/17) Installing pcre2 (10.34-r1)
(16/17) Installing git (2.24.4-r0)
(17/17) Installing musl-dev (1.1.24-r3)
Executing busybox-1.31.1-r9.trigger
OK: 128 MiB in 31 packages
Removing intermediate container 72db009345b5
---> 10308d64b5e2
Step 3/15 : RUN go get -u github.com/pottava/aws-s3-proxy
---> Running in 4cc26d32a974

github.com/go-openapi/swag

src/github.com/go-openapi/swag/loading.go:44:28: undefined: os.ReadFile
src/github.com/go-openapi/swag/loading.go:50:28: undefined: os.ReadFile
src/github.com/go-openapi/swag/loading.go:119:10: undefined: io.ReadAll
The command '/bin/sh -c go get -u github.com/pottava/aws-s3-proxy' returned a non-zero code: 2
`
Is there opportunity to fix it?

SignatureDoesNotMatch error

Getting a nebulous error:
SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
I am sure that I have IAM credentials for the bucket in question. Anyone else run into this issue?

Display INDEX_DOCUMENT if exists and DIRECTORY_LISTINGS=true

Feature or enhancement request

It's a feature enhancement to display folder content or index.html if exists. New variable DIRECTORY_LISTINGS_CHECK_INDEX is introduced to control the behavior.

Given s3 bucket content:

├── folder
├── ── subfolder1
│   └────  index.html
├── ── subfolder2
│   └────  index.html
└── index.html

Desired result if DIRECTORY_LISTINGS=true and DIRECTORY_LISTINGS_CHECK_INDEX=true:
GET / returns /index.html content
GET /folder/ returns list of folders - [subfolder1, subfolder2]
GET /folder/subfolder1 returns /folder/subfolder1/index.html content
GET /folder/subfolder2 returns /folder/subfolder2/index.html content

New release with fixed GET_ALL_PAGES_IN_DIR

Any chance to see new release with this fix?
#31

Enabling CONTENT-ENCODING yields incorrect Content-Length header

When gzipping/deflating content, the body payload is properly encoded, but the Content-Length header is not updated.

The Content-Length header should be updated to reflect the length of the payload returned, while it currently contains the length of the payload before the encoding

Error 500 if file exists

When I try to add download a file using wget and the file is already present, the server returns 500. I think it makes more sense to return a 204, not an Internal server error

Unable to log to stdout - ACCESS_LOG only to stderr

Regardless how ACCESS_LOG is set, aws-s3-proxy logs to stderr only. Documentation appears to show setting ACCESS_LOG=true would log to to stdout.

Doesn't work behind Nginx reverse proxy

Whilst this seems to be reliable when accessed directly, when behind my Kubernetes cluster's Nginx ingress I get a lot of "upstream prematurely closed connection while sending to client" errors and Chrome fails to load a high percentage of the resources. If I request a single file using curl it seems to work. This is using HTTP/1.1.

Returns a 404 error when tried to access on the browser

Hello,
I am running the following

1. docker run command docker run -d -p 8080:80 -e AWS_REGION -e AWS_S3_BUCKET -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY pottava/s3-proxy
2. Container started running
   CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
   2fffaa2150e0 pottava/s3-proxy "/aws-s3-proxy" 4 seconds ago Up 4 seconds 0.0.0.0:8080->80/tcp, :::8080->80/tcp quirky_tharp
3. Here are the docker logs
   docker logs 2fffaa2150e0 2023/02/13 04:00:27 [config] Proxy to bni-software-test-reports 2023/02/13 04:00:27 [config] AWS Region: us-east-1 2023/02/13 04:00:27 [service] listening on :80
4. When I try to open on the browser with the following url http://<host-ip>:8080/<bucket-name>/<folder>/<index.html>
   It gives me the below error
   NoSuchKey: The specified key does not exist.
   status code: 404, request id:

Could you take a look and let me know if this url and port is correct? Thank you!

Feature Request: Configurable max objects

Great work @pottava on the tool, was a life saver when I need to proxy some s3 request.

Please is it possible to list objects more than 1000 in a path?

Thanks

Contrary to documentation, CONTENT_ENCODING defaults to true

The dockerfile sets CONTENT_ENCODING to true by default

add support for arm64 for docker images

Please add support for arm64 for docker images

Response handling

Hi, is it possible to add features for response handling?

We use this proxy to provide a static angular website to the internal vpc only. We need to route back requests with 404 or 403 code to index.html and code 200.

Thanks in advance.

aws-s3-proxy fails to proxy encrypted objects from encrypted buckets

Hi,

it looks like aws-s3-proxy cannot proxy encrypted objets on a bucket with Server Side Encryption (SSE) configured. The browser shows the following error from S3

<Error>
<Code>InvalidArgument</Code>
<Message>
Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.
</Message>
<ArgumentName>Authorization</ArgumentName>
<ArgumentValue>null</ArgumentValue>
<RequestId>3431EC777CCE0481</RequestId>
<HostId>
KB4eUwlajOnuFcHuxFmpGoCy3s+kftrtdlB3JrALmCg9qEw9LvkHmtSkvOk2xhIZmgSOfOzYgrM=
</HostId>
</Error>

Looking at the documentation, AWS Signature Version 4 should be managed by the golang SDK. Am I missing something ?