nginxinc/nginx-s3-gateway

NGINX S3 Caching Gateway

NGINX S3 Gateway

Introduction

This project provides a working configuration of NGINX configured to act as an authenticating and caching gateway for to AWS S3 or another S3 compatible service. This allows you to proxy a private S3 bucket without requiring users to authenticate to it. Within the proxy layer, additional functionality can be configured such as:

• Listing the contents of a S3 bucket
• Providing an authentication gateway using an alternative authentication system to S3
• Caching frequently accessed S3 objects for lower latency delivery and protection against S3 outages
• For internal/micro services that can't authenticate against the S3 API (e.g. don't have libraries available) the gateway can provide a means to accessing S3 objects without authentication
• Compressing objects (gzip, brotli) from gateway to end user
• Protecting S3 bucket from arbitrary public access and traversal
• Rate limiting S3 objects
• Protecting a S3 bucket with a WAF
• Serving static assets from a S3 bucket alongside a dynamic application endpoints all in a single RESTful directory structure

All such functionality can be enabled within a standard NGINX configuration because this project is nothing other than NGINX with additional configuration that allows for proxying S3. It can be used as-is if the predefined configuration is sufficient, or it can serve as a base example for a more customized configuration.

If the predefined configuration does not meet your needs, it is best to borrow from the patterns in this project and build your own configuration. For example, if you want to enable SSL/TLS and compression in your NGINX S3 gateway configuration, you will need to look at other documentation because this project does not enable those features of NGINX.

Usage

This project can be run as a stand-alone container or as a Systemd service. Both modes use the same NGINX configuration and are functionally equal in terms features. However, in the case of running as a Systemd service, other services can be configured that additional functionality such as certbot for Let's Encrypt support.

Getting Started

Refer to the Getting Started Guide for how to build and run the gateway.

Directory Structure and File Descriptions

common/                          contains files used by both NGINX OSS and Plus configurations
  etc/nginx/include/
    awscredentials.js            common library to read and write credentials
    awssig2.js                   common library to build AWS signature 2
    awssig4.js                   common library to build AWS signature 4 and get a session token
    s3gateway.js                 common library to integrate the s3 storage from NGINX OSS and Plus
    utils.js                     common library to be reused by all of NJS codebases
deployments/                     contains files used for deployment technologies such as
                                 CloudFormation
docs/                            contains documentation about the project
examples/                        contains additional `Dockerfile` examples that extend the base
                                 configuration
jsdoc                            JSDoc configuration files
oss/                             contains files used solely in NGINX OSS configurations
plus/                            contains files used solely in NGINX Plus configurations
test/                            contains automated tests for validang that the examples work
Dockerfile.oss                   Dockerfile that configures NGINX OSS to act as a S3 gateway
Dockerfile.plus                  Dockerfile that builds a NGINX Plus instance that is configured
                                 equivelently to NGINX OSS - instance is configured to act as a
                                 S3 gateway with NGINX Plus additional features enabled
Dockerfile.buildkit.plus         Dockerfile with the same configuration as Dockerfile.plus, but
                                 with support for hiding secrets using Docker's Buildkit
Dockerfile.latest-njs            Dockerfile that inherits from the last build of the gateway and
                                 then builds and installs the latest version of njs from source
Dockerfile.unprivileged          Dockerfiles that inherits from the last build of the gateway and
                                 makes the necessary modifications to allow running the container
                                 as a non root, unprivileged user.
package.json                     Node.js package file used only for generating JSDoc
settings.example                 Docker env file example
standalone_ubuntu_oss_install.sh install script that will install the gateway as a Systemd service
test.sh                          test launcher

Development

Refer to the Development Guide for more information about extending or testing the gateway.

Contributing

Please see the contributing guide for guidelines on how to best contribute to this project.

License

Apache License, Version 2.0

© F5, Inc. 2020 - 2024

nginx-s3-gateway's Issues

Error 404 Handling During Debugging

Discussed in #37

^{Originally posted by rmcsqrd May 24, 2022}
Hi there, thanks for making this - I got it working successfully and it is very useful.

I ran into issues with the REST authentication because MFA was enabled for the AWS account tied to my access key/secret key pair and ended up needing to make a new role with reduced permissions and no MFA (obviously this is not at all related to your package).

During the debugging process I found the the default 404 response behavior was a bit of a red herring in the debugging process (but understand its utility for production systems). I spent a lot of time trying a lot of different things before I dug into the actual nginx conf file and found the part of the configuration that changed the error response. Once I commented these lines out locally I saw I was actually getting a 403 Access Denied error which helped narrow down the debugging process.

I think it would be useful to mention this within the "Getting Started Guide" as a potential way to debug any authentication error. I also think it might be useful to mention the MFA thing as well. I am happy to file a documentation PR calling this out if you think it would be useful.

Object names with percentage signs do not have valid links on index pages

Generated nginx config is invalid if namesevers are IPv6-only

Describe the bug
This is the same bug as nginx-proxy/nginx-proxy#938

Currently, if there are IPv6 nameservers in /etc/resolv.conf, they are not properly formatted when they are inserted into the Nginx template. Nginx requires them to be surrounded by [] like [2a02::1:1].

To Reproduce
I'm not sure how to manage a resolv.conf inside the container to have an IPv6 nameserver (it's the case in my setup using an EKS cluster, probably with some IPv6 settings).

Steps to reproduce the behavior:

1. Start container with a line nameserver: 2a02::1:1 in /etc/resolv.conf
2. Check logs

[emerg] 1#1: invalid port in resolver "2a02::1:1" in /etc/nginx/conf.d/upstreams.conf:2

Expected behavior

No error on load up.

Your environment

• Version of the repo: ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest-20221216
• S3 backend implementation you are using (AWS, Ceph, NetApp StorageGrid, etc): AWS
• How you are deploying: Helm (kubernetes)
• NGINX type (OSS/Plus): OSS
• Authentication method (IAM, IAM with Fargate, IAM with K8S, AWS Credentials, etc): IAM with K8S

Indicate what environment variables are optional in getting started docs

Currently, some optional environment variables are not coded as such. This issue is to improve the documentation by explicitly showing what is required and what isn't.

Example of alternative auth methods

Is there any doc or example of the "Providing an authentication gateway using an alternative authentication system to S3" feature?

I have a s3 compatible endpoint with a keypair to access it. I would like to run nginx-s3-gateway in front of it to be able to generate additional keypairs and restrict access for specific keys to specific buckets. Is there any way to pass a list of keypairs to nginx and let it handle the authentication?

Add documentation or message showing how to install wait-for-it

Container image runs as root and does not work in Openshift

Some Kubernetes distributions like Openshift are more restrictive by default regarding running container with root privileges.
Thus, It would be nice to have an image supporting non-root execution.
Here is some extended info about the issue https://torstenwalter.de/openshift/nginx/2017/08/04/nginx-on-openshift.html

I opened a PR adding a Dockerfile that modify parent OSS image with necessary changes to run the container with a non-root user, based on the steps done in https://github.com/nginxinc/docker-nginx-unprivileged/blob/main/Dockerfile-debian.template

#72

nginx.service failed due to the unknown "cors_allowed_origin" variable

Describe the bug
Failed to start nginx.service if CORS_ALLOWED_ORIGIN is not defined in /etc/nginx/environment when running nginx-s3-gateway as a Systemd Service using standalone_ubuntu_oss_install.sh.

To Reproduce
Steps to reproduce the behavior:

1. Provision a VM (Ubuntu 20.04) on vSphere
2. Switch user as a root via sudo su.
3. Clone this repo.
4. Copy and note AWS credentials:
   • AWS_ACCESS_KEY_ID
   • AWS_SECRET_ACCESS_KEY
   • AWS_SESSION_TOKEN
5. Update standalone_ubuntu_oss_install.sh with the following information:

   S3_BUCKET_NAME=your-bucket
   S3_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
   S3_SECRET_KEY=${AWS_SECRET_ACCESS_KEY}
   S3_SESSION_TOKEN=${AWS_SESSION_TOKEN}
   S3_SERVER=s3-us-east-2.amazonaws.com
   S3_SERVER_PORT=443
   S3_SERVER_PROTO=https
   S3_REGION=us-east-2
   S3_STYLE=virtual
   S3_DEBUG=true
   AWS_SIGS_VERSION=4
   ALLOW_DIRECTORY_LIST=true
   PROVIDE_INDEX_PAGE=false
   APPEND_SLASH_FOR_POSSIBLE_DIRECTORY=false
   PROXY_CACHE_VALID_OK=1h
   PROXY_CACHE_VALID_NOTFOUND=1m
   PROXY_CACHE_VALID_FORBIDDEN=30s
   CORS_ENABLED=0
   

6. Install NGINX using standalone_ubuntu_oss_install.sh
7. See the following errors:

   ▶ Starting NGINX
   Job for nginx.service failed because the control process exited with error code.
   See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.

Expected behavior
The NGINX service should be successfully started as the following message:

▶ Creating NGINX configuration for S3 Gateway
▶ Creating directory for proxy cache
▶ Starting NGINX
root@ip-172-31-41-141:/home/ubuntu/nginx-s3-gateway-v2# ps -ef | grep nginx
root        2783       1  0 13:13 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx       2784    2783  0 13:13 ?        00:00:00 nginx: worker process
nginx       2785    2783  0 13:13 ?        00:00:00 nginx: cache manager process
nginx       2786    2783  0 13:13 ?        00:00:00 nginx: cache loader process
root        2787    2600  0 13:13 pts/3    00:00:00 systemctl status nginx.service

Your environment

• Version of the repo - master branch
• Version of the container used (if downloaded from Docker Hub or Github): N/A
• S3 backend implementation you are using (AWS, Ceph, NetApp StorageGrid, etc): AWS
• How you are deploying Docker/Stand-alone, etc: Stand-alone
• NGINX type: OSS
• Authentication method (IAM, IAM with Fargate, IAM with K8S, AWS Credentials, etc): AWS Credentials

Additional context
You could find the error message as follows using the command of systemctl status nginx.service.

$ systemctl status nginx.service

● nginx.service - nginx - high performance web server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/nginx.service.d
             └─override.conf
     Active: failed (Result: exit-code) since Thu 2023-03-02 04:05:00 UTC; 9s ago
       Docs: https://nginx.org/en/docs/
    Process: 5488 ExecStartPre=/usr/local/bin/template_nginx_config.sh (code=exited, status=0/SUCCESS)
    Process: 5554 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=1/FAILURE)

Mar 02 04:04:59 testenv-f2b73179-host-4 template_nginx_config.sh[5501]: template_nginx_config.sh: Running envsubst on /etc/nginx/templates/gateway/v2_headers.conf.template to /etc/nginx/conf.d/gateway/v2_headers.conf
Mar 02 04:04:59 testenv-f2b73179-host-4 template_nginx_config.sh[5501]: template_nginx_config.sh: Running envsubst on /etc/nginx/templates/gateway/v4_js_vars.conf.template to /etc/nginx/conf.d/gateway/v4_js_vars.conf
Mar 02 04:04:59 testenv-f2b73179-host-4 template_nginx_config.sh[5501]: template_nginx_config.sh: Running envsubst on /etc/nginx/templates/gateway/v4_headers.conf.template to /etc/nginx/conf.d/gateway/v4_headers.conf
Mar 02 04:04:59 testenv-f2b73179-host-4 template_nginx_config.sh[5501]: template_nginx_config.sh: Running envsubst on /etc/nginx/templates/gateway/v2_js_vars.conf.template to /etc/nginx/conf.d/gateway/v2_js_vars.conf
Mar 02 04:04:59 testenv-f2b73179-host-4 template_nginx_config.sh[5501]: template_nginx_config.sh: Running envsubst on /etc/nginx/templates/gateway/cors.conf.template to /etc/nginx/conf.d/gateway/cors.conf
Mar 02 04:04:59 testenv-f2b73179-host-4 template_nginx_config.sh[5501]: template_nginx_config.sh: Running envsubst on /etc/nginx/templates/gateway/js_fetch_trusted_certificate.conf.template to /etc/nginx/conf.d/gateway/js_fetc>
Mar 02 04:05:00 testenv-f2b73179-host-4 nginx[5554]: nginx: [emerg] unknown "cors_allowed_origin" variable
Mar 02 04:05:00 testenv-f2b73179-host-4 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE

Sensitive Information
N/A

get 404 error when object name contains space

When the object name contains space, the browser converts the spaces with %20 and returns 404 however the file exists in s3 storage.

Add sample Docker file configurations for gzip and brotli compression

Upgrade MinIO

MinIO is used for integration tests. We are currently using an old version and it would be good to upgrade.

Docker DNS resolver specified in NGINX Plus Docker configuration

The resolver specified in upstreams.conf is the Docker Compose DNS server:

# Docker DNS server IP
resolver 127.0.0.11;

However, this DNS server isn't available when running Docker in stand-alone mode or in Kubernetes. The DNS server used by NGINX should be configurable in order to avoid hard coded incompatibilities.

Gateway leaks x-goog- headers when running on Google Storage Engine

When using Google Storage Engine, the gateway fails to strip Google-specific headers from the response and thereby potentially leaks metadata about the object and bucket.

For example:

Server: nginx
Date: Thu, 03 Nov 2022 22:08:51 GMT
Content-Type: text/plain
Content-Length: 2132
Connection: keep-alive
X-GUploader-UploadID: ADPycdt0MrPnE__VwGIJ4k7a0XvuYOk6fel0l5WLn1mgo97NQnKsnrxY6hq85PCW64L6xJ2n_D7CgAI1Za3p9eKh4hru7A
Cache-Control: public
Expires: Thu, 03 Nov 2022 22:08:14 GMT
Last-Modified: Thu, 03 Nov 2022 21:57:33 GMT
ETag: "7d59bff32f153073ff8e8a461cb4979b"
x-goog-generation: 1667512653610340
x-goog-metageneration: 3
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 2132
x-goog-hash: crc32c=0AaUeg==
x-goog-hash: md5=fVm/8y8VMHP/jopGHLSXmw==
x-goog-storage-class: STANDARD
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Accept-Ranges: bytes

No useful error message emitted when running tests when wait-for-it is not installed

If wait-for-it is not installed on the system, when test.sh is executed, it will exit without returning an error message.

I'm getting connection reset by peer issue since the :latest-20221216 , using idrive

Hello,

I've been getting issues since :latest-20221216. :latest-20221215 was working nicely.

I'm using Idrive E2 for storage solution.

Here's my docker compose related section:

nginx_s3_gateway:
    image: nginxinc/nginx-s3-gateway:latest-20221215
    ports:
      - "1234:8080"
    restart: "no"
    environment:
      S3_BUCKET_NAME: "mybucketname"
      S3_ACCESS_KEY_ID: "myaccesskey"
      S3_SECRET_KEY: "mysecretkey"
      S3_SERVER: "my.idrivee2.server.url.com"
      S3_SERVER_PORT: "443"
      S3_SERVER_PROTO: "https"
      S3_REGION: ""
      S3_DEBUG: "true"
      S3_STYLE: "virtual" # virtual
      ALLOW_DIRECTORY_LIST: "false"
      PROVIDE_INDEX_PAGE:
      APPEND_SLASH_FOR_POSSIBLE_DIRECTORY:
      AWS_SIGS_VERSION: 4
      STATIC_SITE_HOSTING:
      PROXY_CACHE_VALID_OK: "1h"
      PROXY_CACHE_VALID_NOTFOUND: "1m"
      PROXY_CACHE_VALID_FORBIDDEN: "30s"

This configuration works nicely with :latest-20221215, but I get "connection reset by peer" issue with :latest-20221216

I get the correct information using the yesterday's tag:

➜  docker curl http://192.168.50.3:1234/nah.jpg -I
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 16 Dec 2022 19:10:03 GMT
Content-Type: image/jpeg
Content-Length: 40131
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
ETag: "yadayadayada"
Last-Modified: Tue, 13 Dec 2022 21:26:25 GMT
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
Accept-Ranges: bytes

But today's tag gives me this:

➜  docker curl http://192.168.50.3:1234/nah.jpg -I
curl: (56) Recv failure: Connection reset by peer

I cannot see any detailed logs in my logs though:

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/00-check-for-required-env.sh
S3 Backend Environment
Access Key ID: myaccesskey
Origin: https://my-idrive-host.com:443
�
Region: auto
Addressing Style: virtual
AWS Signatures Version: v4
DNS Resolvers: 127.0.0.11
Directory Listing Enabled: false
Provide Index Pages Enabled: 
Append slash for directory enabled: 
Stripping the following headers from responses: x-amz-;
CORS Enabled: 0
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/s3_server.conf.template to /etc/nginx/conf.d/gateway/s3_server.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v2_headers.conf.template to /etc/nginx/conf.d/gateway/v2_headers.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/s3_location.conf.template to /etc/nginx/conf.d/gateway/s3_location.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/s3listing_location.conf.template to /etc/nginx/conf.d/gateway/s3listing_location.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v4_headers.conf.template to /etc/nginx/conf.d/gateway/v4_headers.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v2_js_vars.conf.template to /etc/nginx/conf.d/gateway/v2_js_vars.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/cors.conf.template to /etc/nginx/conf.d/gateway/cors.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v4_js_vars.conf.template to /etc/nginx/conf.d/gateway/v4_js_vars.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/js_fetch_trusted_certificate.conf.template to /etc/nginx/conf.d/gateway/js_fetch_trusted_certificate.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/upstreams.conf.template to /etc/nginx/conf.d/upstreams.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/22-enable_js_fetch_trusted_certificate.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/12/16 19:26:18 [notice] 1#1: using the "epoll" event method
2022/12/16 19:26:18 [notice] 1#1: nginx/1.23.3
2022/12/16 19:26:18 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 
2022/12/16 19:26:18 [notice] 1#1: OS: Linux 5.10.0-19-amd64
2022/12/16 19:26:18 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2022/12/16 19:26:18 [notice] 1#1: start worker processes
2022/12/16 19:26:18 [notice] 1#1: start worker process 75
2022/12/16 19:26:18 [notice] 1#1: start cache manager process 76
2022/12/16 19:26:18 [notice] 1#1: start cache loader process 77
2022/12/16 19:27:22 [notice] 77#77: http file cache: /var/cache/nginx/s3_proxy 0.000M, bsize: 4096
2022/12/16 19:27:22 [notice] 1#1: signal 17 (SIGCHLD) received from 77
2022/12/16 19:27:22 [notice] 1#1: cache loader process 77 exited with code 0
2022/12/16 19:27:22 [notice] 1#1: signal 29 (SIGIO) received

So I was wondering, what has changed since yesterday, that would result breaking it, and how do I resolve it?

Thanks in advance,

Version '1.22.0+0.7.4-1~bullseye' for 'nginx-module-njs' was not found

Steps to reproduce:

• git clone https://github.com/nginxinc/nginx-s3-gateway.git
• cd nginx-s3-gateway
• docker build --file ./Dockerfile.oss --tag nginx-s3-gateway .

During the build, this command fails:
apt-get install --no-install-recommends --no-install-suggests --yes nginx-module-njs=1.22.0+0.7.4-1~bullseye

With the error message:

The command '/bin/sh -c set -eux     export DEBIAN_FRONTEND=noninteractive;     mkdir -p /var/cache/nginx/s3_proxy;     chown nginx:nginx /var/cache/nginx/s3_proxy;     chmod -R -v +x /docker-entrypoint.sh /docker-entrypoint.d/*.sh;     echo "deb https://nginx.org/packages/mainline/debian/ $(echo $PKG_RELEASE | cut -f2 -d~) nginx" >> /etc/apt/sources.list.d/nginx.list;     apt-get update;     apt-get install --no-install-recommends --no-install-suggests --yes       nginx-module-njs=${NGINX_VERSION}+${NJS_VERSION}-${PKG_RELEASE};     apt-get remove --purge --auto-remove --yes;     rm -rf /var/lib/apt/lists/* /etc/apt/sources.list.d/nginx.list' returned a non-zero code: 100

This worked 8 days ago as described above.

Unable to run docker image

I'm unable to run the docker image (I'm new to both Docker and nginx). Here is what I did:

1. Clone the repository
2. Create a "settings" file with:

S3_BUCKET_NAME=my-bucket-name
S3_ACCESS_KEY_ID=my-access-key-id
S3_SECRET_KEY=my-secret-key
S3_SERVER=s3.amazonaws.com
S3_SERVER_PORT=443
S3_SERVER_PROTO=https
S3_REGION=us-west-1
S3_STYLE=virtual
S3_DEBUG=false
AWS_SIGS_VERSION=4
ALLOW_DIRECTORY_LIST=false

3. Built:
   docker build -f Dockerfile.oss -t nginx-oss-s3-gateway .

4. Attempted to run with specified command:

docker run --env-file ./settings -p8080:80 --name nginx-oss-s3-gateway nginx-s3-gateway
Unable to find image 'nginx-s3-gateway:latest' locally
docker: Error response from daemon: pull access denied for nginx-s3-gateway, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.

5. Attempted to modify the run command:
   docker run -p8080:80 --env-file ./settings nginx-oss-s3-gateway
   standard_init_linux.go:228: exec user process caused: no such file or directory

I'm using Docker Destop 3.5.2 on Windows 10. A basic docker file with nginx:latest (and just a copy of index.html file into the image) works.

Not sure what I'm doing wrong, please help. :)

Failed to start nginx.service when running standalone_ubuntu_oss_install.sh

Describe the bug
Failed to start nginx.service when running nginx-s3-gateway as a Systemd Service using standalone_ubuntu_oss_install.sh

To Reproduce
Steps to reproduce the behavior:

1. Create an IAM role
2. Provision an EC2 instance which associates with AWS instance profile credentials.
3. Switch user as a root via sudo su
4. Install NGINX using standalone_ubuntu_oss_install.sh
5. See the following errors:

   ▶ Starting NGINX
   Job for nginx.service failed because the control process exited with error code.
   See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.

Expected behavior
The NGINX service should be successfully started as the following message:

▶ Creating NGINX configuration for S3 Gateway
▶ Creating directory for proxy cache
▶ Starting NGINX
root@ip-172-31-41-141:/home/ubuntu/nginx-s3-gateway-v2# ps -ef | grep nginx
root        2783       1  0 13:13 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx       2784    2783  0 13:13 ?        00:00:00 nginx: worker process
nginx       2785    2783  0 13:13 ?        00:00:00 nginx: cache manager process
nginx       2786    2783  0 13:13 ?        00:00:00 nginx: cache loader process
root        2787    2600  0 13:13 pts/3    00:00:00 systemctl status nginx.service

A clear and concise description of what you expected to happen.

Your environment

• Version of the repo - master branch
• Version of the container used (if downloaded from Docker Hub or Github): N/A
• S3 backend implementation you are using (AWS, Ceph, NetApp StorageGrid, etc): AWS
• How you are deploying Docker/Stand-alone, etc: Stand-alone
• NGINX type: OSS
• Authentication method (IAM, IAM with Fargate, IAM with K8S, AWS Credentials, etc): IAM

Additional context

Same error occurs when the following files do not exist:

• cors.conf
• js_fetch_trusted_certificate.conf
• s3listing_location.conf
• s3_location.conf
• s3_server.conf

You could fine the error message as follows using the command of systemctl status nginx.service.

    root@ip-172-31-41-141:/home/ubuntu# systemctl status nginx.service
    × nginx.service - nginx - high performance web server
        Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
        Drop-In: /etc/systemd/system/nginx.service.d
                └─override.conf
        Active: failed (Result: exit-code) since Mon 2023-02-13 12:38:26 UTC; 35s ago
          Docs: https://nginx.org/en/docs/
        Process: 2463 ExecStartPre=/usr/local/bin/template_nginx_config.sh (code=exited, status=0/SUCCESS)
        Process: 2507 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=1/FAILURE)
            CPU: 54ms

    Feb 13 12:38:26 ip-172-31-41-141 template_nginx_config.sh[2476]: template_nginx_config.sh: Running envsubst on /etc/nginx/templates/gateway/js_fetch_trusted_certificat>
    Feb 13 12:38:26 ip-172-31-41-141 template_nginx_config.sh[2476]: template_nginx_config.sh: Running envsubst on /etc/nginx/templates/gateway/cors.conf.template to /etc/>
    Feb 13 12:38:26 ip-172-31-41-141 template_nginx_config.sh[2476]: template_nginx_config.sh: Running envsubst on /etc/nginx/templates/gateway/v4_js_vars.conf.template to>
    Feb 13 12:38:26 ip-172-31-41-141 template_nginx_config.sh[2476]: template_nginx_config.sh: Running envsubst on /etc/nginx/templates/gateway/v4_headers.conf.template to>
    Feb 13 12:38:26 ip-172-31-41-141 template_nginx_config.sh[2476]: template_nginx_config.sh: Running envsubst on /etc/nginx/templates/gateway/v2_headers.conf.template to>
    Feb 13 12:38:26 ip-172-31-41-141 template_nginx_config.sh[2476]: template_nginx_config.sh: Running envsubst on /etc/nginx/templates/gateway/s3_server.conf.template to >
    Feb 13 12:38:26 ip-172-31-41-141 nginx[2507]: nginx: [emerg] open() "/etc/nginx/conf.d/gateway/s3_location.conf" failed (2: No such file or directory) in /etc/nginx/co>
    Feb 13 12:38:26 ip-172-31-41-141 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
    Feb 13 12:38:26 ip-172-31-41-141 systemd[1]: nginx.service: Failed with result 'exit-code'.
    Feb 13 12:38:26 ip-172-31-41-141 systemd[1]: Failed to start nginx - high performance web server.

Sensitive Information
N/A

Add temporary security credentials without IAM policy

Is your feature request related to a problem? Please describe.
As a Software Engineer,

• Currently, I can run nginx-s3-gateway using AWS Access key id and secret access key without an IAM Policy.
• In addition to that I want to run and test it either in a container or on a laptop using temporary security credentials which include a session token.

Describe the solution you'd like

• Read a session token from the environment variable of S3_SESSION_TOKEN if the variable exists and is not empty.
• Otherwise, set to null to return credentials which only contain AWS access key ID and secret key.

Describe alternatives you've considered

• In case the S3_SESSION_TOKEN isn't specified in your AWS profile, the nginx-s3-gateway can be accessed using access key id and secret access key as is.

Additional context

• This PR is used when a session token is required to access your service without an IAM Policy as the following picture.

updated metadata headers are not served until a restart

Describe the bug
When you change the metadata information on an object (eg Cache-Control), the headers are not served until a restart of container image.

To Reproduce
Steps to reproduce the behavior:

1. start nginx
2. curl to an object on the bucket, check Cache-Control header is missing

curl -D - -o /dev/null http://nginx-s3-gateway/image.jpg

3. update all objects

aws s3  cp s3://bucket  s3://bucket \
   --recursive \
   --metadata-directive REPLACE \
   --exclude "*" \
   --include "*.jpg" \
   --cache-control public,max-age=1209600 \
   --content-type image/jpeg

4. All .jpg objects now metadata headers
5. curl to after the 5min cache time. Still no cache header.

curl -D - -o /dev/null http://nginx-s3-gateway/image.jpg

6. restart the nginx container
7. Presto! Cache-Control header of the object appears

curl -D - -o /dev/null http://nginx-s3-gateway/image.jpg

Expected behavior
we expect the cache-control header to appear

Your environment

• Version of the repo - a specific commit or tag
  ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest-njs-oss-20220815
• Version of the container used (if downloaded from Docker Hub or Github)
  ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest-njs-oss-20220815
• S3 backend implementation you are using (AWS, Ceph, NetApp StorageGrid, etc)
  AWS
• How you are deploying Docker/Stand-alone, etc
  k8s
• NGINX type (OSS/Plus)
  no
• Authentication method (IAM, IAM with Fargate, IAM with K8S, AWS Credentials, etc)
  AWS Credentials

Add modsecurity installation example

Viewing an empty bucket returns "404 Not Found"

Describe the bug
An empty bucket displays "404 Not Found - nginx" and returns HTTP status code 404.

To Reproduce
Steps to reproduce the behavior:

1. Start nginx-s3-gateway service configured to attach to a bucket without content.
2. Open browser to nginx-s3-gateway service.
3. See error "404 Not Found - nginx"

Expected behavior
Show an empty directory listing and return HTTP status code 200.
This is vital to distinguish the empty-bucket scenario from a failure scenario.

Your environment

• Version of the container used (if downloaded from Docker Hub or Github)
  nginxinc/nginx-s3-gateway:latest-20221104
• S3 backend implementation you are using (AWS, Ceph, NetApp StorageGrid, etc)
  Google Cloud Storage
• How you are deploying Docker/Stand-alone, etc
  Docker on MacOS using Rancher Desktop.
• NGINX type (OSS/Plus)
  OSS
• Authentication method (IAM, IAM with Fargate, IAM with K8S, AWS Credentials, etc)
  S3 authentication using Google Service Account with HMAC keys.

Additional context
We have buckets with short lifecycles on objects. The buckets contain artifacts from various builds to be used by manual and automatic tests. Periodically, the buckets are empty and the nginx-s3-gateway then returns "404 Not Found" on the empty buckets.
The 404 error code is easily perceived as a service failure. Although it is not.

Startup fails when using IAM Roles for Service Accounts

nginx-s3-gateway container does not start when using IAM roles for service accounts (in EKS) as authentication mechanism.

To Reproduce
start the container with the following variables:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: example
  template:
    metadata:
      labels:
        app.kubernetes.io/name: example
    spec:
      serviceAccountName: my-sa
      containers:
        - name: nginx-s3-gateway
          image: nginxinc/nginx-s3-gateway:latest
          env:
            - name: AWS_SIGS_VERSION
              value: "4"
            - name: S3_BUCKET_NAME
              value: my-bucket
            - name: S3_REGION
              value: us-east-1
            - name: S3_SERVER
              value: s3.us-east-1.amazonaws.com
            - name: S3_SERVER_PORT
              value: "443"
            - name: S3_SERVER_PROTO
              value: "https"
            - name: S3_STYLE
              value: default

causes this error on startup:

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/00-check-for-required-env.sh
Required S3_ACCESS_KEY_ID environment variable missing
Required S3_SECRET_KEY environment variable missing

Expected behavior
The container startup should not fail.

Your environment

• Version of the repo: latest (3052946)
• Version of the container used: latest (42aa78b1f5d7c316ea93611ccae44170519b434e96070f17b388b07bf4cf4c67)
• S3 backend implementation: AWS
• How you are deploying: EKS
• NGINX type: OSS
• Authentication method: IAM role for service account

Support for index documents

Thank you for an extremely useful proxy!

At the moment the S3 signature is calculated with the original URI, therefore there's no ability to rewrite this URI to access an index.html or equivalent document.

My crude attempt to redirect root requests to an index document is below. It basically bypasses the proxy for any path without a file extension and sends it directly to S3. Not ideal.

    location ~ ^[^\.]*\/?$ {
        proxy_pass_request_headers off;
        proxy_set_header Host '${S3_BUCKET_NAME}.${S3_SERVER}';
        proxy_http_version 1.1;
        proxy_set_header Connection '';
        proxy_intercept_errors on;
        js_header_filter s3gateway.filterOutAmzHeaders;
        rewrite ^[^\.]*\/?$ /index.html break;  # don't care
        proxy_pass ${S3_SERVER_PROTO}://${S3_SERVER};
    }

Any further ideas would be appreciated.

response 404 from aws s3 for filenames with spaces

Hi.
in AWS, when filename with spaces is uploaded, in the generated URL, every space replaced with '+'.
example: i uploaded a filename: "n ginx.conf".
the URL generated by aws: https://.s3.eu-central-1.amazonaws.com/n+ginx.conf

when i try to download the file using s3 gw using path "n+ginx.conf", i get 404.

since the GW replaces '+' with '%2B', i made a try and in s3gateway.ls, in "_encodeURIComponent", i replaced back from '%2B' to '+', but then i got from AWS "SignatureDoesNotMatch" response, and in the details i see that the '+' was changed to '%20' which is the space code.

can you please help me understand the issue?

Best,
Chen

Drop support for Sigv2

Even though, existing buckets are still able to use Sigv2 for S3 (new buckets can't do so for quite some time now already), I don't see any legitimate argument, why this gateway should support Sigv2 any longer:
https://aws.amazon.com/blogs/aws/amazon-s3-update-sigv2-deprecation-period-extended-modified/

• every bucket is able to use Sigv4 (as far as I know)
• Only a small subset of regions even still support Sigv2 (https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html)
• Implementing new features (like #8) might get harder to implement when both signature versions are going to be supported (as both versions need to be maintained)

What do others think about that? :)

Unable to list contents of unicode prefixes/directories

This error reproduces with both AWS S3 and Minio. If you try to access an object under a unicode prefix/directory, it will return without problems. However, if you try to list the contents of a prefix/directory, it will return the error reported above. The string junk is returned from this line: https://github.com/nginxinc/nginx-s3-gateway/blob/master/common/etc/nginx/include/s3gateway.js#L291

Originally posted by @dekobon in #17 (reply in thread)

Instance profile support does not work in systemd configuration

No useful error message emitted when running tests when docker is not installed

If Docker is not installed on the system, when test.sh is executed, it will exit without returning an error message.

Running React app with Nginx

Hi,

I'm trying to run React app in S3 using Nginx. The default behaviour for non-existent pages should be routing to /index.html. Here is my default.conf file for Nginx. Unfortunately it doesn't work as expected and I get 404 error on any non-existent page. I also tried to use proxy_pass in location @error404 block but I get auth error from AWS. I would appreciate any help on this issue. Thanks in advance.

# We include only the variables needed for the authentication signatures that
# we plan to use.
include /etc/nginx/conf.d/gateway/v4_js_vars.conf;

error_log /dev/stderr debug;

# Extracts only the path from the requested URI. This strips out all query
# parameters and anchors in order to prevent extranous data from being sent to
# S3.
map $request_uri $uri_path {
    "~^(?P<path>.*?)(\?.*)*$"  $path;
}

map virtual $s3_host_hdr {
    virtual "${S3_BUCKET_NAME}.${S3_SERVER}";
    path    "${S3_SERVER}:${S3_SERVER_PORT}";
    default "${S3_BUCKET_NAME}.${S3_SERVER}";
}

js_var $indexIsEmpty true;
# This creates the HTTP authentication header to be sent to S3
js_set $s3auth s3gateway.s3auth;
js_set $s3SecurityToken s3gateway.s3SecurityToken;
js_set $s3uri s3gateway.s3uri;

server {
    include /etc/nginx/conf.d/gateway/server_variables.conf;
    error_log /dev/stderr debug;

    # Don't display the NGINX version number because we don't want to reveal
    # information that could be used to find an exploit.
    server_tokens off;

    # Uncomment this for a HTTP header that will let you know the cache status
    # of an object.
    # add_header X-Cache-Status $upstream_cache_status;

    # Proxy caching configuration. Customize this for your needs.
    proxy_cache s3_cache;
    proxy_cache_valid 200 302 0;
    proxy_cache_valid 404 0;
    proxy_cache_valid 403 0;
    proxy_cache_methods GET HEAD;
    # When this is enabled a HEAD request to NGINX will result in a GET
    # request upstream. Unfortunately, proxy_cache_convert_head has to be
    # disabled because there is no way for the signatures generation code to
    # get access to the metadata in the GET request that is sent upstream.
    proxy_cache_convert_head off;
    proxy_cache_revalidate on;
    proxy_cache_background_update on;
    proxy_cache_lock on;
    proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
    proxy_cache_key "$request_method$host$uri";

    # If you need to support proxying range request, refer to this article:
    # https://www.nginx.com/blog/smart-efficient-byte-range-caching-nginx/

    # Do not proxy the S3 SOAP API. The S3 API has a less-documented feature
    # where the object name "soap" is used for the SOAP API. We don't allow
    # access to it.
    location /soap {
        return 404;
    }

    location /health {
        return 200;
    }

    location / {
        auth_request /aws/credentials/retrieve;
        error_log /var/log/nginx/debug.log debug;

        # Redirect to the proper location based on the client request - either
        # @s3, @s3Listing or @error405.

        js_content s3gateway.redirectToS3;
    }

    location /aws/credentials/retrieve {
        internal;
        js_content s3gateway.fetchCredentials;
    }

    location @s3 {
        # We include only the headers needed for the authentication signatures that
        # we plan to use.
        include /etc/nginx/conf.d/gateway/v4_headers.conf;

        # Don't allow any headers from the client - we don't want them messing
        # with S3 at all.
        proxy_pass_request_headers off;

        # Set the Authorization header to the AWS Signatures credentials
        proxy_set_header Authorization $s3auth;
        proxy_set_header X-Amz-Security-Token $s3SecurityToken;

        # We set the host as the bucket name to inform the S3 API of the bucket
        proxy_set_header Host $s3_host_hdr;

        # Use keep alive connections in order to improve performance
        proxy_http_version 1.1;
        proxy_set_header Connection '';

        # We strip off all of the AWS specific headers from the server so that
        # there is nothing identifying the object as having originated in an
        # object store.
        js_header_filter s3gateway.editAmzHeaders;

        # Catch all errors from S3 and sanitize them so that the user can't
        # gain intelligence about the S3 bucket being proxied.
        proxy_intercept_errors on;

        # Comment out this line to receive the error messages returned by S3
        error_page 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 420 422 423 424 426 428 429 431 444 449 450 451 500 501 502 503 504 505 506 507 508 509 510 511 =404 @error404;

        proxy_pass https://storage_urls$s3uri;
    }

    location @s3Listing {
        # We include only the headers needed for the authentication signatures that
        # we plan to use.
        include /etc/nginx/conf.d/gateway/v4_headers.conf;

        # Don't allow any headers from the client - we don't want them messing
        # with S3 at all.
        proxy_pass_request_headers off;

        # Set the Authorization header to the AWS Signatures credentials
        proxy_set_header Authorization $s3auth;
        proxy_set_header X-Amz-Security-Token $s3SecurityToken;

        # We set the host as the bucket name to inform the S3 API of the bucket
        proxy_set_header Host $s3_host_hdr;

        # Use keep alive connections in order to improve performance
        proxy_http_version 1.1;
        proxy_set_header Connection '';

        # We strip off all of the AWS specific headers from the server so that
        # there is nothing identifying the object as having originated in an
        # object store.
        js_header_filter s3gateway.editAmzHeaders;

        # Apply XSL transformation to the XML returned from S3 directory listing
        # results such that we can output an HTML directory contents list.
        xslt_stylesheet /etc/nginx/include/listing.xsl;
        xslt_types application/xml;

        # We apply an output filter to the XML input received from S3 before it
        # is passed to XSLT in order to determine if the resource is not a valid
        # S3 directory. If it isn't a valid directory, we do a dirty hack to
        # corrupt the contents of the XML causing the XSLT to fail and thus
        # nginx to return a 404 to the client. If you don't care about empty
        # directory listings for invalid directories, remove this.
        js_body_filter s3gateway.filterListResponse;

        # Catch all errors from S3 and sanitize them so that the user can't
        # gain intelligence about the S3 bucket being proxied.
        proxy_intercept_errors on;

        # Comment out this line to receive the error messages returned by S3
        error_page 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 420 422 423 424 426 428 429 431 444 449 450 451 500 501 502 503 504 505 506 507 508 509 510 511 =404 @error404;

        proxy_pass https://storage_urls$s3Uri;
    }

    location @error404 {
        # return 404;
        try_files $uri /index.html;
    }

    # Provide a hint to the client on 405 errors of the acceptable request methods
    error_page 405 @error405;
    location @error405 {
        add_header Allow "GET, HEAD" always;
        return 405;
    }
}

Objects with a unicode character under a non-unicode directory are failing with S3 but not Minio

For example, the request /a/c/あ will fail to be signed properly.

No caching when using Docker image.

Describe the bug
Using the latest Docker image, no data is being cached.

To Reproduce
Steps to reproduce the behavior:

1. Start container
   docker run --rm -ti -p 80:80 -e S3_SERVER=storage.googleapis.com -e S3_ACCESS_KEY_ID="<key>" -e S3_SECRET_KEY="<secret>" --env-file s3.env nginxinc/nginx-s3-gateway:latest-20221026

s3.env file:

S3_BUCKET_NAME=<bucket-name>
S3_SERVER_PORT=443
S3_SERVER_PROTO=https
S3_REGION=us-east-1
S3_STYLE=virtual
S3_DEBUG=false
AWS_SIGS_VERSION=4
ALLOW_DIRECTORY_LIST=true
PROVIDE_INDEX_PAGE=false
APPEND_SLASH_FOR_POSSIBLE_DIRECTORY=false
PROXY_CACHE_VALID_OK=1h
PROXY_CACHE_VALID_NOTFOUND=1m
PROXY_CACHE_VALID_FORBIDDEN=30s

2. Pull multiple files from the S3 gateway at http://localhost

I can successfully browse the S3 bucket directory structure and download objects without any issue.
Although, when downloading the same object multiple times I cannot see any performance increase from a cache hit.

3. Exec into container
   docker exec -ti <container> bash
   Run the following command:
   ls -la /var/cache/nginx/s3_proxy/

The cache directory is empty.
I also looked for looked for any disk usage increase with the command du -sh /* but no cached data is being stored in the container.

Expected behavior
According to the documentation, data should be cached when accessed multiple times and not reloaded from the remote S3 bucket at each access.

Your environment

• Version of the container used (if downloaded from Docker Hub or Github)
  Docker image: nginxinc/nginx-s3-gateway:latest-20221026
• S3 backend implementation you are using (AWS, Ceph, NetApp StorageGrid, etc)
  Google Cloud Storage
• How you are deploying Docker/Stand-alone, etc
  Docker on MacOS using Rancher Desktop.
• Authentication method (IAM, IAM with Fargate, IAM with K8S, AWS Credentials, etc)
  S3 authentication using Google Service Account with HMAC keys.

nginx s3 gateway handshake failure

Hello guys,

I am trying to use this repo to setup an s3 proxy using below settings.

My settings file (name --> my.settings) looks like -

S3_BUCKET_NAME=my-s3-bucket
S3_SERVER=s3.amazonaws.com
S3_REGION=us-east-1
S3_ACCESS_KEY_ID=myaccesskey
S3_SECRET_KEY=mysecretkey
S3_SERVER_PORT=443
S3_SERVER_PROTO=https
S3_STYLE=path
S3_DEBUG=true
AWS_SIGS_VERSION=4

I built the image as suggested in the repo using -

docker build -f Dockerfile.oss -t nginx-oss-s3-gateway .

and when I run the image using - docker run -p8080:80 --env-file ./my.settings nginx-oss-s3-gateway

the image starts up fine as expected.
I attempt to send a request to retrieve an object from the bucket - curl -v http://localhost:8080/index.html and I get back a HTTP 404. I look at nginx container that is running and see this -

2021/05/12 02:39:17 [error] 58#58: *1 peer closed connection in SSL handshake while SSL 
handshaking to upstream, client: 172.17.0.1, server: , request: "GET /index.html HTTP/1.1", 
upstream: "https://52.217.165.200:443/mys3bucket/index.html", host: "localhost:8080"
172.17.0.1 - - [12/May/2021:02:39:17 +0000] "GET /index.html HTTP/1.1" 404 146 "-" "curl/7.64.1" "-"
2021/05/12 02:39:17 [info] 58#58: *1 client 172.17.0.1 closed keepalive connection

1. Tried adding ssl_verify_client off; to the nginx.conf and reloading the config, but that was not helpful.
2. Tried adding RUN apt-get update -y && update-ca-certificates --fresh to the Dockerfile.oss and rebuilding the docker image and running it. Did not help.

I have a feeling this is because nginx is unable to verify aws s3 certificate? in which case, should I be importing the s3 cert to a trust store and pointing to nginx config? I am trying to avoid certificate pinning by downloading a specific cert.

Any advice on how to resolve this?

Also posted here - https://stackoverflow.com/questions/67496717/nginx-s3-gateway-handshake-failure

Please help.

Is it possible get list of files in the path

Hi!
Thanks for a great app!

Private HTML file opens nice:
http://127.0.0.1/2021/test.html

But when I open http://127.0.0.1/2021/ - it responses 404

Is it possible to get a list of files?

Thank you!

No useful error message emitted when running tests when docker-compose is not installed

If Docker Compose is not installed on the system, when test.sh is executed, it will exit without returning an error message.

Add support for HTTP proxies to allow for debugging requests

If it was easy to hook up the gateway to a HTTP proxy utility then we would be able to debug authentication issues more effectively. Currently, a proxy can be added manually by editing the oss/etc/nginx/templates/upstreams.conf.template file and then hardcoding in the proxy server and port. The host continues to map to the correct value because it is coded elsewhere in the configuration.

Add directory listing integration tests

Cache signing keys when using v4 signatures

Rationale
When using v4 signatures you need to create a signing key that is a composition of 4 separate HMACs. The data used to generate those HMACs includes the secret key, the date (day level resolution), the region and the service name. This signing key (HMAC digest) can be cached for as long as the date is still valid (24 hours).

If this signing operation can be done once every 24 hours, it reduces the per request computational load which in turn will reduce the overall response time.

Support for docker secrets in ENV

Would it be possible to add additional ENV options to read from a file in order to support docker secrets?

Describe the solution you'd like
Rather than setting S3_ACCESS_KEY_ID=mysecretkey

I'd like to instead set a _FILE version e.g S3_ACCESS_KEY_ID_FILE=/run/secrets/mydockersecret

Reading that file will then provide you with "mysecretkey". Having _FILE support for the S3_ACCESS_KEY_ID and S3_SECRET_KEY would be useful, as well as any future ENV items that may be expected to hold secrets.

Describe alternatives you've considered
Mounting an entire env file as a docker secret is probably a viable workaround but it's a bit hacky.

The _FILE option of env vars is how other projects have provided support for docker secrets and seems to be the "norm". For example MySQL. See "MYSQL_PASSWORD_FILE" at the following link https://docs.docker.com/engine/swarm/secrets/#advanced-example-use-secrets-with-a-wordpress-service

Filenames with brackets () return 404 errors

Hi,

I'm running the the stand-alone container on one of my servers. Problem is, whenever one of the files stored in the S3 container contains brackets, the gateway will return a 404 error.

An example:

Both image.jpg and image(1).jpg exists in the same bucket in the same folder.

This:

curl -I "http://127.0.0.1:8011/image.jpg"

Returns:

HTTP/1.1 200 OK
Server: nginx
Date: Wed, 31 Aug 2022 17:45:21 GMT
Content-Type: image/jpeg
Content-Length: 466483
Connection: keep-alive
Last-Modified: Wed, 31 Aug 2022 17:44:50 GMT
ETag: "b34cf0ebd821a9f892b361e6812571eb"
Accept-Ranges: bytes

While this:

curl -I "http://127.0.0.1:8011/image(1).jpg"

Returns:

HTTP/1.1 404 Not Found
Server: nginx
Date: Wed, 31 Aug 2022 17:45:27 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive

The gateway log shows this:

127.0.0.1 - - [31/Aug/2022:17:45:21 +0000] "HEAD /image.jpg HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
2022/08/31 17:45:21 [info] 61#61: *159 client 127.0.0.1 closed keepalive connection
127.0.0.1 - - [31/Aug/2022:17:45:27 +0000] "HEAD /image(1).jpg HTTP/1.1" 404 0 "-" "curl/7.64.0" "-"
2022/08/31 17:45:27 [info] 61#61: *161 client 127.0.0.1 closed keepalive connection

Server details:

Debian 8.3.0-6 GNU/Linux 10 (buster)
Linux version 4.19.0-14-cloud-amd64

Possible to support OPTIONS requests for CORS?

Is your feature request related to a problem? Please describe.

Just tried this project out for the first time and it's great. Docs were clear and easy to get started.

One issue I ran into is that it appears OPTIONS requests currently are rejected by the JS module: https://github.com/nginxinc/nginx-s3-gateway/blob/master/common/etc/nginx/include/s3gateway.js#L410-L415

It would be great to be able to enable CORS settings since that would make it easier to use this proxy directly from the browser for client-side applications.

Is it possible to add support there, along with some means to configure CORS settings?

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered

I tried adding CORS settings via an nginx conf file under /etc/nginx/conf.d/ but I think the JS module is taking precedence and rejecting these requests before my settings take effect.

That said, it's definitely possible I am configuring something wrong...I am by no means an expert on either NGINX or CORS.

Here is the dockerfile I was using:

FROM ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest

COPY nginx.conf /etc/nginx/conf.d/default.conf

And this is the config file i tried:

server {
    echo
    listen       80;
    server_name  localhost;

    #access_log  /var/log/nginx/host.access.log  main;

    location / {
        echo "hello"
        if ($request_method = 'OPTIONS') {
           echo "method is options"
           add_header 'Access-Control-Allow-Origin' '*';
           add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
           #
           # Custom headers and headers various browsers *should* be OK with but aren't
           #
           add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range';
           #
           # Tell client that this pre-flight info is valid for 20 days
           #
           add_header 'Access-Control-Max-Age' 1728000;
           add_header 'Content-Type' 'text/plain; charset=utf-8';
           add_header 'Content-Length' 0;
           return 204;
        }
        if ($request_method = 'POST') {
            add_header 'Access-Control-Allow-Origin' '*' always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always;
            add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range' always;
            add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range' always;
        }
        if ($request_method = 'GET') {
            add_header 'Access-Control-Allow-Origin' '*' always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always;
            add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range' always;
            add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range' always;
        }

        # root   /usr/share/nginx/html;
        # index  index.html index.htm;

    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

Apologies in advance if those are way off base, but maybe someone can steer me in a better direction if this is not the right way to configure these settings.

Additional context

Example:

nginx-proxy ➸ curl -I -X OPTIONS http://localhost/example.txt
HTTP/1.1 405 Not Allowed
Server: nginx
Date: Thu, 13 Oct 2022 17:33:32 GMT
Content-Type: text/html
Content-Length: 150
Connection: keep-alive
Allow: GET, HEAD

Remove double/multiple forward slashes from the request path

When using the gateway in directory listing mode, if you have a double slash in the path, the request won't find the folder or files.

I suggest removing extra slashes from the path using a rewrite, a map in the config or from JavaScript.

Example:
https://example.com/pathFoo//file.txt would become https://example.com/pathFoo/file.txt
https://example.com/pathFoo//subpathFoo/// would become https://example.com/pathFoo/subpathFoo/

Unprivileged Containers do not work as non-root

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:
docker run --env-file ./s3-settings.env -p 8081:8080 --user 1000:1000 nginxinc/nginx-s3-gateway:unprivileged-oss-20230210

I get the following error indicating its trying to start on port 80, which is not allowed by non-root. It looks like the conf file is not writable. The 'nginxinc/nginx-unprivileged:1.23-alpine` image works fine with the same settings.

Directory Listing Enabled: false
Provide Index Pages Enabled:
Append slash for directory enabled:
Stripping the following headers from responses: x-amz-;
CORS Enabled: 0
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: ERROR: /etc/nginx/templates exists, but /etc/nginx/conf.d is not writable
/docker-entrypoint.sh: Launching /docker-entrypoint.d/22-enable_js_fetch_trusted_certificate.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)
2023/02/20 21:48:02 [emerg] 1#1: bind() to 0.0.0.0:80 failed (13: Permission denied)

Expected behavior
Container should start on 8080

Your environment

• Version of the repo - a specific commit or tag: unprivileged-oss-20230210
• Version of the container used (if downloaded from Docker Hub or Github) : Dockerhub
• S3 backend implementation you are using (AWS, Ceph, NetApp StorageGrid, etc) : Minio
• How you are deploying Docker/Stand-alone, etc : Racher Desktop and K8s
• NGINX type (OSS/Plus): OSS
• Authentication method (IAM, IAM with Fargate, IAM with K8S, AWS Credentials, etc): IAM

Additional context
Add any other context about the problem here. Any log files you want to share.

Sensitive Information
Be sure to redact any sensitive information such as your AWS authentication keys.

Running with IAM Profile on EKS

Hey,
I'm running the nginx-s3-gateway over EKS, while the POD is not able to work when I'm attaching the SA to an IAM Role.
I'm getting the following error:
2022/05/30 14:11:38 [error] 59#59: *396 js exception: Error: unhandled promise rejection: Error: read timed out, client: 127.0.0.1, server: , request: "GET /3_5.txt HTTP/1.1", subrequest: "/aws/credentials/retrieve", host: "127.0.0.1:7000"

We use the same framework as explained: https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html
The POD is aware of the SA, but can't get it to work :(
AWS_ROLE_ARN='arn:aws:iam::xxx:role/nginx-s3-gateway-D2A48-nginx-s3-gateway'
AWS_SIGS_VERSION='4'
AWS_WEB_IDENTITY_TOKEN_FILE='/var/run/secrets/eks.amazonaws.com/serviceaccount/token'

Change MIME type coming from S3

I would like to use this proxy to list files in my S3.
However some of the files are in format such as YAML, or simply gzipped plain text which the browser downloads instead of displaying them.

Would it be possible to set the MIME type according to mime.types and default_type instead of the type coming from S3 request?
Thanks

Use Instance profile credential

Is it possible to use temporary credential retrieved from instance profile/role?

Unit tests on Windows hosts

I had a number of issues with unit test on windows hosts.
Key ones were the space in c:\Program Files" and MINGW path conversion
docker/cli#2204 (comment)

one weird issue in adding data to minio is the container name was showing up as:

"${test_compose_project}"-minio-1 vs _minio_1
I can include on #7 or as a separate pr.

I have resolved all except for non-ascii characters. Any hints?

Testing object: HEAD b/ブツブツ.txt
Response code didn't match expectation. Request [HEAD http://localhost:8989/b/ブツブツ.txt] Expected [200] Actual [404]
curl command: /mingw64/bin/curl -s -o /dev/null -w '%{http_code}' --head 'http://localhost:8989/b/ブツブツ.txt'
Error running tests - outputting container logs
ngt-minio-1 |
ngt-minio-1 | You are running an older version of MinIO released 1 year ago
ngt-minio-1 | Update: Run mc admin update
ngt-minio-1 |
ngt-minio-1 |
ngt-minio-1 | Attempting encryption of all config, IAM users and policies on MinIO backend
ngt-minio-1 | Endpoint: http://172.18.0.2:9000 http://127.0.0.1:9000
ngt-minio-1 |
ngt-minio-1 | Object API (Amazon S3 compatible):
ngt-minio-1 | Go: https://docs.min.io/docs/golang-client-quickstart-guide
ngt-minio-1 | Java: https://docs.min.io/docs/java-client-quickstart-guide
ngt-minio-1 | Python: https://docs.min.io/docs/python-client-quickstart-guide
ngt-minio-1 | JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
ngt-minio-1 | .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide
ngt-nginx-s3-gateway-1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
ngt-minio-client-1 | bucket-1.minio [REQUEST s3.HeadObject] [2022-08-08T15:46:22.968] [Client IP: 172.18.0.4]
ngt-minio-client-1 | bucket-1.minio HEAD /a.txt
ngt-minio-client-1 | bucket-1.minio Proto: HTTP/1.1
ngt-minio-client-1 | bucket-1.minio Host: bucket-1.minio
ngt-minio-client-1 | bucket-1.minio Authorization: AWS AKIAIOSFODNN7EXAMPLE:fTI5YYzR6CQ5KAsKutGxnP+hBgE=
ngt-minio-client-1 | bucket-1.minio Content-Length: 0
ngt-minio-client-1 | bucket-1.minio Date: Mon, 08 Aug 2022 15:46:22 GMT
ngt-minio-client-1 | bucket-1.minio
ngt-nginx-s3-gateway-1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
ngt-nginx-s3-gateway-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/00-check-for-required-env.sh
ngt-nginx-s3-gateway-1 | S3 Backend Environment
ngt-nginx-s3-gateway-1 | Access Key ID: AKIAIOSFODNN7EXAMPLE
ngt-nginx-s3-gateway-1 | Origin: http://bucket-1.minio:9000
ngt-nginx-s3-gateway-1 | Region: us-east-1
ngt-nginx-s3-gateway-1 | Addressing Style: virtual
ngt-nginx-s3-gateway-1 | AWS Signatures Version: v2
ngt-nginx-s3-gateway-1 | DNS Resolvers: 127.0.0.11
ngt-nginx-s3-gateway-1 | Directory Listing Enabled: 0
ngt-nginx-s3-gateway-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
ngt-nginx-s3-gateway-1 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
ngt-nginx-s3-gateway-1 | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
ngt-nginx-s3-gateway-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
ngt-nginx-s3-gateway-1 | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
ngt-minio-client-1 | bucket-1.minio [RESPONSE] [2022-08-08T15:46:22.969] [ Duration 940µs ↑ 39 B ↓ 334 B ]
ngt-minio-client-1 | bucket-1.minio 200 OK
ngt-minio-client-1 | bucket-1.minio X-Amz-Bucket-Region: us-east-1
ngt-minio-client-1 | bucket-1.minio X-Amz-Request-Id: 1709692DC564B13D
ngt-minio-client-1 | bucket-1.minio Accept-Ranges: bytes
ngt-minio-client-1 | bucket-1.minio Content-Security-Policy: block-all-mixed-content
ngt-minio-client-1 | bucket-1.minio ETag: "00000000000000000000000000000000-1"
ngt-minio-client-1 | bucket-1.minio Last-Modified: Wed, 27 Jul 2022 20:17:26 GMT
ngt-minio-client-1 | bucket-1.minio Server: MinIO
ngt-minio-client-1 | bucket-1.minio Vary: Origin
ngt-minio-client-1 | bucket-1.minio X-Xss-Protection: 1; mode=block
ngt-minio-client-1 | bucket-1.minio Content-Length: 24
ngt-minio-client-1 | bucket-1.minio Content-Type: text/plain
ngt-minio-client-1 | bucket-1.minio
ngt-minio-client-1 | bucket-1.minio
ngt-minio-client-1 | bucket-1.minio [REQUEST s3.HeadObject] [2022-08-08T15:46:23.106] [Client IP: 172.18.0.4]
ngt-minio-client-1 | bucket-1.minio HEAD /b/c/d.txt
ngt-minio-client-1 | bucket-1.minio Proto: HTTP/1.1
ngt-minio-client-1 | bucket-1.minio Host: bucket-1.minio
ngt-minio-client-1 | bucket-1.minio Authorization: AWS AKIAIOSFODNN7EXAMPLE:kdC8I23LkTNy+vL7a60K6JaOufs=
ngt-minio-client-1 | bucket-1.minio Content-Length: 0
ngt-minio-client-1 | bucket-1.minio Date: Mon, 08 Aug 2022 15:46:23 GMT
ngt-minio-client-1 | bucket-1.minio
ngt-minio-client-1 | bucket-1.minio [RESPONSE] [2022-08-08T15:46:23.107] [ Duration 422µs ↑ 39 B ↓ 334 B ]
ngt-minio-client-1 | bucket-1.minio 200 OK
ngt-minio-client-1 | bucket-1.minio Content-Length: 97
ngt-minio-client-1 | bucket-1.minio Content-Type: text/plain
ngt-minio-client-1 | bucket-1.minio ETag: "00000000000000000000000000000000-1"
ngt-minio-client-1 | bucket-1.minio Server: MinIO
ngt-minio-client-1 | bucket-1.minio Vary: Origin
ngt-minio-client-1 | bucket-1.minio X-Xss-Protection: 1; mode=block
ngt-minio-client-1 | bucket-1.minio Accept-Ranges: bytes
ngt-minio-client-1 | bucket-1.minio Content-Security-Policy: block-all-mixed-content
ngt-minio-client-1 | bucket-1.minio Last-Modified: Wed, 27 Jul 2022 20:17:26 GMT
ngt-minio-client-1 | bucket-1.minio X-Amz-Bucket-Region: us-east-1
ngt-minio-client-1 | bucket-1.minio X-Amz-Request-Id: 1709692DCD9AA4FA
ngt-minio-client-1 | bucket-1.minio
ngt-minio-client-1 | bucket-1.minio
ngt-nginx-s3-gateway-1 | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v4_js_vars.conf.template to /etc/nginx/conf.d/gateway/v4_js_vars.conf
ngt-nginx-s3-gateway-1 | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v2_headers.conf.template to /etc/nginx/conf.d/gateway/v2_headers.conf
ngt-nginx-s3-gateway-1 | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v2_js_vars.conf.template to /etc/nginx/conf.d/gateway/v2_js_vars.conf
ngt-nginx-s3-gateway-1 | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v4_headers.conf.template to /etc/nginx/conf.d/gateway/v4_headers.conf
ngt-nginx-s3-gateway-1 | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/upstreams.conf.template to /etc/nginx/conf.d/upstreams.conf
ngt-nginx-s3-gateway-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
ngt-nginx-s3-gateway-1 | /docker-entrypoint.sh: Configuration complete; ready for start up
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:22 [notice] 1#1: using the "epoll" event method
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:22 [notice] 1#1: nginx/1.23.1
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:22 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:22 [notice] 1#1: OS: Linux 5.10.16.3-microsoft-standard-WSL2
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:22 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:22 [notice] 1#1: start worker processes
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:22 [notice] 1#1: start worker process 59
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:22 [notice] 1#1: start cache manager process 60
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:22 [notice] 1#1: start cache loader process 61
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:22 [info] 59#59: *1 client closed connection while waiting for request, client: 172.18.0.1, server: 0.0.0.0:80
ngt-nginx-s3-gateway-1 | 172.18.0.1 - - [08/Aug/2022:15:46:22 +0000] "HEAD / HTTP/1.1" 404 0 "-" "curl/7.84.0" "-"
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:22 [info] 59#59: *2 client 172.18.0.1 closed keepalive connection
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:22 [info] 59#59: *3 js: S3 Request URI: HEAD /a.txt
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:22 [info] 59#59: *3 js: AWS v2 Auth Signing String: [HEAD
ngt-nginx-s3-gateway-1 |
ngt-nginx-s3-gateway-1 |
ngt-nginx-s3-gateway-1 | Mon, 08 Aug 2022 15:46:22 GMT
ngt-minio-client-1 | bucket-1.minio [REQUEST s3.HeadObject] [2022-08-08T15:46:23.175] [Client IP: 172.18.0.4]
ngt-minio-client-1 | bucket-1.minio HEAD /b/e.txt
ngt-minio-client-1 | bucket-1.minio Proto: HTTP/1.1
ngt-minio-client-1 | bucket-1.minio Host: bucket-1.minio
ngt-minio-client-1 | bucket-1.minio Date: Mon, 08 Aug 2022 15:46:23 GMT
ngt-minio-client-1 | bucket-1.minio Authorization: AWS AKIAIOSFODNN7EXAMPLE:LPRC71ZP0c1Hq6zK/16ATxV8MxQ=
ngt-minio-client-1 | bucket-1.minio Content-Length: 0
ngt-minio-client-1 | bucket-1.minio
ngt-minio-client-1 | bucket-1.minio [RESPONSE] [2022-08-08T15:46:23.175] [ Duration 910µs ↑ 39 B ↓ 334 B ]
ngt-minio-client-1 | bucket-1.minio 200 OK
ngt-minio-client-1 | bucket-1.minio Accept-Ranges: bytes
ngt-minio-client-1 | bucket-1.minio Content-Type: text/plain
ngt-minio-client-1 | bucket-1.minio ETag: "00000000000000000000000000000000-1"
ngt-minio-client-1 | bucket-1.minio Last-Modified: Wed, 27 Jul 2022 20:17:26 GMT
ngt-minio-client-1 | bucket-1.minio X-Amz-Bucket-Region: us-east-1
ngt-minio-client-1 | bucket-1.minio Content-Length: 43
ngt-minio-client-1 | bucket-1.minio Content-Security-Policy: block-all-mixed-content
ngt-nginx-s3-gateway-1 | /bucket-1/a.txt]
ngt-nginx-s3-gateway-1 | 172.18.0.1 - - [08/Aug/2022:15:46:22 +0000] "HEAD /a.txt HTTP/1.1" 200 0 "-" "curl/7.84.0" "-"
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:22 [info] 59#59: *3 client 172.18.0.1 closed keepalive connection
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:23 [info] 59#59: *5 js: S3 Request URI: HEAD /a.txt
ngt-nginx-s3-gateway-1 | 172.18.0.1 - - [08/Aug/2022:15:46:23 +0000] "HEAD /a.txt?some=param&that=should&be=stripped HTTP/1.1" 200 0 "-" "curl/7.84.0" "-"
ngt-minio-client-1 | bucket-1.minio Server: MinIO
ngt-minio-client-1 | bucket-1.minio Vary: Origin
ngt-minio-client-1 | bucket-1.minio X-Amz-Request-Id: 1709692DD1ACE851
ngt-minio-client-1 | bucket-1.minio X-Xss-Protection: 1; mode=block
ngt-minio-client-1 | bucket-1.minio
ngt-minio-client-1 | bucket-1.minio
ngt-minio-client-1 | bucket-1.minio [REQUEST s3.HeadObject] [2022-08-08T15:46:23.366] [Client IP: 172.18.0.4]
ngt-minio-client-1 | bucket-1.minio HEAD /b/
ngt-minio-client-1 | bucket-1.minio Proto: HTTP/1.1
ngt-minio-client-1 | bucket-1.minio Host: bucket-1.minio
ngt-minio-client-1 | bucket-1.minio Authorization: AWS AKIAIOSFODNN7EXAMPLE:6BTWvC4+NQhPLzcqg44sm80ODkE=
ngt-minio-client-1 | bucket-1.minio Content-Length: 0
ngt-minio-client-1 | bucket-1.minio Date: Mon, 08 Aug 2022 15:46:23 GMT
ngt-minio-client-1 | bucket-1.minio
ngt-minio-client-1 | bucket-1.minio [RESPONSE] [2022-08-08T15:46:23.366] [ Duration 265µs ↑ 39 B ↓ 227 B ]
ngt-minio-client-1 | bucket-1.minio 403 Forbidden
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:23 [info] 59#59: *5 client 172.18.0.1 closed keepalive connection
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:23 [info] 59#59: *6 js: S3 Request URI: HEAD /b/c/d.txt
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:23 [info] 59#59: *6 js: AWS v2 Auth Signing String: [HEAD
ngt-nginx-s3-gateway-1 |
ngt-nginx-s3-gateway-1 |
ngt-nginx-s3-gateway-1 | Mon, 08 Aug 2022 15:46:23 GMT
ngt-nginx-s3-gateway-1 | /bucket-1/b/c/d.txt]
ngt-nginx-s3-gateway-1 | 172.18.0.1 - - [08/Aug/2022:15:46:23 +0000] "HEAD /b/c/d.txt HTTP/1.1" 200 0 "-" "curl/7.84.0" "-"
ngt-minio-client-1 | bucket-1.minio Content-Length: 0
ngt-minio-client-1 | bucket-1.minio Content-Security-Policy: block-all-mixed-content
ngt-minio-client-1 | bucket-1.minio Server: MinIO
ngt-minio-client-1 | bucket-1.minio Vary: Origin
ngt-minio-client-1 | bucket-1.minio X-Amz-Bucket-Region: us-east-1
ngt-minio-client-1 | bucket-1.minio X-Amz-Request-Id: 1709692DDD1868B7
ngt-minio-client-1 | bucket-1.minio X-Xss-Protection: 1; mode=block
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:23 [info] 59#59: *6 client 172.18.0.1 closed keepalive connection
ngt-minio-client-1 | bucket-1.minio Accept-Ranges: bytes
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:23 [info] 59#59: *8 js: S3 Request URI: HEAD /b/e.txt
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:23 [info] 59#59: *8 js: AWS v2 Auth Signing String: [HEAD
ngt-nginx-s3-gateway-1 |
ngt-nginx-s3-gateway-1 |
ngt-nginx-s3-gateway-1 | Mon, 08 Aug 2022 15:46:23 GMT
ngt-nginx-s3-gateway-1 | /bucket-1/b/e.txt]
ngt-nginx-s3-gateway-1 | 172.18.0.1 - - [08/Aug/2022:15:46:23 +0000] "HEAD /b/e.txt HTTP/1.1" 200 0 "-" "curl/7.84.0" "-"
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:23 [info] 59#59: *8 client 172.18.0.1 closed keepalive connection
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:23 [info] 59#59: *10 js: S3 Request URI: HEAD /b/e.txt
ngt-nginx-s3-gateway-1 | 172.18.0.1 - - [08/Aug/2022:15:46:23 +0000] "HEAD /b/e.txt HTTP/1.1" 200 0 "-" "curl/7.84.0" "-"
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:23 [info] 59#59: *10 client 172.18.0.1 closed keepalive connection
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:23 [info] 59#59: *11 js: S3 Request URI: HEAD /b//e.txt
ngt-nginx-s3-gateway-1 | 172.18.0.1 - - [08/Aug/2022:15:46:23 +0000] "HEAD /b//e.txt HTTP/1.1" 200 0 "-" "curl/7.84.0" "-"
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:23 [info] 59#59: *11 client 172.18.0.1 closed keepalive connection
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:23 [info] 59#59: *12 js: S3 Request URI: HEAD /b/
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:23 [info] 59#59: *12 js: AWS v2 Auth Signing String: [HEAD
ngt-nginx-s3-gateway-1 |
ngt-nginx-s3-gateway-1 |
ngt-nginx-s3-gateway-1 | Mon, 08 Aug 2022 15:46:23 GMT
ngt-nginx-s3-gateway-1 | /bucket-1/]
ngt-nginx-s3-gateway-1 | 172.18.0.1 - - [08/Aug/2022:15:46:23 +0000] "HEAD /b/????.txt HTTP/1.1" 404 0 "-" "curl/7.84.0" "-"
ngt-nginx-s3-gateway-1 | 2022/08/08 15:46:23 [info] 59#59: *12 client 172.18.0.1 closed keepalive connection
ngt-minio-client-1 | bucket-1.minio
ngt-minio-client-1 | bucket-1.minio
▶ Cleaning up Docker compose environment

Replace deprecated `set-output` command with environment file

Is your feature request related to a problem? Please describe.

In .github/workflow/main.yml, set-output command is deprecated and will be disabled soon.

Describe the solution you'd like

Please upgrade to using Environment Files. For more information, see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Describe alternatives you've considered
N/A

Additional context
N/A

Cloudflare R2 error

Describe the bug
I'm getting the following errors when using Cloudflare R2;
2022/12/11 16:15:46 [error] 64#64: *1 SSL_do_handshake() failed (SSL: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:SSL alert number 40) while SSL handshaking to upstream
2022/12/11 16:15:46 [warn] 64#64: *1 upstream server temporarily disabled while SSL handshaking to upstream

I can fix the errors and it works fine if I add the following line to default.conf (inside location @s3):
proxy_ssl_server_name on;
and "hardcoding" my R2 domain in proxy pass:
proxy_pass https://abcdefghijklmnopqrstuvwxz1234567.r2.cloudflarestorage.com$s3uri;
but having to do that everytime I restart the container is a PITA.

Posting this issue so maybe a kind sould can fix it or post a better solution.

Your environment

• Version of the repo - a specific commit or tag
  latest-20221104

• Version of the container used (if downloaded from Docker Hub or Github)
  ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest

• S3 backend implementation you are using (AWS, Ceph, NetApp StorageGrid, etc)
  Cloudflare R2

• How you are deploying Docker/Stand-alone, etc
  Docker

• NGINX type (OSS/Plus)
  OSS

• Authentication method (IAM, IAM with Fargate, IAM with K8S, AWS Credentials, etc)
  Secret and access keys

Add script to allow for stand-alone demo using Minio