zdurham/s3-upload-github-action

Upload github action for sending files to s3

s3-upload-github-action

Upload github action for sending files to s3

Usage

This is a super straightforward action that uses the aws cli tool to copy a particular file to an s3 bucket. The file can come from your code directly or generated by an earlier part of your github actions flow. Check out the example below to get started.

Please note that each env var is required. It is recommended to put your AWS credentials in as repository secrets, as well as your bucket name.

# inside .github/workflows/your-action.yml
name: Add File to Bucket
on: push

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    steps:
   - uses: actions/checkout@master
   
   - name: Upload file to bucket
   uses: zdurham/s3-upload-github-action@master
   with:
     args: --acl public-read
   env:
    FILE: ./lambda.zip
    AWS_REGION: 'us-east-1'
    S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
    S3_KEY: ${{ secrets.S3_KEY }}
    AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
    AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

s3-upload-github-action's Issues

Folder upload

Does this upload also folders? Or only files can be uploaded

Documentation Clarification

I see a requirement for an S3 key to be stored in github secrets. Where does that map to in s3? or in other words do you know where I would find the s3 key in AWS?

Unknown options error

I have a workflow that is erroring with the following error

Unknown options: [path redacted]

I’ve redacted the path but it’s just the target path of the file.

The step is as follows:

      - name: Upload backup to s3 bucket
        if: ${{ contains(github.ref, 'main') || env.backup_enabled == 'yes' }}
        uses: zdurham/s3-upload-github-action@master
        with:
          args: --acl private
        env:
          FILE: ${{ env.src_file }}
          AWS_REGION: 'eu-west-2'
          S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
          S3_KEY: ${{ env.tgt_file }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

I have the very same configuration working in other repos. In fact it’s copy and pasted from a repo where it’s working without errors.

Why might this error occur?

Wanted to upload only files which are changed or new that commit

Docker File Failing To Build

`Build container for action use: '/home/runner/work/_actions/zdurham/s3-upload-github-action/master/Dockerfile'.
/usr/bin/docker build -t c9a4a5:15422b252de74fda9fe62e80ae2f9479 -f "/home/runner/work/_actions/zdurham/s3-upload-github-action/master/Dockerfile" "/home/runner/work/_actions/zdurham/s3-upload-github-action/master"
Sending build context to Docker daemon 8.704kB

Step 1/5 : FROM python:3.7-alpine
3.7-alpine: Pulling from library/python
31e352740f53: Already exists
cfcc276e4459: Pulling fs layer
8455851b3630: Pulling fs layer
c41f20535e03: Pulling fs layer
f053299a814f: Pulling fs layer
f053299a814f: Waiting
c41f20535e03: Verifying Checksum
c41f20535e03: Download complete
cfcc276e4459: Verifying Checksum
cfcc276e4459: Download complete
8455851b3630: Verifying Checksum
8455851b3630: Download complete
f053299a814f: Verifying Checksum
f053299a814f: Download complete
cfcc276e4459: Pull complete
8455851b3630: Pull complete
c41f20535e03: Pull complete
f053299a814f: Pull complete
Digest: sha256:94c0a807604a8da09cf6597f11080ad8cdb62f9b4dc1e8cb8134e487f33aa15f
Status: Downloaded newer image for python:3.7-alpine
---> fe33630f7d7a
Step 2/5 : RUN pip install --quiet --no-cache-dir awscli
---> Running in ae26dd84fe4a
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [62 lines of output]
    running egg_info
    writing lib3/PyYAML.egg-info/PKG-INFO
    writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
    writing top-level names to lib3/PyYAML.egg-info/top_level.txt
    /tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
    !!
    
            ********************************************************************************
            The license_file parameter is deprecated, use license_files instead.
    
            By 2023-Oct-30, you need to update your project and remove deprecated calls
            or your builds will no longer be supported.
    
            See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
            ********************************************************************************
    
    !!
      parsed = self.parsers.get(option_name, lambda x: x)(value)
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
        main()
      File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line [11](https://github.com/TrustBooks/wc-journal-entry/actions/runs/5578753580/jobs/10193371383#step:2:11)8, in get_requires_for_build_wheel
        return hook(config_settings)
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
        return self._get_build_requires(config_settings, requirements=['wheel'])
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
        self.run_setup()
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 338, in run_setup
        exec(code, locals())
      File "<string>", line 295, in <module>
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 107, in setup
        return distutils.core.setup(**attrs)
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 185, in setup
        return run_commands(dist)
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
        dist.run_commands()
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
        self.run_command(cmd)
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/dist.py", line [12](https://github.com/TrustBooks/wc-journal-entry/actions/runs/5578753580/jobs/10193371383#step:2:12)34, in run_command
        super().run_command(command)
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
        cmd_obj.run()
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 3[14](https://github.com/TrustBooks/wc-journal-entry/actions/runs/5578753580/jobs/10193371383#step:2:14), in run
        self.find_sources()
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
        mm.run()
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 551, in run
        self.add_defaults()
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
        sdist.add_defaults(self)
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
        super().add_defaults()
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
        self._add_defaults_ext()
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
        self.filelist.extend(build_ext.get_source_files())
      File "<string>", line 201, in get_source_files
      File "/tmp/pip-build-env-h0xzup84/overlay/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
        raise AttributeError(attr)
    AttributeError: cython_sources
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Notice: A new release of pip is available: 23.0.1 -> 23.2
Notice: To update, run: pip install --upgrade pip
The command '/bin/sh -c pip install --quiet --no-cache-dir awscli' returned a non-zero code: 1

Warning: Docker build failed with exit code 1, back off 5.775 seconds before retry.
/usr/bin/docker build -t c9a4a5:15422b252de74fda9fe62e80ae2f9479 -f "/home/runner/work/_actions/zdurham/s3-upload-github-action/master/Dockerfile" "/home/runner/work/_actions/zdurham/s3-upload-github-action/master"
Sending build context to Docker daemon 8.704kB

Step 1/5 : FROM python:3.7-alpine
---> fe33630f7d7a
Step 2/5 : RUN pip install --quiet --no-cache-dir awscli
---> Running in f078dca3485a
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [62 lines of output]
    running egg_info
    writing lib3/PyYAML.egg-info/PKG-INFO
    writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
    writing top-level names to lib3/PyYAML.egg-info/top_level.txt
    /tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
    !!
    
            ********************************************************************************
            The license_file parameter is deprecated, use license_files instead.
    
            By 2023-Oct-30, you need to update your project and remove deprecated calls
            or your builds will no longer be supported.
    
            See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
            ********************************************************************************
    
    !!
      parsed = self.parsers.get(option_name, lambda x: x)(value)
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
        main()
      File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 1[18](https://github.com/TrustBooks/wc-journal-entry/actions/runs/5578753580/jobs/10193371383#step:2:18), in get_requires_for_build_wheel
        return hook(config_settings)
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
        return self._get_build_requires(config_settings, requirements=['wheel'])
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
        self.run_setup()
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 338, in run_setup
        exec(code, locals())
      File "<string>", line 295, in <module>
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 107, in setup
        return distutils.core.setup(**attrs)
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 185, in setup
        return run_commands(dist)
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line [20](https://github.com/TrustBooks/wc-journal-entry/actions/runs/5578753580/jobs/10193371383#step:2:20)1, in run_commands
        dist.run_commands()
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
        self.run_command(cmd)
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 1234, in run_command
        super().run_command(command)
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
        cmd_obj.run()
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 314, in run
        self.find_sources()
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 3[22](https://github.com/TrustBooks/wc-journal-entry/actions/runs/5578753580/jobs/10193371383#step:2:22), in find_sources
        mm.run()
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 551, in run
        self.add_defaults()
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
        sdist.add_defaults(self)
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
        super().add_defaults()
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
        self._add_defaults_ext()
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
        self.filelist.extend(build_ext.get_source_files())
      File "<string>", line 201, in get_source_files
      File "/tmp/pip-build-env-w2u1offc/overlay/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
        raise AttributeError(attr)
    AttributeError: cython_sources
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Notice: A new release of pip is available: 23.0.1 -> 23.2
Notice: To update, run: pip install --upgrade pip
The command '/bin/sh -c pip install --quiet --no-cache-dir awscli' returned a non-zero code: 1

Warning: Docker build failed with exit code 1, back off 8.411 seconds before retry.
/usr/bin/docker build -t c9a4a5:15422b252de74fda9fe62e80ae2f9479 -f "/home/runner/work/_actions/zdurham/s3-upload-github-action/master/Dockerfile" "/home/runner/work/_actions/zdurham/s3-upload-github-action/master"
Sending build context to Docker daemon 8.704kB

Step 1/5 : FROM python:3.7-alpine
---> fe33630f7d7a
Step 2/5 : RUN pip install --quiet --no-cache-dir awscli
---> Running in 5bb2e88359b9
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [62 lines of output]
    running egg_info
    writing lib3/PyYAML.egg-info/PKG-INFO
    writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
    writing top-level names to lib3/PyYAML.egg-info/top_level.txt
    /tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/config/setupcfg.py:[29](https://github.com/TrustBooks/wc-journal-entry/actions/runs/5578753580/jobs/10193371383#step:2:29)3: _DeprecatedConfig: Deprecated config in `setup.cfg`
    !!
    
            ********************************************************************************
            The license_file parameter is deprecated, use license_files instead.
    
            By 2023-Oct-[30](https://github.com/TrustBooks/wc-journal-entry/actions/runs/5578753580/jobs/10193371383#step:2:30), you need to update your project and remove deprecated calls
            or your builds will no longer be supported.
    
            See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
            ********************************************************************************
    
    !!
      parsed = self.parsers.get(option_name, lambda x: x)(value)
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
        main()
      File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
        return hook(config_settings)
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
        return self._get_build_requires(config_settings, requirements=['wheel'])
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
        self.run_setup()
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 338, in run_setup
        exec(code, locals())
      File "<string>", line 295, in <module>
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 107, in setup
        return distutils.core.setup(**attrs)
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 185, in setup
        return run_commands(dist)
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
        dist.run_commands()
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
        self.run_command(cmd)
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 1234, in run_command
        super().run_command(command)
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
        cmd_obj.run()
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/command/egg_info.py", line [31](https://github.com/TrustBooks/wc-journal-entry/actions/runs/5578753580/jobs/10193371383#step:2:31)4, in run
        self.find_sources()
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/command/egg_info.py", line [32](https://github.com/TrustBooks/wc-journal-entry/actions/runs/5578753580/jobs/10193371383#step:2:32)2, in find_sources
        mm.run()
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 551, in run
        self.add_defaults()
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
        sdist.add_defaults(self)
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
        super().add_defaults()
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
        self._add_defaults_ext()
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/_distutils/command/sdist.py", line [33](https://github.com/TrustBooks/wc-journal-entry/actions/runs/5578753580/jobs/10193371383#step:2:33)6, in _add_defaults_ext
        self.filelist.extend(build_ext.get_source_files())
      File "<string>", line 201, in get_source_files
      File "/tmp/pip-build-env-zv4uke2g/overlay/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line [107](https://github.com/TrustBooks/wc-journal-entry/actions/runs/5578753580/jobs/10193371383#step:2:107), in __getattr__
        raise AttributeError(attr)
    AttributeError: cython_sources
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Notice: A new release of pip is available: 23.0.1 -> 23.2
Notice: To update, run: pip install --upgrade pip
The command '/bin/sh -c pip install --quiet --no-cache-dir awscli' returned a non-zero code: 1

Error: Docker build failed with exit code 1`

Container action is only supported on Linux

This action does not work for MacOS. Can you update to work with MacOS?

upload of multiple files

is it possible to upload multiple files with the same action or should i use it multiple times?
will it work if i put the FILE variable multiple times?