waderobson/s3-auth

Generate s3 authentication headers for munki

s3-auth's Issues

wiki modification to aws s3 sync command

hey man -

love this - was just walking through a few things and found out the aws s3 sync command is finicky about the exclude. Please consider changing the following line in your wiki.

From:
aws s3 sync /path/to/munki/ s3://<S3_BUCKET_GOES_HERE>/ --exclude '*.git/*' --exclude '.DS_Store' --delete

To:
aws s3 sync /path/to/munki/ s3://<S3_BUCKET_GOES_HERE>/ --exclude "*.git/*" --exclude "*.DS_Store*" --delete

Thanks!

S3-auth not working with Munki 4 Release Candidate 1

Even though I'm seeing commits for Python 3 compatibility, it seems to just bunk out. Anything obvious I'm missing?

sudo managedsoftwareupdate -vvv
Password:
Managed Software Update Tool
Copyright 2010-2019 The Munki Project
https://github.com/munki/munki

Starting...
    No CA cert info provided, so nothing to add to System keychain.
    No client cert info provided, so no client keychain will be created.
Checking for available updates...
    No client id specified. Requesting SERIAL...
    Manifest base URL is: https://URLDETAILS/munki_repo/manifests/
    Getting manifest SERIAL...
    Options: {'url': 'https://URLDETAILS/munki_repo/manifests/SERIAL', 'file': '/Library/Managed Installs/manifests/SERIAL.download', 'follow_redirects': 'none', 'ignore_system_proxy': False, 'can_resume': False, 'additional_headers': {'User-Agent': 'managedsoftwareupdate/4.0.0.3881 Darwin/19.0.0'}, 'download_only_if_changed': True, 'cache_data': {
    etag = "\"2171375bed808ff2289b6e8e2f3178de\"";
    "last-modified" = "Fri, 06 Dec 2019 20:55:21 GMT";
}, 'logging_function': <function display_debug2 at 0x112d06f80>}
    Processing options through middleware
ERROR: Unexpected error in updatecheck:
Traceback (most recent call last):
  File "/usr/local/munki/managedsoftwareupdate", line 1141, in <module>
    main()
  File "/usr/local/munki/managedsoftwareupdate", line 862, in main
    client_id=unicode_or_str(options.id))
  File "/usr/local/munki/munkilib/updatecheck/core.py", line 82, in check
    mainmanifestpath = manifestutils.get_primary_manifest(client_id)
  File "/usr/local/munki/munkilib/updatecheck/manifestutils.py", line 175, in get_primary_manifest
    manifest = get_manifest(clientidentifier, suppress_errors=True)
  File "/usr/local/munki/munkilib/updatecheck/manifestutils.py", line 122, in get_manifest
    manifesturl, manifestpath, message=message)
  File "/usr/local/munki/munkilib/fetch.py", line 440, in munki_resource
    verify=verify)
  File "/usr/local/munki/munkilib/fetch.py", line 393, in getResourceIfChangedAtomically
    message=message, resume=resume, follow_redirects=follow_redirects)
  File "/usr/local/munki/munkilib/fetch.py", line 525, in getHTTPfileIfChangedAtomically
    follow_redirects=follow_redirects)
  File "/usr/local/munki/munkilib/fetch.py", line 246, in get_url
    options = middleware.process_request_options(options)
  File "/usr/local/munki/middleware_s3.py", line 117, in process_request_options
    headers = s3_auth_headers(options['url'])
  File "/usr/local/munki/middleware_s3.py", line 97, in s3_auth_headers
    signing_key = get_signature_key(SECRET_KEY, datestamp, REGION, SERVICE)
  File "/usr/local/munki/middleware_s3.py", line 49, in get_signature_key
    kdate = sign(('AWS4' + key).encode('utf-8'), datestamp)
TypeError: can only concatenate str (not "NoneType") to str

Authentication not passing through

The error I was receiving looked like this:

iMac:~ admin$ sudo managedsoftwareupdate -v
Managed Software Update Tool
Copyright 2010-2017 The Munki Project
https://github.com/munki/munki

Starting...
Checking for available updates...
Getting manifest ms/ms...
ERROR: Unexpected error in updatecheck:
Traceback (most recent call last):
File "/usr/local/munki/managedsoftwareupdate", line 1055, in
main()
File "/usr/local/munki/managedsoftwareupdate", line 781, in main
client_id=options.id.decode('UTF-8'))
File "/usr/local/munki/munkilib/updatecheck/core.py", line 76, in check
mainmanifestpath = manifestutils.get_primary_manifest(client_id)
File "/usr/local/munki/munkilib/updatecheck/manifestutils.py", line 156, in get_primary_manifest
manifest = get_manifest(clientidentifier)
File "/usr/local/munki/munkilib/updatecheck/manifestutils.py", line 114, in get_manifest
manifesturl, manifestpath, message=message)
File "/usr/local/munki/munkilib/fetch.py", line 425, in munki_resource
verify=verify)
File "/usr/local/munki/munkilib/fetch.py", line 378, in getResourceIfChangedAtomically
message=message, resume=resume, follow_redirects=follow_redirects)
File "/usr/local/munki/munkilib/fetch.py", line 510, in getHTTPfileIfChangedAtomically
follow_redirects=follow_redirects)
File "/usr/local/munki/munkilib/fetch.py", line 236, in get_url
options = middleware.process_request_options(options)
File "/usr/local/munki/middleware_s3.py", line 114, in process_request_options
headers = s3_auth_headers(options['url'])
File "/usr/local/munki/middleware_s3.py", line 94, in s3_auth_headers
signing_key = get_signature_key(SECRET_KEY, datestamp, REGION, SERVICE)
File "/usr/local/munki/middleware_s3.py", line 45, in get_signature_key
kdate = sign(('AWS4' + key).encode('utf-8'), datestamp)
TypeError: cannot concatenate 'str' and 'NoneType' objects

It turns out the latest version of s3-auth changes the location where it reads the preferences. The old location for the settings was /Library/Preferences/com.github.waderobson.s3-auth and the new location is /Library/Preferences/ManagedInstalls. This makes sense as the rest of the munki preference keys are stored there too.

Possible Wiki update?

Thanks for producing and maintaining this :)

I was getting 403 access denied errors when I first tried using this.
After some digging around I found post on MacAdmins slack recommending adding this preference:

sudo defaults write /Library/Preferences/ManagedInstalls S3Endpoint 's3.your.s3.domain'

After adding the preference the 403 error went way and everything worked :) So I was wondering if it's worth adding to the wiki?

Issue with redirects in s3.aws.com urls

There seems to be an issue with redirections depending on the URL of the S3 bucket.

For example, if a bucket has a name blah.s3.amazonaws.com, S3 may return a HTTP 307 when requested. This can happen if your AWS region is anything but us-east.

If you are using the ap-southeast-2 region, and request blah.s3.amazonaws.com, you are redirected to blah.s3-ap-southeast-2.amazonaws.com

As a result, you end up with this error upon running managedsoftwareupdate:

Retrieving list of software for this machine...
ERROR: Could not retrieve manifest YOUR_MANIFEST from the server: HTTP result 307: temporarily redirected
ERROR: Could not retrieve managed install primary manifest.
Finishing.

As a workaround, you can modify the following function to look for a different string. Instead of if 's3.amazonaws.com' in options['url']: you could substitute region specific information such as if 's3-ap-southeast-2.amazonaws.com' in options['url']:

def process_request_options(options):
    """Make changes to options dict and return it.
       This is the fuction that munki calls."""
    if 's3.amazonaws.com' in options['url']:
        headers = s3_auth_headers(options['url'])
        options['additional_headers'].update(headers)
    return options

So, two options:

1. Add region info to that search string from the region data already in /Library/Preferences/com.github.waderobson.s3-auth.plist
2. Add support for URL redirection adding headers to anything directed from *.s3.amazonaws.com

Allow link timeout to be set

For the life of me I can't seem to work out out to set the link timeout to be greater than the 15 minute default.

I'm doing my best to wrap my head around the way this middleware and AWS4 authentication header work, but I'm just not there.

Is there a simple way to inject Amz-Expires=XXX into the link request? It seems like it's defaulting to a 900 second timeout, but I'm having issues pulling down large updates like an OS install.

I'll keep poking a stick at it, but I'm sure there have to be few folks that have hit this limit.

Thanks for the good work.

S3Auth doesn't work in Munki

Hello,

First of all thanks for your work on s3-auth, i have a private repo on S3 bucket and i have follow your process to make it work with munki, but it's not working, below you will find the error i'm getting with munki:

XeroxBrain:munki admin$ sudo /usr/local/munki/managedsoftwareupdate
Managed Software Update Tool
Copyright 2010-2020 The Munki Project
https://github.com/munki/munki

Starting...
Checking for available updates...
Retrieving list of software for this machine...
ERROR: Could not retrieve manifest Packages from the server: HTTP result 403: forbidden
ERROR: Could not retrieve managed install primary manifest.
Finishing...

My bucket URL is working as i have try it by making the bucket repo public.
My Access key ID and Secret access key are also good.
I have copy middleware_s3.py in /usr/local/munki/ folder

I have also set
sudo defaults write /Library/Preferences/ManagedInstalls AccessKey 'xxxxxxx'
sudo defaults write /Library/Preferences/ManagedInstalls 'xxxxxx'
sudo defaults write /Library/Preferences/ManagedInstalls Region 'eu-west-1’
sudo defaults write /Library/Preferences/ManagedInstalls SoftwareRepoURL ‘xxxxxxxxx'

Thanks for you help.

Error running with munki4

Hi,

I am getting this error running the middleware with munki4.

\nAuthorizationHeaderMalformedThe authorization header is malformed; the authorization header requires three components: Credential, SignedHeaders, and Signature.

I get error with munki3 as well.

ERROR: Unexpected error in updatecheck:
Traceback (most recent call last):
File "/usr/local/munki/managedsoftwareupdate", line 1140, in
main()
File "/usr/local/munki/managedsoftwareupdate", line 861, in main
client_id=options.id.decode('UTF-8'))
File "/usr/local/munki/munkilib/updatecheck/core.py", line 81, in check
mainmanifestpath = manifestutils.get_primary_manifest(client_id)
File "/usr/local/munki/munkilib/updatecheck/manifestutils.py", line 166, in get_primary_manifest
manifest = get_manifest(clientidentifier, suppress_errors=True)
File "/usr/local/munki/munkilib/updatecheck/manifestutils.py", line 114, in get_manifest
manifesturl, manifestpath, message=message)
File "/usr/local/munki/munkilib/fetch.py", line 428, in munki_resource
verify=verify)
File "/usr/local/munki/munkilib/fetch.py", line 381, in getResourceIfChangedAtomically
message=message, resume=resume, follow_redirects=follow_redirects)
File "/usr/local/munki/munkilib/fetch.py", line 513, in getHTTPfileIfChangedAtomically
follow_redirects=follow_redirects)
File "/usr/local/munki/munkilib/fetch.py", line 235, in get_url
options = middleware.process_request_options(options)
File "/usr/local/munki/middleware_s3.py", line 117, in process_request_options
headers = s3_auth_headers(options['url'])
File "/usr/local/munki/middleware_s3.py", line 90, in s3_auth_headers
credential_scope = '{}/{}/{}/aws4_request'.format(datestamp, REGION, SERVICE)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 0: ordinal not in range(128)