googlecloudplatform/gcs-oauth2-boto-plugin

Auth plugin allowing use the use of OAuth 2.0 credentials for Google Cloud Storage in the Boto library.

gcs-oauth2-boto-plugin

gcs-oauth2-boto-plugin is a Python application whose purpose is to behave as an auth plugin for the boto auth plugin framework for use with OAuth 2.0 credentials for the Google Cloud Platform. This plugin is compatible with both user accounts and service accounts, and its functionality is essentially a wrapper around oauth2client with the addition of automatically caching tokens for the machine in a thread- and process-safe fashion.

For more information about how to use this plugin to access Google Cloud Storage via boto in your application, see the GCS documentation.

If you wish to use this plugin without using the PyPI install as instructed in the documentation (e.g., for development), then you will need to manually acquire the modules from the requirements.txt file.

When using this plugin, you must specify a client ID and secret. We offer the following methods for providing this information; if multiple methods are used, we will choose them in the following order:

1. .boto config, if not set
2. environment variables (OAUTH2_CLIENT_ID and OAUTH2_CLIENT_SECRET), if not set
3. CLIENT_ID and CLIENT_SECRET values set by SetFallbackClientIdAndSecret function.

Service accounts are supported via key files in either JSON or .p12 format. The file is first interpreted as JSON, with .p12 format as a fallback.

The default locking mechanism used is threading.Lock. You can switch to using another locking mechanism by calling SetLock. Example:

SetLock(multiprocessing.Manager().Lock())

Before submitting any code, please run the tests (e.g., by creating a new virtualenv and running the following commands from the root of this repository):

pip install -r requirements.txt
PYTHONPATH="." python -m gcs_oauth2_boto_plugin.test_oauth2_client

gcs-oauth2-boto-plugin's Issues

Service account authentication requires PyOpenSSL. Please install this library and try again.

My python scripts works as expected for about an hour or so. Then it gives me the error:

Service account authentication requires PyOpenSSL. Please install this library and try again.

"/usr/local/lib/python2.7/dist-packages/boto/storage_uri.py", line 342, in get_key generation=self.generation) File "/usr/local/lib/python2.7/dist-packages/boto/gs/bucket.py", line 108, in get_key query_args_l=query_args_l) File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 200, in _get_key_internal query_args=query_args) File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 665, in make_request retry_handler=retry_handler File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1071, in make_request retry_handler=retry_handler) File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 927, in _mexe request.authorize(connection=self) File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 377, in authorize connection._auth_handler.add_auth(self, **kwargs) File "/usr/local/lib/python2.7/dist-packages/gcs_oauth2_boto_plugin/oauth2_plugin.py", line 69, in add_auth self.oauth2_client.GetAuthorizationHeader() File "/usr/local/lib/python2.7/dist-packages/gcs_oauth2_boto_plugin/oauth2_client.py", line 345, in GetAuthorizationHeader return 'Bearer %s' % self.GetAccessToken().token File "/usr/local/lib/python2.7/dist-packages/gcs_oauth2_boto_plugin/oauth2_client.py", line 316, in GetAccessToken access_token = self.FetchAccessToken() File "/usr/local/lib/python2.7/dist-packages/gcs_oauth2_boto_plugin/oauth2_client.py", line 387, in FetchAccessToken credentials = self.GetCredentials() File "/usr/local/lib/python2.7/dist-packages/gcs_oauth2_boto_plugin/oauth2_client.py", line 432, in GetCredentials 'Service account authentication requires PyOpenSSL. Please install ' MissingDependencyError: Service account authentication requires PyOpenSSL. Please install this library and try again.

pyopenssl is installed and as said the script works for some time and then not any more. any ideas?

Question: pyOpenSSL dependency?

Hi! We use this plugin in https://github.com/grow/grow to support one-line deployments to Google Cloud Storage via boto. We package this library with our distribution, but because of the pyOpenSSL dependency in this package's setup.py, installs take significantly longer because of the compilation (and non-pure Python code) that that package contains.

Doing a cursory audit of gcs-oauth2-boto-plugin it doesn't really look like pyOpenSSL modules are called directly anywhere. I also tested the functionality (oauth2 login to GCS via boto) without installing pyOpenSSL and all seems to work well. So if it's not needed, can we remove it from setup.py?

Update non-browser-interactive auth URIs to use oauth2.googleapis.com variants

As part of https://issuetracker.google.com/issues/79946689, we're updating gsutil and the libraries it relies on to use the newer oauth2.googleapis.com-based URIs, away from the current accounts.google.com and googleapis.com/oauth2 equivalents.

Browser-interactive endpoints (like the auth endpoint, https://accounts.google.com/o/oauth2/auth) can remain unchanged.

Occurrences can be found by greping for "accounts.google.com" or "googleapis.com/oauth2" within the source files.

Error with boto / service key file

Here's the code I'm using:

boto.config.add_section('Credentials')
boto.config.set('Credentials', 'gs_service_key_file', '/path/to/service/account.json')
boto.config.set('Credentials', 'gs_service_client_id', '[email protected]')
connection = boto.connect_s3(provider='google')
my_bucket = connection.get_bucket('my-cool-bucket')

and the error I'm seeing:

  File "/usr/local/lib/python2.7/site-packages/gcs_oauth2_boto_plugin/oauth2_client.py", line 322, in GetAccessToken
    access_token = self.FetchAccessToken()
  File "/usr/local/lib/python2.7/site-packages/gcs_oauth2_boto_plugin/oauth2_client.py", line 393, in FetchAccessToken
    credentials = self.GetCredentials()
  File "/usr/local/lib/python2.7/site-packages/gcs_oauth2_boto_plugin/oauth2_client.py", line 523, in GetCredentials
    scopes=[DEFAULT_SCOPE])
TypeError: __init__() takes at least 3 arguments (4 given)

It looks like the init signature for ServiceAccountCredentials takes an additional parameter called signer which is not being sent. Is my approach incorrect or does this need to be changed?

Update development status in setup.py

Currently the development status shows Development Status :: 7 - Inactive, which means this package is deprecated. We should change to an active status like Development Status :: 5 - Production/Stable or Development Status :: 6 - Mature.

Python3 wheels on pypi

Can python wheels be built and released for this up on https://pypi.org/project/gcs-oauth2-boto-plugin/#files

404 on GCS documentation

https://cloud.google.com/storage/docs/gspythonlibrary
Need to update the link GCS documentation.

Service accounts do not provide `client_secret`

This might just be a documentation issue, but the docs and the readme state that this is compatible with google service accounts. However, service accounts do not provide a client_secrets but instead a private key that needs to be used in an OAuth2 flow to sign a JWT token to get an access token.

As far as I can tell, this flow is not supported by this plugin. Either new functionality needs to be added, or documentation needs to be updated.

incompatible with python 3 syntax

version 1.14

>>> import gcs_oauth2_boto_plugin
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/gcs_oauth2_boto_plugin/__init__.py", line 26, in <module>
    from gcs_oauth2_boto_plugin import oauth2_plugin
  File "/usr/local/lib/python3.6/site-packages/gcs_oauth2_boto_plugin/oauth2_plugin.py", line 22, in <module>
    from gcs_oauth2_boto_plugin import oauth2_client
  File "/usr/local/lib/python3.6/site-packages/gcs_oauth2_boto_plugin/oauth2_client.py", line 125
    except KeyError, e:
                   ^
SyntaxError: invalid syntax

Oauth2GoogleClient object has no attribute cache_key_base

I'm trying to port gcs-oauth2-boto-plugin to make it Python2+Python3 compatible, but ran into this issue when cloning from master:

File "/enplore-code/environments/enplore-worker-aircraft-dfdr/site-packages/boto/connection.py", line 377, in authorize
connection._auth_handler.add_auth(self, **kwargs)
File "/enplore-code/environments/enplore-worker-aircraft-dfdr/site-packages/gcs_oauth2_boto_plugin/oauth2_plugin.py", line 69, in add_auth
self.oauth2_client.GetAuthorizationHeader()
File "/enplore-code/environments/enplore-worker-aircraft-dfdr/site-packages/gcs_oauth2_boto_plugin/oauth2_client.py", line 345, in GetAuthorizationHeader
return 'Bearer %s' % self.GetAccessToken().token
File "/enplore-code/environments/enplore-worker-aircraft-dfdr/site-packages/gcs_oauth2_boto_plugin/oauth2_client.py", line 310, in GetAccessToken
cache_key = self.CacheKey()
File "/enplore-code/environments/enplore-worker-aircraft-dfdr/site-packages/gcs_oauth2_boto_plugin/oauth2_client.py", line 335, in CacheKey
h.update(self.cache_key_base)
AttributeError: 'Oauth2GoogleClient' object has no attribute 'cache_key_base'

I'm using a boto config, with a gs service key file:
gs_service_key_file = /etc/enplore/vagrant-etc/testdemo-s652y8s5.json

The issue doesn't exist when installing from pip but appeared in the January 27 commit (53df987)

I've spent a few hours on this, and would love some help. From what I can tell:

oauth2_helper.py

if json_key:
  return oauth2_client.Oauth2GoogleClient(
    oauth2client.client.GoogleCredentials.from_stream(private_key_filename)
  )

Doesn't return a oauth2_client with cache_key_base attribute which is later needed for CacheKey():

 def CacheKey(self):
"""Computes a cache key.
The cache key is computed as the SHA1 hash of the refresh token for user
accounts, or the hash of the gs_service_client_id for service accounts,
which satisfies the FileSystemTokenCache requirement that cache keys do not
leak information about token values.
Returns:
  A hash key.
"""
    h = sha1()
    h.update(self.cache_key_base.encode('utf-8'))
    return h.hexdigest()

However, I think there is a bunch of stuff missing on the Oauth2GoogleClient which is available on OAuth2ServiceAccountClient, like access_token_cache, FetchAccessToken etc.

I'm gonna do a pull request on my own branch and link to it from here, hopefully someone other then me can tell if I re-implemented it correctly (I've tested on py2 and py3 and it works on both)

Import gcs-oauth2-boto-plugin results in KeyError

When following the steps on https://cloud.google.com/storage/docs/xml-api/gspythonlibrary I get the following error

import gcs_oauth2_boto_plugin
  File "/usr/local/lib/python3.6/site-packages/gcs_oauth2_boto_plugin/__init__.py", line 26, in <module>
    from gcs_oauth2_boto_plugin import oauth2_plugin
  File "/usr/local/lib/python3.6/site-packages/gcs_oauth2_boto_plugin/oauth2_plugin.py", line 22, in <module>
    from gcs_oauth2_boto_plugin import oauth2_client
  File "/usr/local/lib/python3.6/site-packages/gcs_oauth2_boto_plugin/oauth2_client.py", line 125
    except KeyError, e:
                   ^
SyntaxError: invalid syntax

Python script looks like:

import boto
import gcs_oauth2_boto_plugin
import os
import shutil
import StringIO
import tempfile
import time

# URI scheme for Cloud Storage.
GOOGLE_STORAGE = 'gs'
# URI scheme for accessing local files.
LOCAL_FILE = 'file'

# Fallback logic. In https://console.cloud.google.com/
# under Credentials, create a new client ID for an installed application.
# Required only if you have not configured client ID/secret in
# the .boto file or as environment variables.
CLIENT_ID = 'your client id'
CLIENT_SECRET = 'your client secret'
gcs_oauth2_boto_plugin.SetFallbackClientIdAndSecret(CLIENT_ID, CLIENT_SECRET)

Python version: 3.6.0

OAuth2Auth will create a OAuth2GCEClient even if it is authorized for no scopes.

https://github.com/GoogleCloudPlatform/gcs-oauth2-boto-plugin/blob/master/gcs_oauth2_boto_plugin/oauth2_plugin.py#L38 doesn't do any checking of the scopes that the resulting OAuth2GCEClient supports. Thus, if a GCE VM is created with no scopes, we'll return an OAuth2Auth handler that looks like a valid credential but in actuality can't access anything.

GCE credentials requested outside a GCE instance

When trying to access GCS from a GCE instance I have create the following /etc/boto.cfg:

[GoogleCompute]
service_account=default

But when I want to use the same configuration from a non-GCE instance, it will give the error message: GCE credentials requested outside a GCE instance

I was expecting gsutil to try to authenticate as an anonymous caller if it fails with default service account credentials.

Am I missing something here?

Implement AuthHandler.sign_string

OAuth2ServiceAccountAuth should implement the sign_string method, so that generate_url can work. The code below works for me, but it should be abstracted and I'm not sure what are all the auth methods that the plugin supports:

  def sign_string(self, string_to_sign):
    import base64
    from oauth2client.crypt import Signer
    signer = Signer.from_string(self.oauth2_client._private_key)
    return base64.b64encode(signer.sign(string_to_sign))

Failing to install gsutil and latest gcs-oauth2-boto-plugin

Downloading gcs-oauth2-boto-plugin-1.13.tar.gz
Running setup.py (path:/tmp/pip-build-vqbH5h/gcs-oauth2-boto-plugin/setup.py) egg_info for package gcs-oauth2-boto-plugin
error in gcs-oauth2-boto-plugin setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers
Complete output from command python setup.py egg_info:
error in gcs-oauth2-boto-plugin setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers

But if I do pip install gcs-oauth2-boto-plugin==1.12 it succeeds.

Missing google-auth-* in setup.py

Hi,
Recently this PR ( #62 ) added two new dependencies:

• google-auth==2.17.0
• google-auth-httplib2>=0.2.0.

They are added to requirements.txt but not setup.py. This caused Module not Found error for any downstream packages trying to use this package. Please consider doing so.

Import fails since oauth2client update.

It seems that there has been a breaking change to oauth2client (#401) that is affecting gcs-oauth2-boto-plugin. Pinning oauth2client to 1.5.2 seems to fix it, but presumably gcs-oauth2-boto-plugin should be updated to use the new hotness.

code does not work with python-3.x

oauth2_client.py does:

      fd = os.open(cache_file, flags, 0600)

python-3.x requires that to be 0o0600. same bug can be seen in test_oauth2_client.py.

Migrate from oauth2client to google-auth

oauth2client is deprecated: https://google-auth.readthedocs.io/en/latest/oauth2client-deprecation.html

Update release on PyPi

It looks like the PyPi release of this package (https://pypi.python.org/pypi/gcs-oauth2-boto-plugin/1.8) is behind master here. Can a release be made on PyPi to fix the oauth2client issues that were fixed by this commit 6492539?

Moving to boto3

What is the timescale from moving from boto to boto3? We routinely encounter a warning in boto (through gcs-oauth2-boto-plugin) that uses the deprecated import imp.

How to get this working?

Hi,
Can you please assist????
I am trying to use this plugin using docker. For some reason I don't seem to be able to get my python code to authenticate with GCP.
I keep getting the following error message:

Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/boto/storage_uri.py", line 569, in create_bucket
conn = self.connect()
File "/usr/local/lib/python2.7/dist-packages/boto/storage_uri.py", line 140, in connect
**connection_args)
File "/usr/local/lib/python2.7/dist-packages/boto/gs/connection.py", line 47, in init
suppress_consec_slashes=suppress_consec_slashes)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 194, in init
validate_certs=validate_certs, profile_name=profile_name)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 569, in init
host, config, self.provider, self._required_auth_capability())
File "/usr/local/lib/python2.7/dist-packages/boto/auth.py", line 1021, in get_auth_handler
'Check your credentials' % (len(names), str(names)))
boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 3 handlers were checked. ['OAuth2Auth', 'OAuth2ServiceAccountAuth', 'HmacAuthV1Handler'] Check your credentials

Steps:

python
import boto
import gcs_oauth2_boto_plugin
OAUTH2_CLIENT_ID = 'client_id'
OAUTH2_CLIENT_SECRET = 'client_secret'
gcs_oauth2_boto_plugin.SetFallbackClientIdAndSecret(OAUTH2_CLIENT_ID, OAUTH2_CLIENT_SECRET)
project_id='project_id' //I have my project id
header_values = {"x-goog-project-id": project_id}
uri = boto.storage_uri('my-bucket', 'gs')
uri.create_bucket()

Consider tagging releases on github

Hi, packager of http://pkgsrc.se/net/py-gcs-oauth2-boto-plugin here.

It would be easier to follow new releases if they were tagged, with release notes, in Github. Or is there a different recommended way to keep up with upstream releases?

Boto Compatibility for OAuth2 Plugin

I am one of the maintainers of Boto. A few weeks ago we began to merge backward-compatible Python 3.3 and 3.4 support into our develop branch. Most modules now support Python 3 with passing unit and integration tests, and I am hoping to do a release soon. I am reaching out to you to ask if you would test our develop branch with gcs-oauth2-boto-plugin.

Testing against Python 2.x will help to ensure that we don't accidentally break third party modules like gcs-oauth2-boto-plugin. If you would like, you may also try out the Python 3 support and let us know any feedback. More information about the Python 3 support can be found in this boto-users post:

https://groups.google.com/forum/#!topic/boto-users/vcvoy8zrfP8

I appreciate any time that you can put into helping the Boto project with this.

This request is part of boto/boto#2437

Add Contributing file