rdegges/flask-dynamo

DynamoDB integration for Flask.

Hey, I'm Randall!

It's great to meet you. I'm a long-time software developer, speaker, author, and entrepreneur.

• 🛠️ I’m currently working at Snyk leading Developer & Security Relations
• 📫 How to reach me: [email protected] (personal) || [email protected] (work)
• 💻 You can see what I've written about on my personal website

flask-dynamo's Issues

Dynamo within application context before and after using tables property

Discovered a state issue with flask-dynamo when running unit test(pytest and unitest) on my flask application. Initially the error was no table was named %s when flask-dynamo attempted to query so I checked the application context state.

from flask import _app_ctx_stack as stack
cox = stack.top
print dir(ctx)

This was in hopes of looking for keys such as dynamo_tables, dynamo_connection, etc; however none of these keys were there. Along with the missing keys, dynamo.app was nil as well. However in the unit test code setUp, dynamo.app was still set and able to create tables.

Our hack fix was to call flask.current_app, set that as dynamo.app, and than call dynamo.tables, which recovered the state due to the check for variables in context when retrieving this property. Is this something that should be set earlier?

Some other flask packages like sqlalchemy and security add the to flask.app.extensions to store specific states. Maybe this would help to store dynamo in context?

This is the text referring to flask.app.extensions (https://github.com/pallets/flask/blob/67e391921c50f02190be8407d4a02efe8d2d28f6/flask/app.py)

#: a place where extensions can store application specific state. For
#: example this is where an extension could store database engines and
#: similar things. For backwards compatibility extensions should register
#: themselves like this::
#:
#: if not hasattr(app, 'extensions'):
#: app.extensions = {}
#: app.extensions['extensionname'] = SomeObject()
#:
#: The key must match the name of the extension module. For example in
#: case of a "Flask-Foo" extension in flask_foo, the key would be
#: 'foo'.
#:
#: .. versionadded:: 0.7
self.extensions = {}

Destroy Tables After Testing

We should destroy the Dynamo tables after testing.

Can't load table configuration without default boto config

The table configuration for flask-dynamo requires a valid connection to be created - This block of code

app.config['DYNAMO_TABLES'] = [
    Table('users', schema=[HashKey('username')]),
    Table('groups', schema=[HashKey('name')]),
]

Causes the following boto code to create a new connection with no parameters:
https://github.com/boto/boto/blob/48c5d178d83bc5f2aa6a8e488396e41b2d867ceb/boto/dynamodb2/table.py#L106

If you have no default credentials in ~/.boto, then this fails with a really cryptic error:

/lib/python2.7/site-packages/boto/auth.pyc in get_auth_handler(host, config, provider, requested_capability)
    973             'No handler was ready to authenticate. %d handlers were checked.'
    974             ' %s '
--> 975             'Check your credentials' % (len(names), str(names)))
    976 
    977     # We select the last ready auth handler that was loaded, to allow users to

NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV4Handler'] Check your credentials

My expectation is that credentials in the flask configuration would be used for configuration instead of ~/.boto or env variables. This is especially unusual if I have local dynamodb configured - it will still go and make a connection with my live credentials before falling back to using the local database.

Use a persistent Dynamo connection across requests

It looks like the default behaviour of storing the connection and tables in _app_ctx_stack.top is per-request, and not persistent across multiple requests.

For optimal performance, the connection should really be re-used, as it's actually implemented as a thread-safe pool of connections in boto. Otherwise, the only re-use of connections is between multiple queries within the same request.

What do you think about changing this, possibly as a opt-in configurable behaviour? It looks like it's safe to simply store the connection in current_app.extensions['dynamo'] instead.

Use DynamoDB local

Is it possible to use DynamoDB Local for development, and then switch it to use the service version when deploying?

Allow for pointing to local dynamodb

This is an enhancement. #6 has the changes, but it needs to be rebased to master. I am making this issue to track getting it done. Thanks!

Update Docs for Boto3 Support

The documentation currently provides only information on using this library with boto2. The code samples should all be updated to reflect boto3 support (which is now included on the develop branch).

We should also mention that this library is Python 3 compatible.

import error

File "flaskapp.py", line 4
from flask-dynamo import abc
^
SyntaxError: invalid syntax

please answer
Thanks

is that possible to use other env name instead of 'AWS_REGION' to define where dynamo db is located?

I would like to use the dynamodb database in the different region instead of the one that specific from 'AWS_REGION' because I use 'AWS_REGION' for other stuff and want to store the database in somewhere else.

use GlobalSecondaryIndexes not work

i use boto3 or boto3 session it query mobile data,but use flask-dynamo not work ,query data is None, why ?
it's my code

    from flask import current_app
    dynamo = current_app.extensions['dynamo'] 
    response = dynamo.tables[ACCOUNT_TABLE].query(IndexName='mobile-index',
                                                  KeyConditionExpression=Key('mobile').eq(mobile))
    account_data = response.get("Item")
    import boto3
    dynamodb = boto3.resource('dynamodb')
    account = dynamodb.Table(ACCOUNT_TABLE)
    resp = account.query(IndexName='mobile-index', KeyConditionExpression=Key('mobile').eq(mobile))
    print(account_data, 80*'#')
    print(resp, 80*'^')
    from boto3.session import Session
    bs = Session()
    s_dy = bs.resource('dynamodb')
    s_ac = s_dy.Table(ACCOUNT_TABLE)
    res = s_ac.query(IndexName='mobile-index', KeyConditionExpression=Key('mobile').eq(mobile))
    print(res, 80*'-')

New release with the existing tables check on create_all

Hi

I've being struggling a bit why I was getting an error about existing tables when using the create_all method as the docs explained until I saw that the current released version doesn't include this feature, but it is already done on master in the following commit:

9b67ff4

I'm just wondering when a new release with this fix will be commited as I need this feature on my project.

Thanks

AWS access key id & secret should not be required

boto just "does the right thing" when running on an EC2 instance if access key id and secret are not set at all but an iam instance profile is enabled on the instance.

So it seems like these lines should just be removed because these parameters aren't actually required.

Thoughts?

A Flask-User DBAdapter?

Recent developments at AWS (Lambda, in particular) have me wanting to use DynamoDB for users/role/etc storage. Flask-User looked like an obvious all-in-one kind of solution, but it has no DynamoDB support.

I started writing my own using boto3 directly but then thought maybe using flask-dynamo might be a better solution... what do you suggest? Any interest in helping out with this?

Bug in documentation

The example in the documentation contains this:

app.config['DYNAMO_TABLES'] = [
    {
         TableName='users',
         KeySchema=[dict(AttributeName='username', KeyType='HASH')],
         AttributeDefinitions=[dict(AttributeName='username', AttributeType='S')],
         ProvisionedThroughput=dict(ReadCapacityUnits=5, WriteCapacityUnits=5)
    }, {
         TableName='groups',
         KeySchema=[dict(AttributeName='name', KeyType='HASH')],
         AttributeDefinitions=[dict(AttributeName='name', AttributeType='S')],
         ProvisionedThroughput=dict(ReadCapacityUnits=5, WriteCapacityUnits=5)
    }
 ]

Which is invalid syntax, because the keys in the dict are not quoted. A quick fix is to change the braces to dict( ... )

Also, the example endpoint:

@app.route('/create_user')
def create_user():
    dynamo.tables['users'].put_item(data={
        'username': 'rdegges',
        'first_name': 'Randall',
        'last_name': 'Degges',
        'email': '[email protected]',
    })

...does not work, because the parameter for put_item() should be Item instead of data, and the view should return some valid response instead of None, for example return "", or better jsonify the value returned by put_item().

Although simple to fix, I find these issues worrying, because they are at least 2 years old, and apparently nobody noticed them yet. Is the library maintained?

Cannot install boto and flask-dynamo at the same time (e.g. with pip)

flask-dynamo is imported when you run pip install. This means that it tries to import the boto library as well.
If you install both with pip at the same time - e.g.

pip install boto flask-dynamo

then it fails to install because boto is not installed during the setup.py egg_info command. See this output:

$ pip install boto flask-dynamo
Downloading/unpacking boto
  Downloading boto-2.34.0-py2.py3-none-any.whl (1.3MB): 1.3MB downloaded
Downloading/unpacking flask-dynamo
  Downloading flask-dynamo-0.0.2.tar.gz (148kB): 148kB downloaded
  Running setup.py (path:/home/alastair/code/drogon/newenv/build/flask-dynamo/setup.py) egg_info for package flask-dynamo
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/home/alastair/code/drogon/newenv/build/flask-dynamo/setup.py", line 18, in <module>
        from flask_dynamo import __version__ as version
      File "flask_dynamo/__init__.py", line 9, in <module>
        from .manager import Dynamo
      File "flask_dynamo/manager.py", line 6, in <module>
        from boto.dynamodb2 import connect_to_region
    ImportError: No module named boto.dynamodb2
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/home/alastair/code/drogon/newenv/build/flask-dynamo/setup.py", line 18, in <module>

    from flask_dynamo import __version__ as version

  File "flask_dynamo/__init__.py", line 9, in <module>

    from .manager import Dynamo

  File "flask_dynamo/manager.py", line 6, in <module>

    from boto.dynamodb2 import connect_to_region

ImportError: No module named boto.dynamodb2

I guess a solution would be to put __version__ in a place that doesn't import all of boto.

manager.Dynamo.init_app

So I bumped to 0.1.1 and am running into the RuntimeError from _get_app when calling self.connection in init_app. It breaks since current_app and self.app do not return. Current app is not set since this is at load time. My question however is if self.app should be set in init_app?

My config

common/
    dynamo = flask_dynamo.Dynamo()

app.py
    dynamo.init_app(app)

Mocking DynamoDB

I would like to use moto to mock DynamoDB when testing my Flask app. Any hints on how this should be done? My initial attempt is:

from moto import mock_dynamodb2
from flask_dynamo import Dynamo
from flask import Flask


@mock_dynamodb2
def test_init():
    app = Flask(__name__)
    dynamo = Dynamo()
    dynamo.init_app(app)
    dynamo.create_all()
    assert True

But this ends up to error:

botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost:8000/"

Mocking the database directly without using flask_dynamo seems to be working.

Flask-Secuity using Flask-Dynamo for persistence

Have you thought of working with the Flask-Security framework to add Flask-Dynamo as a data persistence layer? If you're interested in such a thing I'd be interested in helping.