rayepps/dynofunc

Functional library for creating and sending dynamo requests on top of boto3

dynofunc's Issues

Add support for boto `ValidationException` in error factory

Getting this error when passing bad query arguments:

try:
            return runner(client, description)
        except ClientError as err:
>           raise factory(err)
E           dynamof.core.exceptions.UnknownDatabaseException: An unkonwn exception occured when executing request to dynamo: Query condition missed key schema element - ValidationException

Need to add a ValidationException in the factory so it can get caught and handled.

Cannot craete a hash or range key with number type

Currently when specifying hash or range keys on a table or index they are always assumed to be of type string.

https://github.com/rayepps/dynamof/blob/928f4cd4216c19a8dc8055ed983202e4f6f8d597/dynamof/core/args.py#L92

We should allow for creating keys with type number.

Missing `begins_with` and `between` key condition

The Ask

The attr module currently only supports some of the basic conditions (==, >, <, >=, <=). Dynamo also supports special BETWEEN and a begins_with functions/operators. Lets support those as well.

Dynamo Key Conditions Docs

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html#Query.KeyConditionExpressions

Add an `estimate` helper that estimates RCU and WCU

Might look like...

operation = add('users', item={ 
    'id': 50, 
    'team_id': 288, 
    'username': 'sunshie', 
    'state': 'active' 
})

est = estimate(operation)

print(est)

{ 'rcu': 0, 'wcu': 45, 'units': 'kbs' }

Installing dynamof does not install dependencies

The Problem

When installing dynamof in a fresh virtual environment with no other dependencies yet installed it errors out saying boto3 is not installed. The expectation is that pip can see that dynamof depends on boto3 and installs it as a dependency before it tries to install dynamof.

Workaround

Install the dependencies (only one is boto3) manually before installing dynamof. Run pip install boto3 then install dynamof.

Steps to reproduce

1. Create a fresh virtual environment: python3 -m venv venv
2. Activate the venv: source venv/bin/activate
3. Install dynamof with pip: pip install dynamof==2.1.7

The result is:

(venv) @rayepps ~/projects zsh $ pip install dynamof==2.1.6
Collecting dynamof==2.1.6
  Using cached https://files.pythonhosted.org/packages/5b/6c/2a3237411d0c5cefbf4307a26a9d80d5908b663349ae612ed1747da16146/dynamof-2.1.6.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/fz/lxvknht5231glfllq3hrwhth0000gn/T/pip-build-i5xz1mpd/dynamof/setup.py", line 3, in <module>
        from dynamof.version import VERSION
      File "/private/var/folders/fz/lxvknht5231glfllq3hrwhth0000gn/T/pip-build-i5xz1mpd/dynamof/dynamof/__init__.py", line 5, in <module>
        from dynamof.operations.add import add
      File "/private/var/folders/fz/lxvknht5231glfllq3hrwhth0000gn/T/pip-build-i5xz1mpd/dynamof/dynamof/operations/add.py", line 2, in <module>
        from dynamof.core import builder as ab
      File "/private/var/folders/fz/lxvknht5231glfllq3hrwhth0000gn/T/pip-build-i5xz1mpd/dynamof/dynamof/core/builder.py", line 1, in <module>
        from boto3.dynamodb.types import TypeSerializer
    ModuleNotFoundError: No module named 'boto3'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/fz/lxvknht5231glfllq3hrwhth0000gn/T/pip-build-i5xz1mpd/dynamof/
You are using pip version 9.0.1, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Simplify + improve the docs

Our docs are many... so much so I think they're unhelpful. Let's simplify. Possibly remove the api documentation... I don't think it's helpful. When introducing new users to dynamof we should start with an example where we only use an operation function (like query) to build the boto3 description. Show them the result of that call and what it does, describe that this is the core function of dynamof. Then we can show them the use of the execute and the extra layer it adds.

Can't install with pip : No matching distribution found for dynofunc

Hi, i just try to install your lib but I have some errors. It seem's that it complain about my python version that doesn't meet the requirement of python >= 3 but I'm using python 3.9, any idea ?

Also you have a warning with the readme

user@MacBook-Pro:~/Documents/⚡ ⇒  pip3 -vvv install dynofunc                
Using pip 21.0.1 from /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip (python 3.9)
Non-user install because site-packages writeable
Created temporary directory: /private/var/folders/93/5gnyqj597qqg9dnrb1nx2jym0000gn/T/pip-ephem-wheel-cache-be37nipu
Created temporary directory: /private/var/folders/93/5gnyqj597qqg9dnrb1nx2jym0000gn/T/pip-req-tracker-r44tjnxh
Initialized build tracking at /private/var/folders/93/5gnyqj597qqg9dnrb1nx2jym0000gn/T/pip-req-tracker-r44tjnxh
Created build tracker: /private/var/folders/93/5gnyqj597qqg9dnrb1nx2jym0000gn/T/pip-req-tracker-r44tjnxh
Entered build tracker: /private/var/folders/93/5gnyqj597qqg9dnrb1nx2jym0000gn/T/pip-req-tracker-r44tjnxh
Created temporary directory: /private/var/folders/93/5gnyqj597qqg9dnrb1nx2jym0000gn/T/pip-install-c_p390fk
1 location(s) to search for versions of dynofunc:
* https://pypi.org/simple/dynofunc/
Fetching project page and analyzing links: https://pypi.org/simple/dynofunc/
Getting page https://pypi.org/simple/dynofunc/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/dynofunc/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/dynofunc/ HTTP/1.1" 304 0
  Found link https://files.pythonhosted.org/packages/58/28/ac1637a50967d3067780cfdcf922e32e2faf24267df0b638358720df5d1e/dynofunc-1.0.0.tar.gz#sha256=35e37ef8e37bebcd7ee07a4b88c2b7cf9e5fb7f3eb9d96db30aa725b5fff4de6 (from https://pypi.org/simple/dynofunc/) (requires-python:>=3), version: 1.0.0
Given no hashes to check 1 links for project 'dynofunc': discarding no candidates
Collecting dynofunc
  Created temporary directory: /private/var/folders/93/5gnyqj597qqg9dnrb1nx2jym0000gn/T/pip-unpack-j7pq2hoc
  Looking up "https://files.pythonhosted.org/packages/58/28/ac1637a50967d3067780cfdcf922e32e2faf24267df0b638358720df5d1e/dynofunc-1.0.0.tar.gz" in the cache
  Current age based on date: 8621
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 8621
  Using cached dynofunc-1.0.0.tar.gz (14 kB)
  Added dynofunc from https://files.pythonhosted.org/packages/58/28/ac1637a50967d3067780cfdcf922e32e2faf24267df0b638358720df5d1e/dynofunc-1.0.0.tar.gz#sha256=35e37ef8e37bebcd7ee07a4b88c2b7cf9e5fb7f3eb9d96db30aa725b5fff4de6 to build tracker '/private/var/folders/93/5gnyqj597qqg9dnrb1nx2jym0000gn/T/pip-req-tracker-r44tjnxh'
    Running setup.py (path:/private/var/folders/93/5gnyqj597qqg9dnrb1nx2jym0000gn/T/pip-install-c_p390fk/dynofunc_c3eaa9e701f442e08560cbb39efec22a/setup.py) egg_info for package dynofunc
    Created temporary directory: /private/var/folders/93/5gnyqj597qqg9dnrb1nx2jym0000gn/T/pip-pip-egg-info-ax3k1sko
    Running command python setup.py egg_info
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/93/5gnyqj597qqg9dnrb1nx2jym0000gn/T/pip-install-c_p390fk/dynofunc_c3eaa9e701f442e08560cbb39efec22a/setup.py", line 5, in <module>
        with open('README.md', 'r') as fh:
    FileNotFoundError: [Errno 2] No such file or directory: 'README.md'
WARNING: Discarding https://files.pythonhosted.org/packages/58/28/ac1637a50967d3067780cfdcf922e32e2faf24267df0b638358720df5d1e/dynofunc-1.0.0.tar.gz#sha256=35e37ef8e37bebcd7ee07a4b88c2b7cf9e5fb7f3eb9d96db30aa725b5fff4de6 (from https://pypi.org/simple/dynofunc/) (requires-python:>=3). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement dynofunc
ERROR: No matching distribution found for dynofunc
Exception information:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 171, in _merge_into_criterion
    crit = self.state.criteria[name]
KeyError: 'dynofunc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 318, in resolve
    name, crit = self._merge_into_criterion(r, parent=None)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _merge_into_criterion
    crit = Criterion.from_requirement(self._p, requirement, parent)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 83, in from_requirement
    raise RequirementsConflicted(criterion)
pip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('dynofunc')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 121, in resolve
    self._result = resolver.resolve(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 453, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 320, in resolve
    raise ResolutionImpossible(e.criterion.information)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('dynofunc'), parent=None)]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 189, in _main
    status = self.run(options, args)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 316, in run
    requirement_set = resolver.resolve(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 127, in resolve
    six.raise_from(error, e)
  File "<string>", line 3, in raise_from
pip._internal.exceptions.DistributionNotFound: No matching distribution found for dynofunc
Removed dynofunc from https://files.pythonhosted.org/packages/58/28/ac1637a50967d3067780cfdcf922e32e2faf24267df0b638358720df5d1e/dynofunc-1.0.0.tar.gz#sha256=35e37ef8e37bebcd7ee07a4b88c2b7cf9e5fb7f3eb9d96db30aa725b5fff4de6 from build tracker '/private/var/folders/93/5gnyqj597qqg9dnrb1nx2jym0000gn/T/pip-req-tracker-r44tjnxh'
Removed build tracker: '/private/var/folders/93/5gnyqj597qqg9dnrb1nx2jym0000gn/T/pip-req-tracker-r44tjnxh'
user@MacBook-Pro:~/Documents/⚡ ⇒   

Automate deployments to PyPi

When a PR is merged to develop we should automatically build the distribution package and upload it to PyPi. This will require that first we build some PR automation that requires that the version is bumped before a PR can be merged. If we don't, errors will occur when the automation attempts to publish a package with the same version as one that already exists.

• Automation in place requiring PRs to include a version bump
• On merging into develop a new distribution is created and uploaded to PyPi

Need a FilterExpression for query operation

Boto3 supports a filter expression on the query operation. We need to implement this. Should be easy, docs here say

The syntax for a filter expression is identical to that of a condition expression. Filter expressions can use the same comparators, functions, and logical operators as a condition expression.

Question on behavior, If i use the update item function and pass a dict, will it update only the keys that are present in the dict ?

Will keys that are not in the dict be removed or will they remain as they were? Im using a user-submitted form to update entries in dynamodb where the user can leave some entries in the form blank to leave the data entry as is in the database, and the entries in the form that do exist will be updated in the database.

Implement batch_write_item operation

Implement batch_write_item operation. See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Client.batch_write_item

Need to setup better logging system so users can control logs with level

Currently the only logging is some print statement (sorry - I'm only one man). Let's build a logging module and start using it to print logs anywhere there is useful information.

We might be able to avoid creating a separate logging module since its gonna be so slim. We really just need to make sure we're using a consistent logger name for now.

All logs should be a debug level. I don't think we're doing any work in dynamof (asside from errors) that merits other levels of logs.