rightlag/aptos

:sunny: Avro, Protobuf, Thrift on Swagger

aptos's Issues

Update AssertionError messages

Currently, the error messages returned from the ValidationVisitor are unclear. The error messages require a standard to follow (e.g. "value (John Doe) for property name 'firstName' is invalid because the 'maxLength' of the property is '5' characters, got '8'").

Fix exception raised in read function

Note: This does not apply to all JSON schemas.

read function raises an exception for this JSON schema.

Python 3.5.2 (default, Sep 15 2016, 07:38:42)
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from aptos.util import read
>>> read(open('./petstore.json'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/walsh/workspace/aptos/aptos/util.py", line 26, in read
    operation, referrant=specification)
  File "/Users/walsh/workspace/aptos/aptos/models.py", line 43, in load
    response, referrant=referrant)
  File "/Users/walsh/workspace/aptos/aptos/models.py", line 64, in load
    instance['schema'] = schema.resolve(referrant)
  File "/Users/walsh/workspace/aptos/aptos/primitives.py", line 219, in resolve
    value = Object.load(value, referrant=referrant)
  File "/Users/walsh/workspace/aptos/aptos/primitives.py", line 178, in load
    return super().load(instance)
  File "/Users/walsh/workspace/aptos/aptos/primitives.py", line 45, in load
    return cls(**instance)
  File "/Users/walsh/workspace/aptos/aptos/primitives.py", line 157, in __init__
    super().__init__(**kwargs)
TypeError: __init__() got an unexpected keyword argument 'items'

Should ValidationVisitor accumulate error state?

Currently, the ValidationVisitor does not contain state for errors.

An AssertionError is raised for invalid client-submitted data. For example:

{
    "title": "Person",
    "type": "object",
    "properties": {
        "firstName": {
            "type": "string"
        },
        "age": {
            "type": "integer",
            "minimum": 0
        }
    },
    "required": ["firstName"]
}

from aptos.visitors import ValidationVisitor

record.accept(ValidationVisitor({'age': -5'}))  # raises an `AssertionError`

The previous example contains two validation errors:

• Age is not a non-negative integer
• firstName is not provided

If the ValidationVisitor contained state, then it could potentially accumulate error messages.

For example:

class ValidationVisitor:

    def __init__(self, instance):
        self.instance = instance
        self.errors = []

    def visit_string(self, string):
        instance = self.instance
        if self.maxLength:
            if not (len(instance) <= string.maxLength):
                self.errors.append('<error message>')

import logging

from aptos.visitors import ValidationVisitor

errors = record.accept(ValidationVisitor({'age': -5}))
if errors:
    logging.error(errors)

Add support for Parquet

Per the Data Intelligence Conference, Parquet is a common serialization format used by data scientists and data engineers.

To implement support for Parquet, the following issues shall be addressed:

• Create a visitor for converting a JSON schema to a Parquet schema

Use 'OrderedDict' for Avro schema serialization

Since Avro specifications are serialized in JSON format, the order of the key-value pairs is never in the same. Using an 'OrderedDict' can resolve this issue.

Clean primitives module

Flatten Schema objects containing 'allOf' property

The allOf property in Swagger specifications is analogous to inheritance in OOP languages. allOf is an array of individually validated Schema objects, but with their properties combined to form a single Schema object.

Rename util module to parsers

Currently, the aptos.util module contains a class Parser with a static method parse to parse JSON formatted specifications.

Ideally, the Parser class should serve more as an interface for concrete parser classes to implement. Therefore, the Parser.parse method should raise a NotImplementedError.

class Parser:
    """Parser interface"""

    @staticmethod
    def parse(instance):
        raise NotImplementedError()


class JSONSchemaParser(Parser):

    @staticmethod
    def parse(instance):
        # Algorithm for parsing JSON schema documents

Update method names to be snake case instead of camel case

Method names such as visitPrimitive should become visit_primitive. Snake casing is considered to be more "Pythonic."

Instances of camel cased method names exist in the following modules:

• https://github.com/pennsignals/aptos/blob/master/aptos/visitors.py
• https://github.com/pennsignals/aptos/blob/master/aptos/primitives.py

Clean visitors module

Add support for statsmodels in Datascience image

When running quay.io/pennsignals/alpine-3.8-python-3.7-jupyter:v1.0

I couldn't import statsmodels into the notebook.

When i tried to pip install statsmodels i received the following output error.

copying statsmodels/tsa/vector_ar/data/e2.dat -> build/lib.linux-x86_64-3.7/statsmodels/tsa/vector_ar/data
    copying statsmodels/tsa/vector_ar/data/e1.dat -> build/lib.linux-x86_64-3.7/statsmodels/tsa/vector_ar/data
    running build_ext
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-5ujpat79/statsmodels/setup.py", line 571, in <module>
        **setuptools_kwargs)
      File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line 143, in setup
        return distutils.core.setup(**attrs)
      File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.7/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/local/lib/python3.7/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.7/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.7/distutils/command/build_ext.py", line 339, in run
        self.build_extensions()
      File "/tmp/pip-install-5ujpat79/statsmodels/setup.py", line 317, in build_extensions
        self.check_cython_extensions(self.extensions)
      File "/tmp/pip-install-5ujpat79/statsmodels/setup.py", line 314, in check_cython_extensions
        """ % src)
    Exception: Cython-generated file 'statsmodels/tsa/kalmanf/kalman_loglike.c' not found.
            Cython is required to compile statsmodels from a development branch.
            Please install Cython or download a source release of statsmodels.
    ```
Error

Command "/usr/local/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-5ujpat79/statsmodels/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-r75i25au/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-5ujpat79/statsmodels/

Read Swagger specification from remote source

Create Avro Visitor class

Separate the algorithm from the object structure on which it operates

Read Swagger specification from local source

Currently, the Specification.load_from_definition classmethod only supports reading Swagger specifications from remote (e.g. HTTP) sources.

Write the functionality to have the method also load Swagger specifications from local sources.

Create JSON primitive types

Create 'resolve' method for Reference types

Handle null value