abatilo/typed-json-dataclass

A python3.7 dataclass supplemental library. Enhances dataclasses to perform basic type checking and makes the dataclass JSON serializable.

My name is Aaron! 👋

Check out my website to see my latest projects, blog posts, or work history.

typed-json-dataclass's Issues

Nested dataclass in Optional not instantiated

Subject of the issue

Last one for tonight. See example and traceback.

Steps to reproduce

from typing import Optional
from dataclasses import dataclass
from typed_json_dataclass import TypedJsonMixin

@dataclass
class A(TypedJsonMixin):
    spam: str

@dataclass
class B(TypedJsonMixin):
    eggs: Optional[A]

B.from_json(B(A('foo')).to_json())

Expected behaviour

Result after from_json should be an instance B whose attribute eggs contains an instance A whose attribute spam contains 'foo'.

Actual behaviour

Traceback (most recent call last):
  File "test.py", line 13, in <module>
    B.from_json(B(A('foo')).to_json())
  File "/Users/ruben/.local/share/virtualenvs/source-dQgmcX_Z/lib/python3.7/site-packages/typed_json_dataclass/typed_json_dataclass.py", line 232, in from_json
    return cls.from_dict(json.loads(raw_json), mapping_mode=mapping_mode)
  File "/Users/ruben/.local/share/virtualenvs/source-dQgmcX_Z/lib/python3.7/site-packages/typed_json_dataclass/typed_json_dataclass.py", line 216, in from_dict
    return cls(**raw_dict)
  File "<string>", line 3, in __init__
  File "/Users/ruben/.local/share/virtualenvs/source-dQgmcX_Z/lib/python3.7/site-packages/typed_json_dataclass/typed_json_dataclass.py", line 65, in __post_init__
    raise TypeError((f'{class_name}.{field_name} was '
TypeError: B.eggs was defined to be any of: (<class '__main__.A'>, <class 'NoneType'>) but was found to be <class 'dict'> instead

Optional[x] (i.e. Union[x, None]) where x is a dataclass should be instantiated from the dictionary if said dictionary is not None, similar to how it gets instantiated without the Optional type.

Can't import from json string when using Optional

Subject of the issue

When using the Optional type hint, I can no longer convert from a json string into the data class.

Steps to reproduce

The following code will trow

TypeError: A.c was defined to be any of: (<class '__main__.B'>, <class 'NoneType'>) but was found to be <class 'dict'> instead

from typing import List, Optional
from dataclasses import dataclass
from typed_json_dataclass import TypedJsonMixin

@dataclass
class B(TypedJsonMixin):
    a: float = 0.0
    b: float = 0.0
    c: float = 0.0

@dataclass
class A(TypedJsonMixin):
    a: Optional[int] = None
    b: Optional[str] = None
    c: Optional[B] = None

b = B(1.0, 1.0, 1.0)
a = A(1, "a", b)
c = A.from_json(a.to_json())

Expected behaviour

Create a data class of type A

Actual behaviour

Does not create the dataclass of type A from the json and trows an error.

Dependabot couldn't parse the config file at .dependabot/config.yml

Dependabot couldn't parse the config file at .dependabot/config.yml. The error raised was:

(<unknown>): did not find expected key while parsing a block mapping at line 3 column 3

Please ensure the config file is a valid YAML file. An online YAML linter is available here.

You can mention @dependabot in the comments below to contact the Dependabot team.

Fails when using `Optional` annotation for a

Subject of the issue

If the dataclass with the TypedJsonMixin has a attribute with None as default value, and you want to properly annotate for mypy, the dataclass becomes unusable.

Steps to reproduce

Here is the code to reproduce the situation

from dataclasses import dataclass
from typing import Optional
from typed_json_dataclass import TypedJsonMixin

@dataclass
class Person(TypedJsonMixin):
    name: str
    age: Optional[int] = None

Person("John", 42)

Expected behaviour

Would have expected it to still be usable. Perhaps it is desired behavior due to how json is specified, and as such a non-issue. But maybe you're still interested in such a report.

Actual behaviour

Ends up raising

TypeError: typing.Union cannot be used with isinstance()

because under the hood Optional is a particular type Union[NonType, int].

Post Scriptum: Nice project and very clear coding! Good job.

Dependabot couldn't parse the config file at .dependabot/config.yml

Dependabot couldn't parse the config file at .dependabot/config.yml. The error raised was:

(<unknown>): did not find expected '-' indicator while parsing a block collection at line 3 column 3

Please ensure the config file is a valid YAML file. On online YAML linter is available here.

You can mention @dependabot in the comments below to contact the Dependabot team.

Dependabot couldn't find a package.json for this project

Dependabot couldn't find a package.json for this project.

Dependabot requires a package.json to evaluate your project's current JavaScript dependencies. It had expected to find one at the path: /package.json.

If this isn't a JavaScript project, or if it is a library, you may wish to disable updates for it in the .dependabot/config.yml file in this repo.

View the update logs.

Attempts to instantiate with Dict in nested list

Example program

from typing import List, Dict
from dataclasses import dataclass
from typed_json_dataclass import TypedJsonMixin

@dataclass
class Test(TypedJsonMixin):
    test: List[Dict[str, int]]

Test([{'a': 1}, {'b': 2, 'c': 3}])

Traceback

Traceback (most recent call last):
  File "test.py", line 9, in <module>
    Test([{'a': 1}, {'b': 2, 'c': 3}])
  File "<string>", line 3, in __init__
  File "/Users/ruben/.local/share/virtualenvs/source-dQgmcX_Z/lib/python3.7/site-packages/typed_json_dataclass/typed_json_dataclass.py", line 118, in __post_init__
    )[i] = expected_element_type(**element)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/typing.py", line 668, in __call__
    raise TypeError(f"Type {self._name} cannot be instantiated; "
TypeError: Type Dict cannot be instantiated; use dict() instead

Your .dependabot/config.yml contained invalid details

Dependabot encountered the following error when parsing your .dependabot/config.yml:

Automerging is not enabled for this account. You can enable it from the [account settings](https://app.dependabot.com/accounts/abatilo/settings) screen in your Dependabot dashboard.
Automerging is not enabled for this account. You can enable it from the [account settings](https://app.dependabot.com/accounts/abatilo/settings) screen in your Dependabot dashboard.
Automerging is not enabled for this account. You can enable it from the [account settings](https://app.dependabot.com/accounts/abatilo/settings) screen in your Dependabot dashboard.
Automerging is not enabled for this account. You can enable it from the [account settings](https://app.dependabot.com/accounts/abatilo/settings) screen in your Dependabot dashboard.

Please update the config file to conform with Dependabot's specification using our docs and online validator.

You can mention @dependabot in the comments below to contact the Dependabot team.

Typesafe getter and setter

Currently setting the attributes of a derived class is not typesafe e.g.

@dataclass
 class Position(TypedJsonMixin):
    x : int 
    y : int 

p = Position(3, 3)
p.x = 1.2

is still possible. Does it make sense to overwrite the setatt method of the class
currently im doing this manually like this :

def __settattr__(self, name, value):
    if isinstance(value, type(getattr(self, name))):
        super().__setattr__(self, name, value)
    else:
        print(f'Type is not matching with that of {name} which is {type(getattr(self, name))}')

im not sure if this has some drawbacks at some point
maybe it would be an option

Dependabot couldn't authenticate with https://pypi.python.org/simple/

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Can't use from __future__ import annotations

Subject of the issue

When using from __future__ import annotations (PEP 563 -- Postponed Evaluation of Annotations) I get Errors of type TypeError: isinstance() arg 2 must be a type or tuple of types

Steps to reproduce

from __future__ import annotations
from dataclasses import dataclass
from typed_json_dataclass import TypedJsonMixin

@dataclass
class B(TypedJsonMixin):
	d1: str
	d2: int

b1 = B(d1="b1", d2=1)

Expected behaviour

The following code should run without any type error

from __future__ import annotations
from typing import List
from dataclasses import dataclass, field
from typed_json_dataclass import TypedJsonMixin

@dataclass
class A(TypedJsonMixin):
	d1: str
	d2: List[B] = field(default_factory=list)

@dataclass
class B(TypedJsonMixin):
	d1: str
	d2: int

breakpoint()
b1 = B(d1="b1", d2=1)
b2 = B("b2",2)

a = A("a")

Actual behaviour

Trows the error TypeError: isinstance() arg 2 must be a type or tuple of types

Wrong pypi link to github repository

Its just a small issue but basically the repo link at

https://pypi.org/project/typed-json-dataclass/

does create a 404, since it links to https://github.com/abatilo/typed_json_dataclass
but it should link to https://github.com/abatilo/typed-json-dataclass

@abatilo Since you are the maintainer I thought I will post it here

Doesn't work with InitVars

Subject of the issue

When a dataclass defines an InitVar attribute, TypedJsonMixin's __post_init__ method tries to validate the type of the InitVar attribute, which doesn't exist as an actual attribute on the class. Additionally, from_dict (and consequently from_json as well) fail to initialize a new instance because of a missing positional argument (more explanation follows).

Steps to reproduce

import dataclasses

from typed_json_dataclass import TypedJsonMixin

@dataclasses.dataclass
class Test(TypedJsonMixin):
    init: dataclasses.InitVar[str]
    a: int = 0
    b: str = ''

    def __post_init__(self, init: str) -> None:
        self.a = len(init)
        self.b = init[0]
        super().__post_init__()  # Necessary to call mixin

Test('foo')
# AttributeError: 'Test' object has no attribute 'init'

This is due to __post_init__ retrieving fields from the __dataclass_fields__ attribute, which includes the initialization variables. However, when trying to access this field, lookup fails as these fields are actually not stored in the instance at all, and only passed to the __post_init__ method to finish initialization. The dataclasses.fields() function would not return these initialization variables, so that would be a possible solution.

However, this still fails the instantiation of a dataclass from a dictionary or JSON string, since these initialization variables are required arguments to a dataclass' __init__. See below.

import dataclasses

from typed_json_dataclass import TypedJsonMixin

@dataclasses.dataclass
class Test(TypedJsonMixin):
    init: dataclasses.InitVar[str]
    a: int = 0
    b: str = ''

    def __post_init__(self, init: str) -> None:
        self.a = len(init)
        self.b = init[0]
        # super().__post_init__()

f = Test('foo')
g = Test.from_json(f.to_json())
# TypeError: __init__() missing 1 required positional argument: 'init'

Disabling the super call to __post_init__ skips the type check but uncovers another issue. Since the to_dict method delegates to dataclasses.asdict, these initialization variables are not saved in the dictionary. They cannot be saved in the dictionary, as they do not exist in the scope of the dataclass instance anymore. Therefore, I believe there is no solution to that issue that does not require either significant hacking of dataclass internals, dropping InitVar from a dataclass declaration, or using a non-elegant workaround.

Two workarounds I have identified so far:

• Supply a default value to the initialization variable.

@dataclasses.dataclass
class Test(TypedJsonMixin):
    init: dataclasses.InitVar[str] = None
    a: int = 0
    b: str = ''

    def __post_init__(self, init: str) -> None:
        if init is None:
            # from_dict
            return
        self.a = len(init)
        self.b = init[0]

This will turn it into a non-required argument and allows instantiation with __init__ to succeed. However, this requires us to check that this initialization variable is not the default value, which would be the case when initialized from from_dict. Furthermore, instantiations without supplying the initialization variable, such as Test(), are now legal (but don't make sense).

• Mimic the initialization variable as a private field that is not considered for comparisons and __repr__.

@dataclasses.dataclass
class Test(TypedJsonMixin):
    _init: str = dataclasses.field(compare=False, repr=False)
    a: int = 0
    b: str = ''

    def __post_init__(self) -> None:
        self.a = len(self._init)
        self.b = self._init[0]
        super().__post_init__()

This makes it impossible to instantiate the dataclass without a value for _init, but keeps this value around throughout the lifetime of the instance. The __post_init__ method would also be called again to process _init after deserialization from JSON. It is possible to del the _init attribute after __post_init__, but this again leads to issues when converting the dataclass to and from a dictionary.

Neither are really good solutions to the problem. I'd love to hear your thoughts on this.

Dependabot couldn't find a Dockerfile for this project

Dependabot couldn't find a Dockerfile for this project.

Dependabot requires a Dockerfile to evaluate your project's current Docker dependencies. It had expected to find one at the path: /actions/make/Dockerfile.

If this isn't a Docker project, or if it is a library, you may wish to disable updates for it in the .dependabot/config.yml file in this repo.

View the update logs.

Dependabot couldn't find a package.json for this project

Dependabot couldn't find a package.json for this project.

Dependabot requires a package.json to evaluate your project's current JavaScript dependencies. It had expected to find one at the path: /package.json.

If this isn't a JavaScript project, or if it is a library, you may wish to disable updates for it in the .dependabot/config.yml file in this repo.

View the update logs.

pydantic comparison

How does this library compare to pydantic

Dependabot couldn't find a Dockerfile for this project

Dependabot couldn't find a Dockerfile for this project.

Dependabot requires a Dockerfile to evaluate your project's current Docker dependencies. It had expected to find one at the path: /actions/version_syncer/Dockerfile.

If this isn't a Docker project, or if it is a library, you may wish to disable updates for it in the .dependabot/config.yml file in this repo.

View the update logs.

TypeError: Params.my_params is expected to be <class 'params.MyParams'>, but value {'host': 'txt', 'project': 'txt', 'roi_term_id': 123} is a dict with unexpected keys

Not sure if this belongs in this repo...

Goal: read in parameters from .yaml to pass to functions during runtime.

I've seen no reference to this error online; so decided to make a post.

I have a user-defined params.yaml:

my_params:
  host: txt
  project: txt
  roi_term_id: 123

# ...

That is read-in by params.py:

import os
from dataclasses import dataclass
from pathlib import Path
import yaml
from decouple import config
from typed_json_dataclass import TypedJsonMixin


@dataclass
class MyParams(TypedJsonMixin):
    host: str
    project: str
    roi_term: str

    def __post_init__(self):
        self.public_key = config('KEY')
        assert isinstance(self.public_key, str)
        self.private_key = config('SECRET')
        assert isinstance(self.private_key, str)
        super().__post_init__()

# ...
@dataclass
class Params(TypedJsonMixin):
    my_params: MyParams
    # ...


def load_params_dict():
    parameter_file = 'params.yaml'
    cwd = Path(os.getcwd())
    params_path = cwd / parameter_file
    if params_path.exists():
        params = yaml.safe_load(open(params_path))
    else:  # If this script is being called from the path directory
        params_path = cwd.parent / parameter_file
        params = yaml.safe_load(open(params_path))
    return params


params_dict = load_params_dict()
print(params_dict)
project_params = Params.from_dict(params_dict)

Traceback:

  File "/home/me/miniconda3/envs/myvenv/lib/python3.7/site-packages/typed_json_dataclass/typed_json_dataclass.py", line 152, in __post_init__
    expected_type(**field_value)
TypeError: __init__() got an unexpected keyword argument 'roi_term_id'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "path/main.py", line 7, in <module>
    from params import project_params
  File "/home/me/PycharmProjects/project/path/params.py", line 89, in <module>
    project_params = Params.from_dict(params_dict)
  File "/home/me/miniconda3/envs/myvenv/lib/python3.7/site-packages/typed_json_dataclass/typed_json_dataclass.py", line 248, in from_dict
    return cls(**raw_dict)
  File "<string>", line 9, in __init__
  File "/home/me/miniconda3/envs/myvenv/lib/python3.7/site-packages/typed_json_dataclass/typed_json_dataclass.py", line 155, in __post_init__
    raise TypeError(f'{class_name}.{field_name} '
TypeError: Params.my_params is expected to be <class 'params.MyParams'>, but value {'host': 'txt', 'project': 'txt', 'roi_term_id': 123} is a dict with unexpected keys

Dependabot couldn't find a <anything>.yml for this project

Dependabot couldn't find a .yml for this project.

Dependabot requires a .yml to evaluate your project's current Github_actions dependencies. It had expected to find one at the path: /.github/workflows/.github/workflows/<anything>.yml.

If this isn't a Github_actions project, or if it is a library, you may wish to disable updates for it in the .dependabot/config.yml file in this repo.

View the update logs.