konradhalas / dacite Goto Github PK
View Code? Open in Web Editor NEWSimple creation of data classes from dictionaries.
License: MIT License
Simple creation of data classes from dictionaries.
License: MIT License
I'm not sure about some features currently implemented in dacite
. The main goal of this project is to build a data class from a plain dictionary. It's not a serialization/desearialization or validation library. There are many such libs, e.g. DRF or marshmallow, and I don't want to create another one.
I'm talking about following features:
Config.remap
Config.flattened
Config.prefixed
Config.cast
Config.transform
Even from code point of view all of those features live in a separate module -config
- and they can be easily decoupled from data classes at all. So maybe this is a good idea for a new library which will allow to transform your dictionary to different dictionary according to provided rules (remap
, flattened
, prefixed
...), but I don't know should we have such features in dacite
.
On the other hand it easier for users to install one lib instead of two.
So I see the following solutions:
dacite.from_dict(
data_class=X,
data=dacite.transform_data(data, config=TransformConfig(...)),
config=Config(...),
)
Nr 2 is my favourite one.
What do you think @rominf @jasisz?
It's a good time for such decisions - I want to release 1.0.0 soon.
Hi, nice nifty little library. I'm trying to understand how the API supports the idea of creating nested dataclasses. The example in the documentation shows a nesting of one level, but if one has to deal with a dict of deeper nesting, then how can the Config
instance for the deeper levels be passed through.
Extending the documentation example:
@dataclass
class A:
x: int
y: int
@dataclass
class B:
a: A
is_cool: bool
B_config = Config(prefixed={'a': 'a_})
@dataclass
class C:
b: B
count: int
C_config = Config(remap={'count': 'number'})
DATA={
'number': 30,
'b': {
'is_cool': False,
'a_x': 30,
'a_y': 55,
}
}
from_dict(data_class=C, data=DATA, config=C_config)
I do not see how the from_dict
API allows the use of B_config. Could you provide an example? Or is there no possibility for configuration beyond the top nesting level?
Thanks in advance!
I wanted to store dataclasses as Flask session values, but Flask complains that I first need to override app.json_decoder
and app.json_encoder
. I thought of your library, but then realized that implementing object hooks for deserialization might actually be non-trivial. Perhaps you'd like to look into this use case and see if it's within the scope of your project?
Ideally I'd like to be able to do something like:
from flask import Flask, session
from dacite.json_handlers import JSONDecoder, JSONEncoder
app = Flask(__name__)
app.secret_key = b'soverysecret' # needed for session storage
app.json_encoder = JSONEncoder
app.json_decoder = JSONDecoder
And then be able to transparently store my dataclasses in a Flask session.
I have been using dacite for a while in many projects, and it's a very useful little tool!
I did notice that as a pattern I prefer to have the methods as part of the class itself, so I have been using it like so:
class Entity(Serializable):
...
entity = Entity.load('file_path')
assert entity == Entity.from_dict(data)
Here is the definition:
@dataclass
class Serializable:
@classmethod
def copy(cls, other):
return dacite.from_dict(cls, asdict(other))
@classmethod
def load(cls, file):
with open(file, "r") as f:
data = json.load(f)
return dacite.from_dict(cls, data)
@classmethod
def from_dict(cls, data):
return dacite.from_dict(cls, data)
def to_json(self):
return json.dumps(asdict(self))
It's pretty simple but IMHO makes it more explicit, and doesn't require using classes to be aware of dacite (thus decoupling the implementation from the interface..)
Would you accept a PR?
https://github.com/samuelcolvin/pydantic/ seems similar to dacite and 5 times more popular. Please, compare them.
Is it possible to add such option to Config?
I have some 3rd party json with keys in camelCase
but that isn't appropriate naming for attributes in dataclass
Example:
>>> import dacite
>>> from dataclasses import dataclass
>>> from typing import TypeVar, Generic
>>>
>>> DataType = TypeVar("DataType")
>>>
>>> @dataclass
... class Data(Generic[DataType]):
... value: DataType
...
>>> class StrData(Data[str]):
... pass
...
>>> dacite.from_dict(StrData, {"value": "I am a string"})
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "dacite.py", line 103, in from_dict
if not _is_instance(field.type, value):
File "dacite.py", line 330, in _is_instance
return isinstance(value, t)
TypeError: isinstance() arg 2 must be a type or tuple of types
dacite currently does not deduce that the "value" field for StrData is a str
type due to its generic inheritance into a concrete type. Instead, it throws the error above.
Currently, using a dataclasses which include non-init fields throws an error.
Example (python 3.7):
>>> import dacite
>>> from dataclasses import dataclass, field
>>>
>>> @dataclass
>>> class A:
... number: int
... text: str
...
... post: str = field(init=False)
...
>>>
>>> data = {
... "_data_type": "A",
... "number": 1,
... "text": "hello",
... "post": "gotcha!"
... }
>>> dacite.from_dict(A, data)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/Users/williampeake/venvs/isle_collections-py-37/lib/python3.7/site-packages/dacite.py", line 96, in from_dict
return data_class(**values)
TypeError: __init__() got an unexpected keyword argument 'post'
...and probably also for sets, etc.
First of all, thank you for comments and changes after my last issue #61. I clearly understand the reasoning behind that.
Now I'm running into issues related to that. When trying to deserialize a dictionary that has a list but the dataclass requires tuples, I can't get the casting to work. I think the code speaks for itsself:
from dataclasses import dataclass
import typing
from dacite import from_dict, Config
TupleOfInts = typing.Tuple[int]
@dataclass
class Dataclass:
values: TupleOfInts
from_dict(
data_class=Dataclass,
data={"values": [1,2,3]},
config=Config(
cast=[
typing.Tuple, # I'd expect either of these to work
tuple,
]
)
)
dacite.exceptions.WrongTypeError: wrong type for field "values" - should be "typing.Tuple[int]" instead of "list"
I expected dacite to cast the list to a tuple and then confirm the actual values are ints. So I thought: Maybe I have to add the TupleOfInts
to the cast
parameter explicitly (which I'd prefer to avoid) but then I get another error:
File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/typing.py", line 671, in __call__
raise TypeError(f"Type {self._name} cannot be instantiated; "
TypeError: Type Tuple cannot be instantiated; use tuple() instead
btw: My editor complains when passing in a set as parameter for cast
. Without having looked at the implementation, it seems the type hint List
is more restrictive than necessary and Iterable
will do the job, too.
If one tries to define a method on a dateless (e.g. diameter as shown below), dacite doesn't seem to want to accept such a case - see tests and output below. This seems to be a collision with the handling of post_init values
========================================================================== test session starts ==========================================================================
platform darwin -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
rootdir: /Users/james/Projects/dacite, inifile:
collected 131 items
test_config.py ............ [ 9%]
test_dataclasses.py ......FF [ 15%]
test_types.py ............................ [ 36%]
core/test_base.py ............ [ 45%]
core/test_collection.py ........ [ 51%]
core/test_config.py ....................................... [ 81%]
core/test_optional.py ............ [ 90%]
core/test_union.py ............ [100%]
=============================================================================== FAILURES ================================================================================
_______________________________________________________________ test_create_instance_with_computed_attr_1 _______________________________________________________________
def test_create_instance_with_computed_attr_1():
@dataclass
class Circle(object):
radius: int
diameter: int = field(init=False)
def diameter(self):
return 2 * self.radius
instance = create_instance(
data_class=Circle,
init_values={'radius':4},
post_init_values={})
assert instance.radius == 4
assert instance.diameter == 8
E assert <function test_create_instance_with_computed_attr_1..Circle.diameter at 0x104a57378> == 8
E + where <function test_create_instance_with_computed_attr_1..Circle.diameter at 0x104a57378> = test_create_instance_with_computed_attr_1..Circle(radius=4, diameter=<function test_create_instance_with_computed_attr_1..Circle.diameter at 0x104a57378>).diameter
test_dataclasses.py:93: AssertionError
_______________________________________________________________ test_create_instance_with_computed_attr_2 _______________________________________________________________
def test_create_instance_with_computed_attr_2():
@dataclass
class Circle(object):
radius: int
diameter: int = field(init=False)
def diameter(self):
return 2 * self.radius
instance = create_instance(
data_class=Circle,
init_values={'radius':4},
post_init_values={'diameter':None})
assert instance.radius == 4
assert instance.diameter == 8
E assert None == 8
E + where None = test_create_instance_with_computed_attr_2..Circle(radius=4, diameter=None).diameter
test_dataclasses.py:111: AssertionError
================================================================= 2 failed, 129 passed in 0.36 seconds ==================================================================
It doesn't make sense in project like dacite
, let use setup.extras_require
It seems that when using Optional
and Union
together, the Optional
attribute is ignored.
This test will fail right now because a MissingValueError
is incorrectly raised:
@dataclass
class X:
i: int
@dataclass
class Y:
s: str
@dataclass
class Z:
x_or_y: Optional[Union[X,Y]]
result = from_dict(Z, {'a': {'s': 'test'}})
assert result == Z(x_or_y=None)
Error:
dacite.MissingValueError: missing value for field x_or_y
Looking into the code it seems related to this function:
def _is_optional(t: Type) -> bool:
return _is_union(t) and type(None) in t.__args__ and len(t.__args__) == 2
The function is returning false
, when it should be returning true
. This is because len(t.__args__) == 2
is returning false. What is the reason for the len == 2
check? It seems that removing this will solve this corner case.
Hello,
thank you for dacite it's a great help working with python dataclasses.
There might be a bug in the current version of 1.1.0. Here is the stacktrace:
Traceback (most recent call last):
File "/test.py", line 33, in <module>
print(dacite.from_dict(data_class=IntegerList, data=dict))
File "C:\Program Files\Python37\lib\site-packages\dacite\core.py", line 64, in from_dict
raise WrongTypeError(field_path=field.name, field_type=field.type, value=value)
dacite.exceptions.WrongTypeError: wrong type for field "some_list" - should be "typing.List[~T]" instead of "list"
Here is a code snippet to reproduce the error:
from dataclasses import dataclass
from typing import List, TypeVar, Generic
import dacite
@dataclass
class Point:
x: int
y: int
@dataclass
class PointA:
a: int
b: int
c: float
T = TypeVar('T', Point, PointA)
@dataclass
class IntegerList(Generic[T]):
some_list: List[T]
ok_dict = {'some_list': [{'x': 1, 'y': 2}]}
print(dacite.from_dict(data_class=IntegerList, data=ok_dict))
Executing with version 1.0.2 prints the expected data structure.
Executing with version 1.1.0 gives the above error.
Not sure what is causing that or if that is intended.
Cheers,
Tobias
Now to perform it correcly, the name of the field need to match the case of the dict key. Would nice if we could use from_dict
as case insensitive.
Example
import dataclasses
from typing import NewType, Optional
CUSTOM_TYPE = NewType('CUSTOM_TYPE', str)
@dataclasses.dataclass
class Test:
usual_field: str
custom_type_field: CUSTOM_TYPE
optional_custom_type_field: Optional[CUSTOM_TYPE]
test = Test('usual', CUSTOM_TYPE('custom'), CUSTOM_TYPE('optional_custom'))
test_dict = dataclasses.asdict(test)
dacite.from_dict(Test, test_dict)
Output
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-22-c59ddd955c43> in <module>
8 test = Test('usual', CUSTOM_TYPE('custom'))
9 test_dict = dataclasses.asdict(test)
---> 10 dacite.from_dict(Test, test_dict)
.venv/lib/python3.7/site-packages/dacite/core.py in from_dict(data_class, data, config)
43 error.update_path(field.name)
44 raise
---> 45 if config.check_types and not is_instance(value, field.type):
46 raise WrongTypeError(
47 field_path=field.name,
.venv/lib/python3.7/site-packages/dacite/types.py in is_instance(value, t)
53 elif is_union(t):
54 types = tuple(extract_origin_collection(t) if is_generic(t) else t for t in extract_generic(t))
---> 55 return isinstance(value, types)
56 elif is_generic_collection(t):
57 return isinstance(value, extract_origin_collection(t))
TypeError: isinstance() arg 2 must be a type or tuple of types
This is happens only if I use Optional[NewType]
. Without Optional
dacite works like charm.
>>> dataclasses.fields(Test)[0].type, dataclasses.fields(Test)[1].type, dataclasses.fields(Test)[2].type
(str,
<function typing.NewType.<locals>.new_type(x)>,
typing.Union[CUSTOM_TYPE, NoneType])
Hi,
I was experimenting with this library, to try to used it to solve some problems I have, and I hit a wall: I need to support flexible types in my dataclasses, and dacite doesn't support typing.TypeVar
.
So it would be nice to have dacite supporting TypeVar
and infer the type that the value needs to be transformed to. For example:
@dataclass
class Car:
model: str
@dataclass
class Person:
age: int
Box = TypeVar('Box', Car, Person)
@dataclass
class Container:
box: Box
car_container = from_dict(Container, {
'box': {
'model': 'chevy',
},
})
assert isinstance(car_container.box, Car)
What do you think? Does this make sense? In that example, there could be a check for Box.__constraints__
and try to match with whatever type it makes sense given the provided fields.
Hey,
very cool library, thanks for it.
Is there any intention to add support for Enums in order let the following code work:
from dataclasses import dataclass
from enum import Enum
from dacite import from_dict
class SomeEnum(Enum):
FIRST_VALUE = "first"
SECOND_VALUE = "second"
@dataclass
class DataClass:
message: str
value: SomeEnum
data = {"message": "hello", "value": "first"}
from_dict(data_class=DataClass, data=data)
Currently this raises
dacite.exceptions.WrongTypeError: wrong type for field "value" - should be "SomeEnum" instead of "str"
But actually it should be quite easy to do the conversion to the enum along the deserialization (SomeEnum("first")
yields <SomeEnum.FIRST_VALUE: 'first'>
)
I am trying to cast a list of strings to UUIDs and get the error. Here is the example:
>>> from dataclasses import dataclass
>>> from typing import List
>>> from uuid import UUID
>>>
>>> import dacite
>>>
>>>
>>> @dataclass
... class A:
... uuid_list: List[UUID]
...
...
>>> data = {'uuid_list': ['3416bc37-9d53-49dc-8361-ad2fb261fb71', 'e81bdbb7-14cd-480b-81ff-369ff49a0bcc']}
>>>
>>> dacite.from_dict(data_class=A, data=data, config=dacite.Config(cast=['uuid_list']))
Traceback (most recent call last):
File "<input>", line 1, in <module>
dacite.from_dict(data_class=A, data=data, config=dacite.Config(cast=['uuid_list']))
File "/Users/dmitry/.local/pyenv/versions/3.7.0/envs/sandbox/lib/python3.7/site-packages/dacite.py", line 93, in from_dict
value = cls(value)
File "/Users/dmitry/.local/pyenv/versions/3.7.0/lib/python3.7/typing.py", line 668, in __call__
raise TypeError(f"Type {self._name} cannot be instantiated; "
TypeError: Type List cannot be instantiated; use list() instead
>>>
This is quite similar to Issue #62, only different being that I've now added an extra Optional
:
import dataclasses
from typing import Optional
import dacite
@dataclasses.dataclass
class Person:
height: Optional[float] = 160
person = Person()
person_dict = dataclasses.asdict(person)
new_person_1 = dacite.from_dict(data_class=Person, data=person_dict)
On the latest version of Dacite (v1.2.0 โ 9c311b1), executing this yields:
WrongTypeError: wrong type for field "height" - should be "typing.Union[float, NoneType]" instead of "int"
Changing Optional[float]
to be just float
will make it work again. This seems to be a bug since Optional[float]
should also work.
I've seen the discussion in #62 and am kind of reopening the issue, because I disagree with the output (if @konradhalas disagrees to my objection, feel free to close this issue for good):
Let me cite PEP383 as an argument: https://www.python.org/dev/peps/pep-0484/#the-numeric-tower
PEP 3141 defines Python's numeric tower, and the stdlib module numbers implements the corresponding ABCs (Number, Complex, Real, Rational and Integral). There are some issues with these ABCs, but the built-in concrete numeric classes complex, float and int are ubiquitous (especially the latter two :-).
Rather than requiring that users write import numbers and then use numbers.Float etc., this PEP proposes a straightforward shortcut that is almost as effective: when an argument is annotated as having type float, an argument of type int is acceptable; similar, for an argument annotated as having type complex, arguments of type float or int are acceptable. This does not handle classes implementing the corresponding ABCs or the fractions.Fraction class, but we believe those use cases are exceedingly rare.
dacite not accepting int
s as float
s or complex
' makes it seem more catholic than the Pope. It requires us to write more complex code without providing any benefit.
We want to have:
black
pylint
mypy
According to #38 instead of:
Config(transform={"my_field": datetime.fromisoformat})
... we want to have:
Config(transform={datetime: datetime.fromisoformat})
It was implemented some time ago in this PR: #32 but it should be implemented from scratch because of latests refactor + we have to think about good name.
Sometimes it is necessary to declare types out-of-order. In this instance, quotes are put around the type to indicate the type has not yet been declared, but will be once the module is done importing.
Here is a trivial example:
>>> @dataclass
... class X:
... text: str
... y_data: "Y"
...
>>> @dataclass
... class Y:
... num: int
...
Obviously, in this case, one could just reverse the declarations, but there are cases where types must remain in quotes, especially when handling cross-dependencies.
Having such a type declaration results in the following:
>>> x_dict = {
... "text": "hello!",
... "y_data": {
... "num": 10
... }
... }
>>> dacite.from_dict(x_dict)
Traceback (most recent call last):
File "<input>", line 1, in <module>
TypeError: from_dict() missing 1 required positional argument: 'data'
>>> dacite.from_dict(X, x_dict)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 93, in from_dict
elif not _is_instance(field.type, value):
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 276, in _is_instance
return isinstance(value, t)
TypeError: isinstance() arg 2 must be a type or tuple of types
Even explicitly having a Y class already in the dict throws the same error:
>>> x_dict = {
... "text": "hello!",
... "y_data": Y(10)
... }
>>> dacite.from_dict(X, x_dict)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 93, in from_dict
elif not _is_instance(field.type, value):
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 276, in _is_instance
return isinstance(value, t)
TypeError: isinstance() arg 2 must be a type or tuple of types
Thanks for your time. This is an awesome library, and I plan to make fairly heavy use of it for serializing / deserializing dataclasses from json.
Dacite supports well unions on list when all the items in the list are of the same class.
However it does not seem to support unions on lists that have different types of classes.
Example of a test case that fails right now:
@dataclass
class X:
i: int
@dataclass
class Y:
s: str
@dataclass
class Z:
x_or_y: List[Union[X,Y]]
result = from_dict(Z, {'x_or_y': [{'s': 'test'}, {'i': 1}]})
assert result == Z(x_or_y=[Y(s='test'), X(i=1)])
I want to use dacite to map json that is returned from http call to dataclass. As soon as there is no UUID type in json the received uuid have string type. Instead of manually converting all UUIDs before calling from_dict() it will be really nice if dacite can check that input type is string and required type is UUID and convert string to UUID in background.
I have a use-case similar to below code ...
from dacite import from_dict
from dataclasses import dataclass, asdict
@dataclass
class A():
x: str = None
# a1 = A('a1')
a1 = A()
a2 = from_dict(A, asdict(a1))
Error:
WrongTypeError: wrong type for field "x" - should be "str" instead of "NoneType"
Is this expected behaviour? Is there a work-around other than using custom asdict function which skips None valued keys?
Thanks!
Example:
import uuid
from dataclasses import dataclass
from enum import Enum
from typing import Optional
import dacite
class E(Enum):
A = 1
B = 2
@dataclass
class X:
test: Optional[E]
data = {
'test': None
}
x = dacite.from_dict(
data_class=X,
data=data,
config=dacite.Config(cast=['test'])
)
Output:
Traceback (most recent call last):
File "<input>", line 4, in <module>
config=dacite.Config(cast=['test'])
File "/Users/dmitry/.local/pyenv/versions/3.7.0/envs/sandbox/lib/python3.7/site-packages/dacite.py", line 93, in from_dict
value = cls(value)
File "/Users/dmitry/.local/pyenv/versions/3.7.0/lib/python3.7/enum.py", line 307, in __call__
return cls.__new__(cls, value)
File "/Users/dmitry/.local/pyenv/versions/3.7.0/lib/python3.7/enum.py", line 555, in __new__
return cls._missing_(value)
File "/Users/dmitry/.local/pyenv/versions/3.7.0/lib/python3.7/enum.py", line 568, in _missing_
raise ValueError("%r is not a valid %s" % (value, cls.__name__))
ValueError: None is not a valid E
According to #38 we will remove:
flattened
remap
prefixed
Example:
from dataclasses import dataclass
from typing import NewType
import dacite
MyStr = NewType("MyStr", str)
@dataclass
class Data:
my_str: MyStr
dacite.from_dict(Data, {"my_str": "foo-bar"})
Traceback (most recent call last):
File "<input>", line 1, in <module>
dacite.from_dict(Data, {"my_str": "foo-bar"})
File "dacite.py", line 103, in from_dict
if not _is_instance(field.type, value):
File "dacite.py", line 337, in _is_instance
return isinstance(value, t)
TypeError: isinstance() arg 2 must be a type or tuple of types
Dacite currently does not support the "NewType" fields. The problem is in the _is_instance
function. Please see the related pull request.
Python supports circular dependencies when an entire package is included through the wildcard. But dacite is unable to deserialize circular dependencies. Example:
from .b import *
@dataclass
class A:
b: Optional[B]
from .a import *
@dataclass
class B:
a: Optional[A]
Then when I try to load this:
from models.a import A
dacite.from_dict(data_class=A, data={
'b': {}
})
I get-
dacite.exceptions.ForwardReferenceError: can not resolve forward reference: name 'B' is not defined
Because it can't find B
in the from .b import *
statement?
What can I do to get around this issue?
tks
The following code does not work as I would expect:
from dataclasses import dataclass, field
import dacite
@dataclass
class item:
item_field: str = 'default_value'
@dataclass
class container:
dict_of_dict_of_dataclass: Dict[str, Dict[str, item]] = field(default_factory=dict)
complex_dict = {'dict_of_dict_of_dataclass': {'outer': {'inner': {'item_field': 'a value'}}}}
obj = dacite.from_dict(container, complex_dict)
type(obj.dict_of_dict_of_dataclass['outer']['inner']) # should return 'item' but it returns 'dict'
If I remove one dict level from the structure, it works as expected:
from dataclasses import dataclass, field
import dacite
@dataclass
class item:
item_field: str = 'default_value'
@dataclass
class simpler_container:
dict_of_dataclass: Dict[str, item] = field(default_factory=dict)
simpler_dict = {'dict_of_dataclass': {'inner': {'item_field': 'a value'}}}
obj = dacite.from_dict(simpler_container, simpler_dict)
type(obj.dict_of_dataclass['inner']) # returns 'item'
Would it be possible for you to create a package of dacite for conda forge?
If you run this code:
import dataclasses
from dataclasses import dataclass
import dacite
@dataclass
class Person:
height: float = 160
person = Person()
person_dict = dataclasses.asdict(person)
new_person_1 = dacite.from_dict(data_class=Person, data=person_dict)
it gives this error:
WrongTypeError: wrong type for field "height" - should be "float" instead of "int"
I think it should be able to safely cast what it interprets as ints to be floats. Or alternatively, perhaps when doing asdict
, it should save it as a float 160.0
. Not sure which is better, or if the current behavior is desired since the user kind of erred with their datatype (although seems a bit user-unfriendly).
BTW, two "workarounds":
height: float = 160.0
or cast to floatnew_person_2 = dacite.from_dict(data_class=Person, data=person_dict,
config=dacite.Config({float: float}))
Happy to submit a PR if you like.
It looks like dacite errors out when given a dict with a list of loaded dataclasses already in it. Because of the way json.loads
works, this is something that pops up when trying to load dataclasses directly. json.loads
starts at the deepest object, and works its way back up. Dataclasses with a list of other dataclasses will have each object in the list loaded first.
Here is an example that manifests the bug:
>>> import dacite
>>>
>>> from dataclasses import dataclass, field
>>> from typing import List
>>>
>>> @dataclass
... class X:
... text: str = "default"
...
>>> @dataclass
... class Y:
... x: X = X()
...
>>> @dataclass
... class Y:
... x_list: List[X] = field(default_factory=list)
...
>>> y_dict = {"x_list": [X(), X()]}
>>> dacite.from_dict(Y, y_dict)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 78, in from_dict
field=field,
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 189, in _inner_from_dict_for_collection
) for item in data)
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 189, in <genexpr>
) for item in data)
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 61, in from_dict
_validate_config(data_class, data, config)
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 101, in _validate_config
_validate_config_data_key(data, config, 'remap')
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 122, in _validate_config_data_key
input_data_keys = set(data.keys())
AttributeError: 'X' object has no attribute 'keys'
And here is an example of when this pops up in practical code:
>>> import json
>>> from typing import Type
>>>
>>> def dataclass_hook(obj):
... class_index = {dc.__name__: dc for dc in [X, Y]}
... try:
... type_name = obj['_type']
... except (KeyError, IndexError):
... return obj
... try:
... data_type = class_index[type_name]
... except KeyError:
... return obj
... # lets print each step so we can see what order objects are being loaded,
... # and what values look like when the error is thrown
... print(data_type, obj)
... return dacite.from_dict(data_type, obj)
...
>>> data = {
... "_type": "Y",
... "x_list": [
... {"_type": "X", "text": "value one"},
... {"_type": "X", "text": "value two"},
... ]
... }
>>> json_string = json.dumps(data)
>>> loaded = json.loads(json_string, object_hook=dataclass_hook)
<class '__main__.X'> {'_type': 'X', 'text': 'value one'}
<class '__main__.X'> {'_type': 'X', 'text': 'value two'}
<class '__main__.Y'> {'_type': 'Y', 'x_list': [X(text='value one'), X(text='value two')]}
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 361, in loads
return cls(**kw).decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
File "<input>", line 14, in dataclass_hook
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 78, in from_dict
field=field,
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 189, in _inner_from_dict_for_collection
) for item in data)
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 189, in <genexpr>
) for item in data)
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 61, in from_dict
_validate_config(data_class, data, config)
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 101, in _validate_config
_validate_config_data_key(data, config, 'remap')
File "/Users/williampeake/venvs/spanreed-py-37/lib/python3.7/site-packages/dacite.py", line 122, in _validate_config_data_key
input_data_keys = set(data.keys())
AttributeError: 'X' object has no attribute 'keys'
... + add info that dacite
is not a serialiation/deserialization library
I have had a bit of a problem with dacite's way of handling type conversions from str
to Enum
. In pre 1.0 I had to use cast
to be able use Enum
's in a dataclass
. For example:
from dataclasses import dataclass
from enum import Enum
import dacite
class Thing(Enum):
Foo = "foo"
Bar = "bar"
@dataclass
class Container:
simple: str
thing: Thing
data = {
"simple": "value",
"thing": "foo"
}
container = dacite.from_dict(data_class=Container, data=data, config=dacite.Config(cast=["thing"]))
It seems that 1.0 removes a lot of functionality, but I can still work with enums by using the type_hooks
-feature in the Config
.
container = dacite.from_dict(data_class=Container, data=data, config=dacite.Config(type_hooks={Thing: Thing}))
I am not sure how dacite
is used but it seems to me that as a user of the library, my expectation is that types are automatically converted according to the definition of the dataclass that is passed to from_dict()
. Attempting to automatically transform the types in the case of non-standard-library types would follow the principle of least astonishment and I do not think it is hard to implement.
For example it might be possible to populate a default type_hooks
in from_dict
, and then update it based on the configuration provided by the user. The user is still able to pass custom transformations so the change should be backwards compatible with 1.0. A very, very crude example
# In this example,data_class is the Container from my example
default = {f.type: f.type for f in dataclasses.fields(data_class)}
default.update(config.type_hooks)
updated_hooks = default
try:
field_data = data[field.name]
transformed_value = transform_value(
type_hooks=updated_hooks, target_type=field.type, value=field_data
)
value = _build_value(type_=field.type, data=transformed_value, config=config)
except DaciteFieldError as error:
error.update_path(field.name)
raise
@konradhalas I might be open to contributing such functionality in a PR if you're not against the idea and if you're willing to provide some idea as a maintainer of how you'd like that implementation to look like.
Hi. Thank you for your project. Looks like just the thing I need.
Running into the following
python
Python 3.7.0 (default, Jul 17 2018, 11:04:33)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dataclasses import dataclass
>>> from dacite import from_dict
>>>
>>>
>>> @dataclass
... class User:
... name: str
... age: int
... is_active: bool
...
>>>
>>> data = {
... 'name': 'john',
... 'age': 30,
... 'is_active': True,
... }
>>>
>>> user = from_dict(data_class=User, data=data)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.7/site-packages/dacite.py", line 90, in from_dict
elif not _is_instance(field.type, value):
File "/usr/local/lib/python3.7/site-packages/dacite.py", line 248, in _is_instance
return isinstance(value, t.__origin__)
AttributeError: type object 'str' has no attribute '__origin__'
>>>
I'm getting a confusing error with dacite 0.0.25
. I'm using dacite in anAWS Lambda function (which is why the traceback looks a little funny), and I'm getting the following exception when importing dacite.config.Config
:
AttributeError: module 'typing' has no attribute '_ClassVar'
Traceback (most recent call last):
File "/var/lang/lib/python3.7/imp.py", line 234, in load_module
return load_source(name, filename, file)
File "/var/lang/lib/python3.7/imp.py", line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 696, in _load
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/var/task/cris/queue_event.py", line 8, in <module>
from dacite import Config, from_dict
File "/var/task/dacite/__init__.py", line 1, in <module>
from dacite.config import Config
File "/var/task/dacite/config.py", line 14, in <module>
@dataclass
File "/var/task/dataclasses.py", line 958, in dataclass
return wrap(_cls)
File "/var/task/dataclasses.py", line 950, in wrap
return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)
File "/var/task/dataclasses.py", line 801, in _process_class
for name, type in cls_annotations.items()]
File "/var/task/dataclasses.py", line 801, in <listcomp>
for name, type in cls_annotations.items()]
File "/var/task/dataclasses.py", line 659, in _get_field
if (_is_classvar(a_type, typing)
File "/var/task/dataclasses.py", line 550, in _is_classvar
return type(a_type) is typing._ClassVar
For some reason, the 0.0.25 release introduces dataclasses==0.6
as a dependency even though I'm on python 3.7.2
% pip --version
pip 19.0.3 from /usr/local/opt/pyenv/versions/3.7.2/envs/test/lib/python3.7/site-packages/pip (python 3.7)
% python --version
Python 3.7.2
% pip freeze
% pip install dacite
Collecting dacite
Using cached https://files.pythonhosted.org/packages/48/a8/218d76025df9b63f6896f91a432a2ccbc658efb8c404e2d0af8c28f89dde/dacite-0.0.25-py3-none-any.whl
Collecting dataclasses (from dacite)
Using cached https://files.pythonhosted.org/packages/26/2f/1095cdc2868052dd1e64520f7c0d5c8c550ad297e944e641dbf1ffbb9a5d/dataclasses-0.6-py3-none-any.whl
Installing collected packages: dataclasses, dacite
Successfully installed dacite-0.0.25 dataclasses-0.6
% pip freeze
dacite==0.0.25
dataclasses==0.6
If I attempt to install the 0.0.24, the dataclasses dependency is not present
pip install dacite==0.0.24
Collecting dacite==0.0.24
Using cached https://files.pythonhosted.org/packages/60/a8/50cc19f7254f688c41140fd33531499f7d0b529617757c119a5b0e95ce01/dacite-0.0.24-py3-none-any.whl
Installing collected packages: dacite
Successfully installed dacite-0.0.24
% pip freeze
dacite==0.0.24
I went through the setup.py
and cannot really understand where the dataclasses dependency leaks in on a 3.7 python. At the moment I've resorted to downgrading to 0.0.24 for my use case.
This is a dead link use here.
Examples for VK_API_VERSION_1_1 and VK_API_VERSION_1_0 should be provided.
Here's minimal example
import dacite
import dataclasses
@dataclasses.dataclass
class Example:
a: int
b: str = dataclasses.field(init=False)
def __post_init__(self):
self.b = 'GOT IT'
example = Example(1)
example_dict = dataclasses.asdict(example)
example_dict.pop('b')
example_from_dacite = dacite.from_dict(Example, example_dict)
Traceback
~/Projects/.venv/lib/python3.7/site-packages/dacite/core.py in from_dict(data_class, data, config)
53 value = get_default_value_for_field(field)
54 except DefaultValueNotFoundError:
---> 55 raise MissingValueError(field.name)
56 if field.init:
57 init_values[field.name] = value
MissingValueError: missing value for field "b"
Is there a way to type cast all fields, including on nested models? I know I can set cast
and include dotted items, but this would not work for complex object models, especially where a particular class could show up in different places in the tree.
Would be nice if I could pass a callable that would return True or False depending on the class + field name, or if dacite could look for a particular method on my class, etc.
Anyway it's not critical for me, but just giving a feature idea.
Probably related to type validation, but I noticed that dacite ignores extraneous data passed to from_dict
.
I expected the last example to also raise an exception.
> @dataclass
> class A:
> x: str
> A(x="hello")
A(x='hello')
> A(x="hello", y="world")
TypeError: __init__() got an unexpected keyword argument 'y'
> dacite.from_dict(data_class=A, data={"x":"hello", "y": "world"})
A(x='hello')
from dataclasses import dataclass
from typing import List
import dacite
@dataclass
class A:
a: List[int]
assert dacite.from_dict(A, {'a': ['1']})
dacite
doesn't raise WrongTypeError
even if we provide different type into A.a
. I know that dacite is not designed to perform data validation, so just wondering it's intended behavior. ๐
1.0.2
3.7.3
When you have field with enum.Enum type it is not possible to parse json string and map it to dataclass. It will be nice to be able to convert string to enum field in background.
from dataclasses import dataclass
from typing import Sequence
import dacite
@dataclass(frozen=True)
class Foo:
bar: Sequence[int]
dacite.from_dict(Foo, {"bar": []}) # FAIL: TypeError("object() takes no parameters")
# raised in dacite/core.py:105
The error is raised right here
Line 105 in 55051de
This is due to the fact that collection_cls is resolved in this case to collections.abc.Sequence
.
So I cannot initialize a field hinted as an abstract type whereas the given value is obviously of a concrete & compatible type ?
Why such a limitation ? Why simply not use the value type as long as it is compatible with the annotated type ?
Given:
import dacite
from dataclasses import dataclass
from typing import List, Union
@dataclass
class Foo:
x: int
@dataclass
class Bar:
y: int
@dataclass
class Action:
target: str
@dataclass
class FooAction(Action):
foo: Foo
@dataclass
class BarAction(Action):
bar: Bar
@dataclass
class Config:
actions: List[Union[FooAction, BarAction]]
The following misses a value for target
cfg = dacite.from_dict(Config, {'actions': [
{'foo': {'x': 1}},
{'bar': {'y': 2}},
]})
and raises:
dacite.exceptions.UnionMatchError: can not match type "dict" to any type of "actions" union: typing.Union[__main__.FooAction, __main__.BarAction]
The very same exception is raised if the type of x
is wrong:
cfg = dacite.from_dict(Config, {'actions': [
{'foo': {'x': '1'}, 'target': 'target'},
{'bar': {'y': 2}, 'target': 'target'},
]})
while this parses fine:
cfg = dacite.from_dict(Config, {'actions': [
{'foo': {'x': 1}, 'target': 'target'},
{'bar': {'y': 2}, 'target': 'target'},
]})
It would be great if dacite could produce more specific error messages when handling unions of complex types.
dacite 0.0.23
>>> from dataclasses import dataclass
>>> from enum import Enum
>>>
>>> import dacite
>>>
>>>
>>> class TestEnum(Enum):
... none = 1
... some_option = 2
...
>>> @dataclass
... class TestData:
... some_field: TestEnum
...
>>>
>>> def _transform_none(enum_field):
... if not enum_field:
... return TestEnum.none
... return TestEnum[enum_field]
...
>>>
>>> dacite.from_dict(TestData, {'some_field': None}, dacite.Config(transform={'some_field': _transform_none}))
Traceback (most recent call last):
File "<input>", line 1, in <module>
dacite.from_dict(TestData, {'some_field': None}, dacite.Config(transform={'some_field': _transform_none}))
File "/Users/dmitry/.local/pyenv/versions/3.7.0/envs/sandbox/lib/python3.7/site-packages/dacite.py", line 92, in from_dict
raise WrongTypeError(field, value)
dacite.WrongTypeError: wrong type for field "some_field" - should be "TestEnum" instead of "NoneType"
Use case is that a script is ingesting some JSON and the incoming key names do not conform to Python naming conventions (e.g., they're Java or JavaScript naming conventions instead). I'd like to be able to supply a function to rename them to the desired target dataclass property names, perhaps using something like 'inflection' package:
Example:
from dataclasses import dataclass
from dacite import from_dict
from inflection import underscore
@dataclass
class Metrics:
blocked_duration_millis: int
blocked_time_millis: int
buildable_duration_millis: int
data = {
"blockedDurationMillis": 1803,
"blockedTimeMillis": 1803,
"buildableDurationMillis": 0
}
result = from_dict(Metrics, data, config=Config(rename_keys=underscore)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.