ramonhagenaars/jsons

🐍 A Python lib for (de)serializing Python objects to/from JSON

• Turn Python objects into dicts or (json)strings and back
• No changes required to your objects
• Easily customizable and extendable
• Works with dataclasses, attrs and POPOs

💗 this lib? Leave a ★ and tell your colleagues!

Example of a model to serialize:

>>> @dataclass
... class Person:
...    name: str
...    birthday: datetime
...
>>> p = Person('Guido van Rossum', birthday_guido)

Example of using jsons to serialize:

>>> out = jsons.dump(p)
>>> out
{'birthday': '1956-01-31T12:00:00Z', 'name': 'Guido van Rossum'}

Example of using jsons to deserialize:

>>> p2 = jsons.load(out, Person)
>>> p2
Person(name='Guido van Rossum', birthday=datetime.datetime(1956, 1, 31, 12, 0, tzinfo=datetime.timezone.utc))

Installation

pip install jsons

Usage

import jsons

some_instance = jsons.load(some_dict, SomeClass)  # Deserialization
some_dict = jsons.dump(some_instance)  # Serialization

In some cases, you have instances that contain other instances that need (de)serialization, for instance with lists or dicts. You can use the typing classes for this as is demonstrated below.

from typing import List, Tuple
import jsons

# For more complex deserialization with generic types, use the typing module
list_of_tuples = jsons.load(some_dict, List[Tuple[AClass, AnotherClass]])

(For more examples, see the FAQ)

Documentation

• Main documentation
• API docs
• FAQ

Meta

Recent updates

1.6.3

• Bugfix: a string was sometimes unintentionally parsed into a datetime.

1.6.2

• Bugfix: fork_insts were not propagated in default_list_deserializer (thanks to patrickguenther).

1.6.1

• Bugfix: Loading dicts with hashed keys could cause an error due to being loaded twice (thanks to georgeharker).
• Bugfix: IntEnums were not serialized with their names when use_enum_name=True (thanks to georgeharker).
• Bugfix: Named tuples did not use typing.get_type_hints for getting the types, causing trouble in future annotations (thanks to georgeharker).

1.6.0

• Feature: Support for Python3.10.
• Feature: Support for attrs.

1.5.1

• Bugfix: ZoneInfo failed to dump if attached to a datetime.

1.5.0

• Feature: Support for ZoneInfo on Python3.9+.
• Change: microseconds are no longer stripped by default (thanks to pietrodn).

1.4.2

• Bugfix: get_origin did not work with python3.9+ parameterized collections (e.g. dict[str, str]).

1.4.1

• Bugfix: Types of attributes that are not in the constructor were not properly looked for. See issue #128.

1.4.0

• Feature: DefaultDicts can now be deserialized.
• Feature: Dicts with any (hashable) key can now be dumped and loaded.
• Feature: Suppress specific warnings.
• Bugfix: Loading a verbose-serialized object in a list could sometimes deserialize that object as a parent class.
• Bugfix: Unwanted stringification of NoneValues is now prevented in Optionals and Unions with NoneType.
• Bugfix: Fixed a bug with postponed annotations and dataclasses. See also Issue34776.
• Bugfix: Types of attributes that are not in the constructor are now looked for in annotations.

1.3.1

• Bugfix: Fixed bug where classmethods were included in the serialized result.

1.3.0

• Feature: Added warn_on_fail parameter to default_list_deserializer that allows to continue deserialization upon errors.
• Feature: Added transform that can transform an object to an object of another type.
• Feature: Added serializer and deserializer for pathlib.Path (thanks to alexmirrington).
• Change: When loading a list fails, the error message now points to the failing index.
• Bugfix: Fixed bug when dumping an object with an innerclass.

1.2.0

• Bugfix: Fixed bug with postponed typehints (PEP-563).
• Bugfix: Loading an invalid value targeting an optional did not raise.
• Bugfix: Loading a dict did not properly pass key_transformers.
• Bugfix: Loading a namedtuple did not properly use key_transformers.
• Bugfix: Utilized __annotations__ in favor _field_types because of deprecation as of 3.8.

1.1.2

• Feature: Added __version__ which can be imported from jsons
• Bugfix: Dumping a tuple with ellipsis failed in strict mode.

1.1.1

• Feature: Added a serializer for Union types.
• Change: Exceptions are more clear upon deserialization failure (thanks to haluzpav).
• Change: You can no longer announce a class with a custom name.
• Bugfix: Fixed dumping optional attributes.
• Bugfix: Dataclasses inheriting from JsonSerializable always dumped their attributes as if in strict mode.

1.1.0

• Feature: Added strict parameter to dump to indicate that dumping a certain cls will ignore any extra data.
• Feature: When using dump(obj, cls=x), x can now be any class (previously, only a class with __slots__).
• Feature: Support for dumping Decimal (thanks to herdigiorgi).
• Feature: Primitives are now cast if possible when dumping (e.g. dump(5, str)).
• Feature: Dumping iterables with generic types (e.g. dump(obj, List[str])) will now dump with respect to that types (if strict)
• Feature: The default_dict serializer now optionally accepts types: Optional[Dict[str, type]].
• Change: Improved performance when dumping using strict=True (up to 4 times faster!).
• Bugfix: set_validator with multiple types did not work.

1.0.0

• Feature: Added a serializer/deserializer for time.
• Feature: Added a serializer/deserializer for timezone.
• Feature: Added a serializer/deserializer for timedelta.
• Feature: Added a serializer/deserializer for date.
• Bugfix: Dumping verbose did not store the types of dicts (Dict[K, V]).
• Bugfix: Loading with List (no generic type) failed.
• Bugfix: Loading with Dict (no generic type) failed.
• Bugfix: Loading with Tuple (no generic type) failed.

Contributors

Special thanks to the following contributors of code, discussions or suggestions:

patrickguenther, davetapley, pietrodn, georgeharker, aecay, bibz, thijss, alexmirrington, tirkarthi, marksomething, herdigiorgi, jochembroekhoff, robinklaassen, ahmetkucuk, casparjespersen, cypreess, gastlich, jmolinski, haluzpav, finetuned89

jsons's Issues

strip_properties does not work on nested objects

Parameter strip_properties of dump function works only for top-level object. @property of nested objects are always included in json. See my gist example:

https://gist.github.com/haluzpav/b578dad460cb3670d55305dc65d9d2bf

Is there a way to also strip properties of nested objects?

Deprecation warnings

Hello!
0.8.5 version introduced 2 deprecation warnings:

.tox/py37/lib/python3.7/site-packages/jsons/__init__.py:88
  /home/glen/prywatne/traktpy/.tox/py37/lib/python3.7/site-packages/jsons/__init__.py:88: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import deque, Mapping

.tox/py37/lib/python3.7/site-packages/jsons/deserializers/default_iterable.py:1
  /home/glen/prywatne/traktpy/.tox/py37/lib/python3.7/site-packages/jsons/deserializers/default_iterable.py:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping, Iterable

-- Docs: https://docs.pytest.org/en/latest/warnings.html

Have a good Sunday :)

Add strip_attr to omit specific attributes from being dumped

Sometimes it might be useful to be able to specify attributes that are not to be dumped.

jsons.dump(obj, strip_attr='attr1')

# Or strip multiple:
jsons.dump(obj, strip_attr=('attr1', 'attr2'))

Poor Performance on Serialization

It seems like serialization using jsons.dump(obj) is at least 10 times slower than writing custom serialization method to each class. In my case, obj was a nested dataclasses and data was in the order of 100MBs.

I can pull up some numbers but I just wanted to start a discussion if this is a known issue and if so, what might cause this?

Serialization of Enum type should use values rather than keys

Given the following example Enum type:

from enum import Enum

class TestEnum(Enum):
    KEY = 'value`

jsons.dumps(TestEnum.KEY) will result in "KEY" (Deserialization works correctly).

However, this is behavior is a little bit irrational in regards to what enum is (used to be in C :) ). Enums classically are used to map some "codenames" for binary values (like numbers 0, 1, 2, 3). Therefore the value of enum should be binary representation generally used in serialization and keys should be something assigned for it on the high level (on runtime/ in the codebase).

---

Update: just for a reference standard lib json module does not serialize Enum's at all.

jsons does not accept None values even if they are valid values

Consider the following examples:

>>> class T(NamedTuple):
               o: Optional[str]

>>> jsons.load({"o": None}, T)

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/jsons/_load_impl.py", line 85, in load
    return deserializer(json_obj, cls, **kwargs_)
  File "/usr/local/lib/python3.7/site-packages/jsons/deserializers/default_tuple.py", line 19, in default_tuple_deserializer
    return default_namedtuple_deserializer(obj, cls, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/jsons/deserializers/default_tuple.py", line 51, in default_namedtuple_deserializer
    raise UnfulfilledArgumentError(msg, field_name, obj, cls)
jsons.exceptions.UnfulfilledArgumentError: No value present in {'o': None} for argument "o"

but also:

>>> class T(NamedTuple):
               o: Any
    
>>> jsons.load({"o": None}, T)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/jsons/_load_impl.py", line 85, in load
    return deserializer(json_obj, cls, **kwargs_)
  File "/usr/local/lib/python3.7/site-packages/jsons/deserializers/default_tuple.py", line 19, in default_tuple_deserializer
    return default_namedtuple_deserializer(obj, cls, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/jsons/deserializers/default_tuple.py", line 51, in default_namedtuple_deserializer
    raise UnfulfilledArgumentError(msg, field_name, obj, cls)
jsons.exceptions.UnfulfilledArgumentError: No value present in {'o': None} for argument "o"

Textual type hints crash deserialization

This issue is possibly related Issue#26.

Textual type hints cannot be deserialized as jsons does not handle string type hints well. This makes it impossible to deserialize into a recursive structure.

Example:

from dataclasses import dataclass
from typing import Optional
import jsons


@dataclass
class Node:
    value: object
    next: Optional['Node']


linked_list = Node(10, Node(20, Node(30, None)))
dumped = jsons.dump(linked_list)

So far so good. The value of dumped is now:

{'next': {'next': {'next': None, 'value': 30}, 'value': 20}, 'value': 10}

Now to deserialize:

jsons.load(dumped, Node)

This raises the following error:

Traceback (most recent call last):
  File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 128, in load
    return deserializer(json_obj, cls, **kwargs_)
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 31, in default_object_deserializer
    constructor_args = _get_constructor_args(obj, cls, **kwargs)
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 58, in _get_constructor_args
    constructor_args_in_obj = {key: value for key, value in args_gen if key}
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 58, in <dictcomp>
    constructor_args_in_obj = {key: value for key, value in args_gen if key}
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 57, in <genexpr>
    in signature_parameters.items() if sig_key != 'self')
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 72, in _get_value_for_attr
    if obj and sig_key in obj:
TypeError: argument of type 'int' is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Projects/Personal/jsons/effe.py", line 20, in <module>
    jsons.load(dumped, Node)
  File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 128, in load
    return deserializer(json_obj, cls, **kwargs_)
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 31, in default_object_deserializer
    constructor_args = _get_constructor_args(obj, cls, **kwargs)
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 58, in _get_constructor_args
    constructor_args_in_obj = {key: value for key, value in args_gen if key}
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 58, in <dictcomp>
    constructor_args_in_obj = {key: value for key, value in args_gen if key}
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 57, in <genexpr>
    in signature_parameters.items() if sig_key != 'self')
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 75, in _get_value_for_attr
    meta_hints, **kwargs)
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 109, in _get_value_from_obj
    value = load(obj[sig_key], arg_cls, meta_hints=new_hints, **kwargs)
  File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 132, in load
    raise DeserializationError(str(err), json_obj, cls)
jsons.exceptions.DeserializationError: argument of type 'int' is not iterable

Unproper handling of Optional

Jsons fails on deserializing into a class with optional fields.

class C:
    def __init__(self, x: typing.Optional[int] = None):
        self.x = x

loaded = jsons.load({"x": 1}, C)

This raises:

AttributeError: '_SpecialForm' object has no attribute '__name__'

[Bug] from __future__ import annotations

Hello!

from __future__ import annotations literally destroys the library.

Quick example:

foo.py

from __future__ import annotations

from dataclasses import dataclass


@dataclass
class Spam:
    eggs: int


@dataclass
class Baz:
    qux: int
    spam: Spam

bar.py

from typing import List

import jsons

from foo import Baz

data = [{"qux": 0, "spam": {"eggs": 1}}]

x = jsons.load(data, List[Baz])

Expected result: [Baz(qux=0, spam=Spam(eggs=1))]
Result:

>>> import bar
Traceback (most recent call last):
  File "/home/glen/traktpy/venv/lib/python3.7/site-packages/jsons/_main_impl.py", line 146, in load
    return deserializer(json_obj, cls, **kwargs_)
TypeError: 'NoneType' object is not callable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

...

  File "/home/glen/traktpy/venv/lib/python3.7/site-packages/jsons/_main_impl.py", line 150, in load
    raise DeserializationError(str(err), json_obj, cls)
jsons.exceptions.DeserializationError: 'NoneType' object is not callable
>>> 

If you remove from __future__ import annotations from foo.py everything works as expected.

https://www.python.org/dev/peps/pep-0563/
from __future__ import annotations stops the interpreter from executing type annotations, they are stored as str instead. Types may be revealed during runtime using typing.get_type_hints(). This is going to be the default behavior in python4.

strip_properties removes also privates

As in title, strip_privates gets "over-written" by strip_properties. Example:

https://gist.github.com/haluzpav/628477bdeee8e33ed40e07c2b786c2dd

Kinda related to previous issue #3.

DeserializationError with datetime or type hint

I'm a beginner with Python. So I'm not sure if I use jsons wrong.

The first exemple below works well.

import jsons
from dataclasses import dataclass

@dataclass
class ChatUser:
    uid: str
    name: str

dumped = jsons.dumps(ChatUser('012451', 'Casimir'), strip_privates=True, strip_microseconds=True, verbose=True)
print(dumped)
instance = jsons.loads(dumped, ChatUser)
print(instance)

Both following examples fail with DeserializationError. Can someone help?

import jsons
from dataclasses import dataclass
import datetime

@dataclass
class ChatUser:
    uid: datetime
    name: str

dumped = jsons.dumps(ChatUser(datetime.datetime.now(), 'Casimir'), strip_privates=True, strip_microseconds=True, verbose=True)
print(dumped)
instance = jsons.loads(dumped, ChatUser)
print(instance)`

import jsons
from dataclasses import dataclass
from typing import List, NewType
Uid = NewType("Uid", str)

@dataclass
class ChatUser:
    uid: Uid
    name: str

dumped = jsons.dumps(ChatUser(Uid('012451'), 'Casimir'), strip_privates=True, strip_microseconds=True, verbose=True)
print(dumped)
instance = jsons.loads(dumped, ChatUser)
print(instance)

Mention strip_* options in README

They are too useful to be left out. :)

Invalid type hints result in cryptic error messages

When a class has an invalid type hint, deserializing into that class may cause a DeserializationError that does not help solving the issue. The error should be more descriptive.

Example:

import datetime  # <-- Note that the module is imported, not the class.
from dataclasses import dataclass
import jsons


@dataclass
class User:
    name: str
    birthday: datetime


dumped = { 'name': 'Albert', 'birthday': '1879-03-14T11:30:00+01:00' }
jsons.load(dumped, User)

This causes:

Traceback (most recent call last):
  File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 128, in load
    return deserializer(json_obj, cls, **kwargs_)
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 31, in default_object_deserializer
    constructor_args = _get_constructor_args(obj, cls, **kwargs)
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 58, in _get_constructor_args
    constructor_args_in_obj = {key: value for key, value in args_gen if key}
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 58, in <dictcomp>
    constructor_args_in_obj = {key: value for key, value in args_gen if key}
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 57, in <genexpr>
    in signature_parameters.items() if sig_key != 'self')
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 75, in _get_value_for_attr
    meta_hints, **kwargs)
  File "C:\Projects\Personal\jsons\jsons\deserializers\default_object.py", line 109, in _get_value_from_obj
    value = load(obj[sig_key], arg_cls, meta_hints=new_hints, **kwargs)
  File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 119, in load
    deserializer = _get_deserializer(cls, fork_inst)
  File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 145, in _get_deserializer
    fork_inst._classes_deserializers, fork_inst)
  File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 153, in _get_lizer
    cls_name = get_class_name(cls, str.lower, fork_inst=fork_inst)
  File "C:\Projects\Personal\jsons\jsons\_common_impl.py", line 51, in get_class_name
    module = _get_module(cls)
  File "C:\Projects\Personal\jsons\jsons\_common_impl.py", line 104, in _get_module
    module = cls.__module__
AttributeError: module 'datetime' has no attribute '__module__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Projects/Personal/jsons/effe.py", line 14, in <module>
    jsons.load(dumped, User)
  File "C:\Projects\Personal\jsons\jsons\_main_impl.py", line 132, in load
    raise DeserializationError(str(err), json_obj, cls)
jsons.exceptions.DeserializationError: module 'datetime' has no attribute '__module__'

Should deserialization silently pass on mismatching arguments?

Let's consider the following example:

In [30]: @dataclass
    ...: class Test:
    ...:     a: int
    ...:     b: int
    ...:

In [31]: Test(1,2)
Out[31]: Test(a=1, b=2)

In [32]: jsons.dumps(Test(1,2))
Out[32]: '{"a": 1, "b": 2}'

In [33]: jsons.loads('{"a": 1, "b": 2, "c": 3}', Test)
Out[33]: Test(a=1, b=2)

I would expect jsons to fail in the last statement, as the protocol is not strictly the same.

Is there any rationale for such behavior? If yes, that would be great to have a strict=True argument for jsons.

DeserializationError: Invalid type: "datetime.datetime" when deserializing correct datetime string into Optional[datetime]

Hi,

I have just started using your library and it works great, but I have encountered this weird error. I have JSON string with correct datetime and it is being deserialized correctly if type hint is just datetime but if I want to support null in JSON (None in Python) I use typing.Optional[datetime] and it gives me this error: jsons.exceptions.DeserializationError: Invalid type: "datetime.datetime", only arguments of the following types are allowed: str, int, float, bool, list, tuple, set, dict, NoneType.

Minimal code example:

import jsons
import typing
import dataclasses
import datetime

@dataclasses.dataclass
class WithDateOK:
    created: datetime.datetime

@dataclasses.dataclass
class WithDateBad:
    created: typing.Optional[datetime.datetime]

withDateStr = '''{"created":"2019-12-19T11:40:48Z"}'''
withDateDict = jsons.loads(withDateStr)
print(withDateDict)
withDateObj = jsons.load(withDateDict, WithDateOK)
withDateObj = jsons.load(withDateDict, WithDateBad)
print(type(withDateObj))
print(withDateObj)

Complete Traceback:

Traceback (most recent call last):
  File "/home/vanko/PycharmProjects/playground/jsonpickle_playing.py", line 641, in <module>
    withDateObj = jsonsLib.load(withDateDict, WithDate)
  File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/_load_impl.py", line 98, in load
    return _do_load(json_obj, deserializer, cls, initial, **kwargs_)
  File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/_load_impl.py", line 110, in _do_load
    result = deserializer(json_obj, cls, **kwargs)
  File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/deserializers/default_object.py", line 39, in default_object_deserializer
    constructor_args = _get_constructor_args(obj, cls, **kwargs)
  File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/deserializers/default_object.py", line 70, in _get_constructor_args
    **kwargs)
  File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/deserializers/default_object.py", line 94, in _get_value_for_attr
    meta_hints, **kwargs)
  File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/deserializers/default_object.py", line 134, in _get_value_from_obj
    value = load(obj[sig_key], cls_, meta_hints=new_hints, **kwargs)
  File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/_load_impl.py", line 82, in load
    json_obj, cls, fork_inst, kwargs.get('_inferred_cls', False))
  File "/home/vanko/PycharmProjects/playground/venv/lib/python3.7/site-packages/jsons/_load_impl.py", line 197, in _check_and_get_cls_and_meta_hints
    raise DeserializationError(msg, json_obj, cls)
jsons.exceptions.DeserializationError: Invalid type: "datetime.datetime", only arguments of the following types are allowed: str, int, float, bool, list, tuple, set, dict, NoneType

If I change datetime string to null, it works fine and created is None as expected.

Here is live demo example demonstrating the error. Is this behaviour correct or is this a bug? Thanks

Regards,
Adam

Support for UUID type

Simply transform as follows:
Serializer: uuid_str = str(uuid)
Deserializer: uuid = uuid.UUID(uuid_str)

I am willing to implement if there is a desire to have this in the library.

Primitive types should be parsed upon loading if possible

Example:

output = jsons.load(42, str)

Current output:

42

Desired output:

'42'

Currently, there is no check whatsoever as jsons gladly loads a primitive of some type into an attribute of another primitive type. This can lead to undesired behaviour.

Another example:

output = jsons.load('fortytwo', int)

Current output:

'fortytwo'

Desired output: a DeserializationError is raised.

Should named tuple be serialized as tuple or like dict

Similarly to #18 I start being biased towards thinking that because of "readability counts" NamedTuples should be expressed like to dicts. The whole point of naming tuple positions is because of poor readability of tuples. Then we totally drop it on serialization level when we already decided we go for readability.

Extra Parameters are set as new class variables

When I try to deserialize with extra parameters, the library creates a new variable in the object. I expect it to not create any new variables and not throw any exceptions.

>>> from dataclasses import dataclass
>>> import jsons
>>> 
>>> @dataclass
... class A:
...     x: int
... 
>>> a = jsons.load({'x': 1, 'y': 'abc'}, A)
>>> a.y
'abc'
>>> import dataclasses
>>> dataclasses.asdict(a)
{'x': 1}

I guess this happens in _set_remaining_attrs. I suggest making this call optional.
OR
When we serialize it again, it should ignore these fields.

Support for NewType

jsons should support NewType so you could do:

Uid = NewType('uid', str)


@dataclass
class User:
    uid: Uid
    name: str

And them dump and load instances without DeserializationErrors.

`jsons.dumps(np.array([0]))` reaches maximum recursion depth

import jsons
import numpy as np
jsons.dumps(np.array([0]))

causes the following error message to repeat:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/anaconda3/lib/python3.7/site-packages/jsons/_main_impl.py", line 63, in dump
    return serializer(obj, cls=cls, **kwargs_)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/jsons/serializers/default_object.py", line 56, in default_object_serializer
    **kwargs_)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/jsons/serializers/default_dict.py", line 25, in default_dict_serializer
    strip_nulls=strip_nulls, **kwargs)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/jsons/_main_impl.py", line 65, in dump
    raise SerializationError(str(err))
jsons.exceptions.SerializationError: maximum recursion depth exceeded in comparison

performance issues

Hi there,

Loading a 1Go JSON file may take a while. Using the standard JSON API with the same JSON file may take approximatively 35 seconds.

with jsons: time~ 20min

    with open(json_path, "r") as json_file:
        content = json_file.readlines()
        json_data = jsons.loads(content[0], CustomObject)

The readlines function take 2 or 3 seconds.

with json: time~ 35s

    with open(json_path, "r") as json_file:
        content = json.load(json_file)
        json_data = CustomObject(**content)

What I'm doing wrong ?

Can't register a forward reference as a deserializer

set_deserializer() accepts either a single type or a sequence of types. While it's fine to register a generic type that holds a forward reference (e.g. a string value whose loading is deferred to later, useful when model would otherwise require a circular import), you can't register just a forward reference.

That's because a forward reference is a string, and strings are sequences. Can an exception be made here? A simple if isinstance(cls, Sequence) and not isinstance(cls, str): should do.

Clearer errors should be raised upon serialization problems

More clear errors should be raised by jsons upon problems during serialization. For instance, it should be clear which attributes cause problems when dumping an object.

Related: Issue#46

mypy type hinting not working

λ  .venv/bin/mypy inpai
inpai/params_definition.py:8: error: Cannot find module named 'jsons'
inpai/execution/monitors/base_monitor.py:3: error: Cannot find module named 'jsons'
inpai/execution/monitors/base_monitor.py:3: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
inpai/execution/monitors/gui_monitor.py:6: error: Cannot find module named 'jsons'
inpai/execution/monitors/elasticsearch_stdout_monitor.py:4: error: Cannot find module named 'jsons'

https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports

If the module is a third party library, you must make sure that there are type hints available for that library. Mypy by default will not attempt to infer the types of any 3rd party libraries you may have installed unless they either have declared themselves to be PEP 561 compliant stub package or have registered themselves on typeshed, the repository of types for the standard library and some 3rd party libraries.

jsons should be stricter on loading None

When deserializing, jsons accepts None when not in "strict-mode". It should not accept None at all, except when the target class is one of Any, NoneType, Optional[...], Union[None, ...].

Rationale:

# Suppose some_obj happens to be None:
loaded_obj = jsons.load(some_obj, SomeClass)  # loaded_obj is now None.
...
# Somewhere else in your code, you do:
loaded_obj.some_method()  # This raises.

The above example shows how accepting a None may be confusing and possibly lead a developer in the wrong direction by falsely declaring the deserialization to be successful. If jsons.load(some_obj, SomeClass) raised an error already, the developer may sooner realize the cause of the problem.

To summarize, here is an example:

jsons.load(None, str)

Current output:

None

Desired output:
DeserializationError is raised.

This was also mentioned in Issue#56.

[Question] mypy_extensions.TypedDict

Hello!

Does the author consider adding support for the TypedDict from mypy_extensions in the future?
TypedDict is extremely helpful for type hinting and code completion and its syntax is very concise.
A detailed description can be found here. It is also likely that it will be found in the standard library typing module in the future versions of Python: python/mypy#5288

As a downside, this would introduce a dependency - mypy_extensions. A solution to that would be making the dependency optional the same way for example Flask has an optional dependency SimpleJSON and only uses it if this package is installed.

JsonSerializable.with_load() forked subclass using .from_json does not pass in forked instance.

You can create a forked JsonSerializable and attach attr_getters to this fork with a single .with_load() call, and then register deserializers. But those deserializers are ignored when using .from_json() on the subclass.

Demo:

import dataclasses, datetime, jsons

defaults = {"ham": lambda: "spam"}
forked = jsons.JsonSerializable.with_load(attr_getters=defaults, fork=True)
forked.set_deserializer(
    lambda td, cls, **kwargs: datetime.timedelta(seconds=td),
    datetime.timedelta
)

@dataclasses.dataclass
class Foo(forked):
    bar: datetime.timedelta
    ham: str

Foo.from_json({"bar": 42})

The above demo leads to a traceback:

Traceback (most recent call last):
  File ".../site-packages/jsons/_load_impl.py", line 85, in load
    return deserializer(json_obj, cls, **kwargs_)
  File ".../site-packages/jsons/deserializers/default_object.py", line 38, in default_object_deserializer
    constructor_args = _get_constructor_args(obj, cls, **kwargs)
  File ".../site-packages/jsons/deserializers/default_object.py", line 69, in _get_constructor_args
    constructor_args_in_obj = {key: value for key, value in args_gen if key}
  File ".../site-packages/jsons/deserializers/default_object.py", line 69, in <dictcomp>
    constructor_args_in_obj = {key: value for key, value in args_gen if key}
  File ".../site-packages/jsons/deserializers/default_object.py", line 68, in <genexpr>
    if sig_key != 'self')
  File ".../site-packages/jsons/deserializers/default_object.py", line 84, in _get_value_for_attr
    if obj and sig_key in obj:
TypeError: argument of type 'int' is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../site-packages/jsons/classes/json_serializable.py", line 123, in from_json
    return cls.load(json_obj, **kwargs)
  File ".../site-packages/jsons/classes/json_serializable.py", line 93, in _wrapper
    return load(inst, cls_, **{**kwargs_, **kwargs})
  File ".../site-packages/jsons/_load_impl.py", line 85, in load
    return deserializer(json_obj, cls, **kwargs_)
  File ".../site-packages/jsons/deserializers/default_object.py", line 38, in default_object_deserializer
    constructor_args = _get_constructor_args(obj, cls, **kwargs)
  File ".../site-packages/jsons/deserializers/default_object.py", line 69, in _get_constructor_args
    constructor_args_in_obj = {key: value for key, value in args_gen if key}
  File ".../site-packages/jsons/deserializers/default_object.py", line 69, in <dictcomp>
    constructor_args_in_obj = {key: value for key, value in args_gen if key}
  File ".../site-packages/jsons/deserializers/default_object.py", line 68, in <genexpr>
    if sig_key != 'self')
  File ".../site-packages/jsons/deserializers/default_object.py", line 87, in _get_value_for_attr
    meta_hints, **kwargs)
  File ".../site-packages/jsons/deserializers/default_object.py", line 122, in _get_value_from_obj
    value = load(obj[sig_key], cls_, meta_hints=new_hints, **kwargs)
  File ".../site-packages/jsons/_load_impl.py", line 89, in load
    raise DeserializationError(str(err), json_obj, cls)
jsons.exceptions.DeserializationError: argument of type 'int' is not iterable

because the replacement implementation for .load() generated by with_load() does not pass in the forked instance:

jsons/jsons/classes/json_serializable.py

Lines 91 to 95 in 03b5ff9

	@classmethod
	def _wrapper(cls_, inst, **kwargs_):
	return load(inst, cls_, **{**kwargs_, **kwargs})
	type_ = cls.fork() if fork else cls
	type_.load = _wrapper

like the original .load() implementation does:

jsons/jsons/classes/json_serializable.py

Lines 134 to 143 in 03b5ff9

	@classmethod
	def load(cls: type, json_obj: object, **kwargs) -> object:
	"""
	See ``jsons.load``.
	:param kwargs: the keyword args are passed on to the serializer
	function.
	:param json_obj: the object that is loaded into an instance of `cls`.
	:return: this instance in a JSON representation (dict).
	"""
	return load(json_obj, cls, fork_inst=cls, **kwargs)

We can work around this by creating a fork first, then using forked.with_default(..., fork_inst=forked), but is less than ideal.

Wrongly serialized utc datetime

The datetime instance is wrongly serialized if initialized from utcnow(). Example:

dt_local = datetime.datetime.now()
print(dt_local)
# >>> 2019-02-16 17:48:34.714603

print(jsons.dump(dt_local))
# >>> 2019-02-16T17:48:34+01:00

dt_utc = datetime.datetime.utcnow()
print(dt_utc)
# >>> 2019-02-16 16:48:34.715108

print(jsons.dump(dt_utc))
# >>> 2019-02-16T16:48:34+01:00
# this last one is clearly wrong

How can I easily skip certain attributes

I can't easily find in the docs how I can specify some attributes of my JsonSerializable class that I don't want to be serialized to JSON.

class MyClass(JsonSerializable):
    def __init__(self):
        self.a = 1
        self.b = 2
        self.log = logging.getLogger(__name__)

In the above case, how can I make self.log get skipped, without having to modify the jsons.dumps call? I want the control of what gets included in the JSON to be with the class, not the place where dump(s/b) is called. So I don't want to have to make them private (with '_') and add strip_privates to dumps. First of all, that would require modifying all dumps calls throughout my code, and secondly, it forces me to change my datamodel, which has consequences elsewhere in the app.
Aside from this example where self.log clearly isn't serializable, and also wouldn't make sense to serialize, there may be other attributes that I want to skip as well. Is this easily possible with jsons? If so, can we add a FAQ question about it? (I'd be happy to contribute it)

Support serialization of recursive datastructures

The following construct causes a problem when serializing:

@dataclass
class Narcissus:
    @property
    def mirror(self):
        return self

Another example:

@dataclass
class A:
    b: object

@dataclass
class B:
    c: object

@dataclass
class C:
    a: object

In case of recursion problems during serialization, the attributes that cause the problem should be omitted and a warning should be triggered.

Issue#46 may be related.

Inherited class is lost while deserialization.

Inherited class is lost while deserialization. In the example below, the object chmsg3 is serialized as a class ChatMessageSection with the weight attribute ('weight': 17, '-cls': 'main.ChatMessageSection'}]). But after deserialization the object becomes a ChatMessage.

import jsons
from dataclasses import dataclass
from typing import List

@dataclass
class ChatUser:
    name: str

@dataclass
class ChatMessage:
    user: ChatUser
    msg_text: str

@dataclass
class ChatMessageSection(ChatMessage):
    weight: int

@dataclass
class ChatMessages:
    msg_list: List[ChatMessage]

chmsg = ChatMessage(ChatUser("Thierry"), "Bonjour")
chmsg2 = ChatMessage(ChatUser("Casimir"), "Hello")
chmsg3 = ChatMessageSection(ChatUser("Leonard"), "Coucou", weight=17)
chat_msgs = ChatMessages([chmsg, chmsg2, chmsg3])

print(chat_msgs, end="\n\n")
dumped = jsons.dump(chat_msgs, strip_microseconds=True, verbose=jsons.Verbosity.WITH_EVERYTHING)
print(dumped, end="\n\n")
instance = jsons.load(dumped)
print(instance, end="\n\n")

Support for all datetime types

Currently, only datetime.datetime is supported. All types from the datetime module should be supported. That includes: date, time, timedelta, timezone, tzinfo.

How to bypass self as a class attribute name error? __init__() got multiple values for argument '__dataclass_self__'

Hi, I have a JSON containing objects that contain field named self and it crashes due to this error: jsons.exceptions.DeserializationError: Could not deserialize value "{'self': 'api/v3/message/0'}" into "__main__.MyClass". __init__() got multiple values for argument '__dataclass_self__'. I understand the error and the cause, but is there a way how to bypass this limition other than renaming the field in the original JSON string before deserializing and also the class attribute? Thanks

Regards,
Adam

Example code (live demo here):

import jsons
import dataclasses


@dataclasses.dataclass
class MyClass:
    self: str


myclassObj = jsons.loads('''{"self":"api/v3/message/0"}''', MyClass)
print(myclassObj)

Registering custom UUID deserializer fails.

Trying to do:

import uuid
import jsons

def custom_uuid_serializer(obj: uuid.UUID, **kwargs) -> str:
	return str(obj)

def custom_uuid_deserializer(obj: str, cls, **kwargs) -> uuid.UUID:
	return uuid.UUID(obj)
	
jsons.set_serializer(custom_uuid_serializer, uuid.UUID) # Failed...
jsons.set_deserializer(custom_uuid_deserializer, uuid.UUID) # Failed...

Exception:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "json_test.py", line 36, in <module>
    data = jsons.dump(hearthbeat) # It will serialize uuid to its bytes
  File "/Users/ahmetkucuk/anaconda3/envs/jsons/lib/python3.7/site-packages/jsons/_dump_impl.py", line 52, in dump
    raise SerializationError(str(err))
jsons.exceptions.SerializationError: 'uuid.UUID'

I think the issue is here: https://github.com/ramonhagenaars/jsons/blob/master/jsons/_lizers_impl.py#L139

result = lizers[pname.lower()] solves the issue.

Private ABC attributes should be stripped by default

When an abstract base class is inherited from, dumping an instance of that implementation will include private abc attributes.

Example:

class A(ABC):
    pass


@dataclass
class B(A):
    x: int


dumped = jsons.dump(B(42))

Output:

{'_abc_cache': [], '_abc_negative_cache': [], '_abc_negative_cache_version': 47, '_abc_registry': [], 'x': 42}

By default, they should not be included.

Should it be possible to discard None-fields from dump?

I'd like to throw in the idea that it should be possible to have the option of omitting a field from the dump, if its value is None. Intended use:

@dataclass
class Foo():
    foo: int
    bar: bool = None

jsons.dump(Foo(1))                          #output: {"foo": 1, "bar": None}
jsons.dump(Foo(1), omit_none=True)          #output: {"foo": 1}
jsons.dump(Foo(1, True))                    #output: {"foo": 1, "bar": True}

Maybe this is already possible using a serializer?

Should dump be able to also pass class type to dictionary

Say I have a class inheritance structure, and I want to generically be able to dump/load with the correct class types in the inheritance structure.

@dataclass
class A(object):
    foo: int

@dataclass
class B(A):
    bar: int

Assume that when loading, I do now know which class type it was dumped from. I would have to analyse the content to determine which class to load as, and then pass this to the cls property of jsons.load method.

Would it make sense to implement an optional flag when dumping to a dictionary that contains information of what class type it was dumped as, that can then be used for loading?

Add support for TypedDict

Python 3.8 introduced the TypedDict:

https://www.python.org/dev/peps/pep-0589/

jsons can't deserialize NamedTuple

When trying to deserialize NamedTuple I got following error:

  File "/Users/cypreess/.virtualenvs/.../lib/python3.6/site-packages/jsons/deserializers.py", line 83, in default_tuple_deserializer
    tuple_types = getattr(cls, '__tuple_params__', cls.__args__)
AttributeError: type object 'MyNamedTuple' has no attribute '__args__'

This is different error than related issue which was solved: #11

AttributeError on deserialization into Tuple[T, ...]

When running following code

from typing import Tuple
import jsons
from dataclasses import dataclass

@dataclass
class A:
    tup: Tuple[str, ...]

a = A(('abc', 'qwe', '123'))
a_ser = jsons.dump(a)
a_deser = jsons.load(a_ser, A)

I get exception AttributeError: 'ellipsis' object has no attribute '__origin__' on the last line.

I can simply use List[str] as type, but it would be nice to be able to deserialize into tuple of variable length.

Inconsistent get_class_name

from datetime import datetime

import jsons
from jsons._common_impl import get_class_name

print(get_class_name(datetime))
# >>> datetime

jsons.dumps(datetime.utcnow())

print(get_class_name(datetime))
# >>> datetime.datetime

It is caused by the fact that classes are automatically announced during dumping, in fully qualified form. If a class is announced, it's name in get_class_name is taken directly from _announced_classes, without taking into account fully_qualified: bool argument.

Verbosity dump does not reconstruct with correct classes

Related to this feature request, I wanted to try it out, but I can not mange to re-construct the objects given the metadata structure. Check out this script:

import jsons
from dataclasses import dataclass

@dataclass
class BarBase():
    pass

@dataclass
class BarA(BarBase):
    a: int

@dataclass
class BarB(BarBase):
    b: int

@dataclass
class Foo():
    bar: BarBase

if __name__ == "__main__":
    f = Foo(bar=BarA(a=5))
    d = jsons.dump(f, verbose=True)
    o = jsons.load(d)
    print("original:", f)
    print("dump:", d)
    print("reconstructed:", o)

It gives the following output (Python 3.7.2 and jsons==0.8.0):

original: Foo(bar=BarA(a=5))
dump: {'bar': {'a': 5}, '-meta': {'classes': {'/bar': '__main__.BarA', '/': '__main__.Foo'}, 'dump_time': '2019-03-20T12:55:38Z'}}
reconstructed: Foo(bar=BarBase())

It appears that the serialization is done properly, but that the re-construction does not take the classes into account?

Datetime offset overflow

It is possible to get following anomaly:

utc = datetime.datetime.now(tz=datetime.timezone.utc)
local = datetime.datetime.now()
print(jsons.dump(utc))    # >>> 2019-02-17T20:14:32Z
print(jsons.dump(local))  # >>> 2019-02-18T00:14:32-20:00

I set my timezone to +04:00, jsons interprets it as -20:00. This suggests I am a day in future.

It is caused by the way you calculate datetime offset

hrs_delta = datetime.now().hour - gmtime().tm_hour

Serializer of datetime with UTC

Hi~, I using udatetime and jsons, the udatetime return UTC timezone is +00:00

udatetime.utcnow()
datetime.datetime(2019, 4, 12, 8, 31, 12, 471970, tzinfo=+00:00)

So, is allow to add +00:00 to _datetime_offset_str in _datetime_impl.py ?

if tzone.tzname(None) not in ('UTC', 'UTC+00:00'):

edit to

if tzone.tzname(None) not in ('UTC', 'UTC+00:00', '+00:00'):

Declaring serializer mapping for a single dump

Is it possible declaring a serializer for a single dump command, instead of declaring it globally?

Currently:

jsons.set_serializer(my_datetime_serializer, datetime)
jsons.dump(foo)

Suggested:

jsons.dump(foo, serializers={datetime: my_datetime_serializer})

Motivation:
I want to have only a particular dump pass datetimes along without string parsing, since the output should be populated in a MongoDB. However, I do other dump's in the script that should use regular behaviour.

jsons.set_serializer(my_datetime_serializer, datetime)
jsons.dump(foo)
jsons.dump(bar) # this should not have my_datetime_serializer

Obviously I could set serializer before and after, but this appears hacky-hacky, and assumes that I know the previous serializer:

jsons.set_serializer(my_datetime_serializer, datetime)
jsons.dump(foo)
jsons.set_serializer(custom_datetime_serializer, datetime)
jsons.dump(bar) # this should not have my_datetime_serializer

Can't deserilize NamedTuple with values evaluating to False

Consider example:

class A(NamedTuple):
   a: int
   b: str

>>> jsons.dumps(A(0, ""))
'[0, ""]'
>>> jsons.loads('[0, ""]', A)
jsons.exceptions.UnfulfilledArgumentError: No value present in [0, ''] for argument "a"
>>> jsons.loads('[1, ""]', A)
jsons.exceptions.UnfulfilledArgumentError: No value present in [1, ''] for argument "b"

Optional attributes not working with .dumps()

In v1.1.0 I am having trouble serializing an object with optional attributes.

import jsons

from dataclasses import dataclass
from typing import List, Optional

@dataclass
class Color:
    name: str
    hex: Optional[str]

@dataclass
class Swatch:
    name: str
    colors: List[Color]

blue = Color('blue', '#0000ff')
green = Color('green', '#00ff00')
swatch = Swatch('swatch 1', [blue, green])

print(swatch)
print(jsons.dumps(swatch))

Result

Swatch(name='swatch 1', colors=[Color(name='blue', hex='hex'), Color(name='green', hex='hex')])
/site-packages/jsons/_common_impl.py:42: UserWarning: Failed to dump attribute "hex" of object of type "__main__.Color". Reason: 'NoneType' object is not callable. Ignoring the attribute. You can suppress warnings like this using jsons.suppress_warnings().
  warnings.warn(msg_, *args, **kwargs)
{"colors": [{}, {}], "name": "swatch 1"}

Result when hex is not optional

Swatch(name='swatch 1', colors=[Color(name='blue', hex='#0000ff'), Color(name='green', hex='#00ff00')])
{"colors": [{"hex": "#0000ff", "name": "blue"}, {"hex": "#00ff00", "name": "green"}], "name": "swatch 1"}

[bug] or [not a bug] @property serialization goes deep into recursion

I don't really know, whether is a bug or it's ok.
Anyway, adding some decorator to not to serialize field_name would be great, I think.

Dict deserializer error

Hi!

>>> jsons.load({'1': 2}, Dict[str, int])
Traceback (most recent call last):
  File "/home/glen/prywatne/traktpy/venv/lib/python3.7/site-packages/jsons/_main_impl.py", line 132, in load
    return deserializer(json_obj, cls, **kwargs_)
TypeError: 'NoneType' object is not callable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/glen/prywatne/traktpy/venv/lib/python3.7/site-packages/jsons/_main_impl.py", line 136, in load
    raise DeserializationError(str(err), json_obj, cls)
jsons.exceptions.DeserializationError: 'NoneType' object is not callable
>>> 

Self-explanatory :)
v0.8.5