Giter Site home page Giter Site logo

lovasoa / marshmallow_dataclass Goto Github PK

View Code? Open in Web Editor NEW
437.0 14.0 75.0 1.15 MB

Automatic generation of marshmallow schemas from dataclasses.

Home Page: https://lovasoa.github.io/marshmallow_dataclass/html/marshmallow_dataclass.html

License: MIT License

Python 100.00%
orm marshmallow serialization dataclass

marshmallow_dataclass's Introduction

marshmallow-dataclass

Test Workflow Status (master branch) PyPI version marshmallow 3 compatible download stats

Automatic generation of marshmallow schemas from dataclasses.

from dataclasses import dataclass, field
from typing import List, Optional

import marshmallow_dataclass
import marshmallow.validate


@dataclass
class Building:
    # field metadata is used to instantiate the marshmallow field
    height: float = field(metadata={"validate": marshmallow.validate.Range(min=0)})
    name: str = field(default="anonymous")


@dataclass
class City:
    name: Optional[str]
    buildings: List[Building] = field(default_factory=list)


city_schema = marshmallow_dataclass.class_schema(City)()

city = city_schema.load(
    {"name": "Paris", "buildings": [{"name": "Eiffel Tower", "height": 324}]}
)
# => City(name='Paris', buildings=[Building(height=324.0, name='Eiffel Tower')])

city_dict = city_schema.dump(city)
# => {'name': 'Paris', 'buildings': [{'name': 'Eiffel Tower', 'height': 324.0}]}

Why

Using schemas in Python often means having both a class to represent your data and a class to represent its schema, which results in duplicated code that could fall out of sync. As of Python 3.6, types can be defined for class members, which allows libraries to generate schemas automatically.

Therefore, you can document your APIs in a way that allows you to statically check that the code matches the documentation.

Installation

This package is hosted on PyPI.

pip3 install marshmallow-dataclass

You may optionally install the following extras:

pip3 install "marshmallow-dataclass[enum,union]"

marshmallow 2 support

marshmallow-dataclass no longer supports marshmallow 2. Install marshmallow_dataclass<6.0 if you need marshmallow 2 compatibility.

Usage

Use the class_schema function to generate a marshmallow Schema class from a dataclass.

from dataclasses import dataclass
from datetime import date

import marshmallow_dataclass


@dataclass
class Person:
    name: str
    birth: date


PersonSchema = marshmallow_dataclass.class_schema(Person)

The type of your fields must be either basic types supported by marshmallow (such as float, str, bytes, datetime, ...), Union, or other dataclasses.

Union (de)serialization coercion

Typically the Union type; Union[X, Y] means—from a set theory perspective—either X or Y, i.e., an unordered set, howevever the order of the sub-types defines the precedence when attempting to ether deserialize or serialize the value per here.

For example,

from typing import Union

from dataclasses import dataclass


@dataclass
class Person:
    name: str
    age: Union[int, float]


PersonSchema = marshmallow_dataclass.class_schema(Person)
PersonSchema().load({"name": "jane", "age": 50.0})
# => Person(name="jane", age=50)

will first (sucessfully) try to coerce 50.0 to an int. If coercion is not desired the Any type can be used with the caveat that values will not be type checked without additional validation.

Customizing generated fields

To pass arguments to the generated marshmallow fields (e.g., validate, load_only, dump_only, etc.), pass them to the metadata argument of the field function.

Note that starting with version 4, marshmallow will disallow passing arbitrary arguments, so any additional metadata should itself be put in its own metadata dict:

from dataclasses import dataclass, field
import marshmallow_dataclass
import marshmallow.validate


@dataclass
class Person:
    name: str = field(
        metadata=dict(
            load_only=True, metadata=dict(description="The person's first name")
        )
    )
    height: float = field(metadata=dict(validate=marshmallow.validate.Range(min=0)))


PersonSchema = marshmallow_dataclass.class_schema(Person)

@dataclass shortcut

marshmallow_dataclass provides a @dataclass decorator that behaves like the standard library's @dataclasses.dataclass and adds a Schema attribute with the generated marshmallow Schema.

# Use marshmallow_dataclass's @dataclass shortcut
from marshmallow_dataclass import dataclass


@dataclass
class Point:
    x: float
    y: float


Point.Schema().dump(Point(4, 2))
# => {'x': 4, 'y': 2}

Note: Since the .Schema property is added dynamically, it can confuse type checkers. To avoid that, you can declare Schema as a ClassVar.

from typing import ClassVar, Type

from marshmallow_dataclass import dataclass
from marshmallow import Schema


@dataclass
class Point:
    x: float
    y: float
    Schema: ClassVar[Type[Schema]] = Schema

Customizing the base Schema

It is also possible to derive all schemas from your own base Schema class (see marshmallow's documentation about extending Schema). This allows you to implement custom (de)serialization behavior, for instance specifying a custom mapping between your classes and marshmallow fields, or renaming fields on serialization.

Custom mapping between classes and fields

class BaseSchema(marshmallow.Schema):
    TYPE_MAPPING = {CustomType: CustomField, List: CustomListField}


class Sample:
    my_custom: CustomType
    my_custom_list: List[int]


SampleSchema = marshmallow_dataclass.class_schema(Sample, base_schema=BaseSchema)
# SampleSchema now serializes my_custom using the CustomField marshmallow field
# and serializes my_custom_list using the CustomListField marshmallow field

Renaming fields on serialization

import marshmallow
import marshmallow_dataclass


class UppercaseSchema(marshmallow.Schema):
    """A Schema that marshals data with uppercased keys."""

    def on_bind_field(self, field_name, field_obj):
        field_obj.data_key = (field_obj.data_key or field_name).upper()


class Sample:
    my_text: str
    my_int: int


SampleSchema = marshmallow_dataclass.class_schema(Sample, base_schema=UppercaseSchema)

SampleSchema().dump(Sample(my_text="warm words", my_int=1))
# -> {"MY_TEXT": "warm words", "MY_INT": 1}

You can also pass base_schema to marshmallow_dataclass.dataclass.

@marshmallow_dataclass.dataclass(base_schema=UppercaseSchema)
class Sample:
    my_text: str
    my_int: int

See marshmallow's documentation about extending Schema.

Custom NewType declarations

This library exports a NewType function to create types that generate customized marshmallow fields.

Keyword arguments to NewType are passed to the marshmallow field constructor.

import marshmallow.validate
from marshmallow_dataclass import NewType

IPv4 = NewType(
    "IPv4", str, validate=marshmallow.validate.Regexp(r"^([0-9]{1,3}\\.){3}[0-9]{1,3}$")
)

You can also pass a marshmallow field to NewType.

import marshmallow
from marshmallow_dataclass import NewType

Email = NewType("Email", str, field=marshmallow.fields.Email)

For convenience, some custom types are provided:

from marshmallow_dataclass.typing import Email, Url

Note: if you are using mypy, you will notice that mypy throws an error if a variable defined with NewType is used in a type annotation. To resolve this, add the marshmallow_dataclass.mypy plugin to your mypy configuration, e.g.:

[mypy]
plugins = marshmallow_dataclass.mypy
# ...

Meta options

Meta options are set the same way as a marshmallow Schema.

from marshmallow_dataclass import dataclass


@dataclass
class Point:
    x: float
    y: float

    class Meta:
        ordered = True

Documentation

The project documentation is hosted on GitHub Pages: https://lovasoa.github.io/marshmallow_dataclass/

Contributing

To install this project and make changes to it locally, follow the instructions in CONTRIBUTING.md.

marshmallow_dataclass's People

Contributors

anis-campos avatar cedriccabessa avatar dairiki avatar dutekvejin avatar eoconnell avatar epenet avatar evanfwelch avatar huwcbjones avatar john-bodley avatar joshfriend avatar karlb avatar kristian-lindin avatar liberforce avatar lovasoa avatar manmime avatar maroux avatar martinaltmayer avatar martinitus avatar mvanderlee avatar noirbee avatar otonnesen avatar prihoda avatar selimb avatar sisp avatar sloria avatar spapinistarkware avatar thatfatarcus avatar tomskikh avatar wpromatt avatar zoranpavlovic avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

marshmallow_dataclass's Issues

Mypy error: Variable "Email" is not valid as a type mypy(error)

Taking the example from the readme:

import marshmallow
from marshmallow_dataclass import NewType

Email = NewType("Email", str, field=marshmallow.fields.Email)

When using Email in a schema, for instance:

@marshmallow_dataclass.dataclass
class MySchema:
    id: str = field(metadata={"validate": marshmallow.validate.uuid})
    email: Email
    Schema: ClassVar[Type[marshmallow.Schema]] = marshmallow.Schema

I'm getting this error from mypy:

Mypy error: Variable "Email" is not valid as a type mypy(error)

versions:

marshmallow==3.2.1
marshmallow-dataclass==7.0.0

Thanks in advance.

Using mypy with marshmallow_dataclass

When checking a simple test program with mypy

from marshmallow_dataclass import dataclass

@dataclass
class Foo:
    bar: int

Foo(bar=1)

I get the following errors

$ mypy type_ex.py
type_ex.py:1: error: Cannot find module named 'marshmallow_dataclass'
type_ex.py:1: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
type_ex.py:9: error: Unexpected keyword argument "bar" for "Foo"

I can easily silence the first error with --ignore-missing-imports, but probably that is related to why I get the second error. Do I have to do any additional mypy setup do get this to validate?

Use NewType for derived List class

Hi there!

Coming from vanilla marshmallow, I had a List class that was deriving from marshmallow.fields.List and was adding some logic to how it was reading the input data on _deserialize.

Now, I am trying to integrate this with marshmallow_dataclass and hence create a new type. I have created a few other types that were not lists and it worked well, but I can't get it to work for lists. Here is what I have tried so far:

import typing

import marshmallow
from marshmallow_dataclass import dataclass, NewType


class MyList(marshmallow.fields.List):
    """A list field type that properly handles lists in MultiDict"""

    def _deserialize(self, value, attr, data, **kwargs):
        # removed the code, just checking if this is called
        raise Exception("Finally Called")


ListType = NewType("ListType", typing.List, field=MyList)


@dataclass 
class TestDataClass1:
    """
    This is the one I was expecting would work, as I thought that it would pass
    `int` to `MyList` which would come back to something similar to vanilla
    marshmallow i.e. `values = MyList(Int())`
    """
    values: ListType(int)

# fails with 'Not a valid integer.'
TestDataClass1.Schema().load({"values": [1,2,3]})

# Second try!
@dataclass 
class TestDataClass2: 
    values: ListType(typing.List[int])

# list is properly loaded, but that means MyList._deserialize was not called
TestDataClass2.Schema().load({"values": [1,2,3]})

I looked in the doc and example but could not find a reference for this use case, could you help me figure it out?

code in readme fails ?

I m just getting familiar with marshmallow_dataclass and the code on the README fails for me me with :

line 23, in <module>
    {"name": "Eiffel Tower", "height": 324}
TypeError: cannot unpack non-iterable City object

My python version is Python 3.7.2

Is it included in the doctests running on travis somehow ?
I see all builds passing there... or is the code in README not uptodate ?

Thanks !

Rename?

Nice work with this project!

Just a few suggestions for naming:

  • "dataclasses" should be plural, for consistency with the stdlib's dataclasses
  • Use - instead of _ in the package name. It's the de facto standard and IIRC, pip automatically normalizes _ to - anyway.

So I recommend renaming to marshmallow-dataclasses.

Or--if you'd prefer more cleverness--I also like schemaclasses.

from schemaclasses import schemaclass

@schemaclass
class Artist:
    name: str

Dataclasses do not implement __iter__, causing webargs.use_kwargs to fail

webargs will attempt to combine the results of schema.load into the resulting kwargs to be passed into a function, as seen here:

https://github.com/marshmallow-code/webargs/blob/5.5.1/src/webargs/core.py#L446-L448

In order to repair this assumption, while still using dataclasses, I had to implement a mixin for my dataclasses to allow them to appear as iterables for the purposes of kwargs.update(dataclass)

class Iterable:
    def __iter__(self):
        """When extending a dataclass, yields all key value pairs as if it were a dictionary"""
        for attr, value in self.__dict__.items():
            if value is not None:
                yield attr, value

@dataclass
class Building(Iterable):
    # field metadata is used to instantiate the marshmallow field
    height: float = field(metadata={"validate": marshmallow.validate.Range(min=0)})
    name: str = field(default="anonymous")

Should I go make noise over in webargs saying they should accept dataclasses? or is this something that we can safely hack in here, as we're are actively transforming the resulting dictionary INTO a dataclass.

Another acceptable solution would be to add a meta argument to allow a developer to skip the post_load step, and allow load to return a dictionary and NOT a dataclass, which is honestly more optimal IMO.

Union dump not working

Hi,
I have noticed the dumping of a Union type is returning an empty object:

from marshmallow_dataclass import dataclass
from typing import List, Union

@dataclass
class Vector:
    x: float
    y: float

@dataclass
class Point:
    x: float
    z: float


@dataclass
class Geometries:
    elements: List[Union[Point, Vector]]

pts = Geometries.Schema().load({"elements": [
    {"x": 1, "y": 1},
    {"x": 1, "z": 1},
]})
schema=Geometries.Schema()
print(schema.dump(schema.load({"elements": [
    {"x": 1, "y": 1},
    {"x": 1, "z": 1},
]})))

gives

{"elements": [{}, {}]}

but should again return

{"elements": [
    {"x": 1, "y": 1},
    {"x": 1, "z": 1},
]}

Port unittest tests to pytest

This project started using pytest to run tests in #55. However, existing tests were left untouched, and still are written in unittest style (and with unittest assertions).

Allow adding custom types to _native_to_marshmallow

Follow up to #15, this is to suggest exposting _native_to_marshmallow publicly or, better yet, allow registring custom types to the list for more flexibility on the types that could be user-defined and which fields these types should translate to.

No support for types declared with `NewType`

Problem

Using types declares via NewType (https://docs.python.org/3/library/typing.html#newtype) causes marshmallow_dataclass to throw an error on import:

Traceback (most recent call last):
  File "type_ex.py", line 8, in <module>
    @dataclass
  File "/home/karl/tools/venv-services/lib/python3.7/site-packages/marshmallow_dataclass/__init__.py", line 77, in dataclass
    return add_schema(dataclasses.dataclass(clazz))
  File "/home/karl/tools/venv-services/lib/python3.7/site-packages/marshmallow_dataclass/__init__.py", line 93, in add_schema
    clazz.Schema = class_schema(clazz)
  File "/home/karl/tools/venv-services/lib/python3.7/site-packages/marshmallow_dataclass/__init__.py", line 193, in class_schema
    for field in fields
  File "/home/karl/tools/venv-services/lib/python3.7/site-packages/marshmallow_dataclass/__init__.py", line 194, in <dictcomp>
    if field.init
  File "/home/karl/tools/venv-services/lib/python3.7/site-packages/marshmallow_dataclass/__init__.py", line 284, in field_for_schema
    nested = forward_reference or class_schema(typ)
  File "/home/karl/tools/venv-services/lib/python3.7/site-packages/marshmallow_dataclass/__init__.py", line 185, in class_schema
    raise TypeError(f"{clazz.__name__} is not a dataclass and cannot be turned into one.")
TypeError: Barlike is not a dataclass and cannot be turned into one.
karl@t480karl:~/raiden-services (master)$

Example:

from typing import NewType
from marshmallow_dataclass import dataclass

T_Barlike = int
Barlike = NewType('Barlike', T_Barlike)


@dataclass
class Foo:
    bar: Barlike

Expected behaviour:

Either treat the type like its base type or provide a way to register converters to a basic python type.

Please add a CHANGELOG

As far as I can tell this project has no CHANGELOG to document changes in each release. It would be nice to have one so that we can tell what new features or other changes exist in each new release.

Error when using default or default_factory

import dataclasses as dc
from marshmallow_dataclass import add_schema

@add_schema
@dc.dataclass
class User:
    id: str = dc.field(default=None)
Traceback (most recent call last):
  File "C:/t/projects/python/src/marshmallow_test.py", line 32, in <module>
    @dataclass
  File "C:\t\programs\python\python37\lib\site-packages\marshmallow_dataclass\__init__.py", line 77, in dataclass
    return add_schema(dataclasses.dataclass(clazz))
  File "C:\t\programs\python\python37\lib\site-packages\marshmallow_dataclass\__init__.py", line 93, in add_schema
    clazz.Schema = class_schema(clazz)
  File "C:\t\programs\python\python37\lib\site-packages\marshmallow_dataclass\__init__.py", line 193, in class_schema
    for field in fields
  File "C:\t\programs\python\python37\lib\site-packages\marshmallow_dataclass\__init__.py", line 194, in <dictcomp>
    if field.init
  File "C:\t\programs\python\python37\lib\site-packages\marshmallow_dataclass\__init__.py", line 255, in field_for_schema
    return _native_to_marshmallow[typ](**metadata)
  File "C:\t\programs\python\python37\lib\site-packages\marshmallow\fields.py", line 166, in __init__
    raise ValueError("'missing' must not be set for required fields.")
ValueError: 'missing' must not be set for required fields.

marshmallow 3.0.0rc4
marshmallow-dataclass 0.5.0

Error when dataclass member names conflicting with Schema members

Hi,

I'm hitting the issue when my dataclass defines property decorated method that is named context

Here is an example:

from dataclasses import dataclass

from marshmallow_dataclass import class_schema


@dataclass
class MyClazz:
    foo: str

    @property
    def context(self) -> None:
        return None


MyClazzSchema = class_schema(MyClazz)
MyClazzSchema().load({'foo': 'bar'})

and the result:

Traceback (most recent call last):
  File "/scratches/issue.py", line 16, in <module>
    MyClazzSchema().load({'foo': 'bar'})
  File "/scratches/venv/lib/python3.7/site-packages/marshmallow/schema.py", line 388, in __init__
    self.context = context or {}
AttributeError: can't set attribute

And this is happening with any dataclass member that is conflicting with marshmallow.Schema attribute names eg. many, load_only, fields...

When I chased down this issue, I found that marshmallow_dataclass.class_schema copying all public members of the dataclass to the schema.

# Copy all public members of the dataclass to the schema
attributes = {k: v for k, v in inspect.getmembers(clazz) if not k.startswith("_")}

When I change that particular line of code to attributes = {} everything starts to work fine as marshmallow_dataclass.class_schema will create marshmallow.fields.Field from every dataclass declared field and pass it into created schema.

For me the solution would be not to add all public members of the dataclass to the schema but just dataclass declared field. I can create PR with this change but I'm not sure if this can break something, and I'm wondering why copying all public members at the first place?

Thanks

dataclasses.InitVar are not loadable

Version: 7.5.1

Fields defined as dataclasses.InitVar are not loadable. This might be solvable by specifying them as load-only fields when building the schema.

import dataclasses as dc

import marshmallow as mm
import marshmallow_dataclass as mmdc


@mmdc.dataclass
class Dog:
    name: str
    foo: dc.InitVar[str]

    def __post_init__(self, foo):
        print('foo:', foo)
In [14]: Dog.Schema().load({'name': 'Dino', 'foo': 'bar'})
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-14-74f6780b0d6e> in <module>
----> 1 Dog.Schema().load({'name': 'Dino', 'foo': 'bar'})

~/.pyenv/versions/3.7.7/lib/python3.7/site-packages/marshmallow_dataclass/__init__.py in load(self, data, many, **kwargs)
    468                 return [clazz(**loaded) for loaded in all_loaded]
    469             else:
--> 470                 return clazz(**all_loaded)
    471
    472     return BaseSchema

TypeError: __init__() missing 1 required positional argument: 'foo'

marshmallow 'attribute' through field metadata not working

When deserializing, the field becomes missing. You can check the snippet

from dataclasses import field
from marshmallow_dataclass import dataclass # Importing from marshmallow_dataclass instead of dataclasses
import marshmallow.validate
from typing import List, Optional

@dataclass
class Building:
  # The field metadata is used to instantiate the marshmallow field
  height: float = field(metadata={'validate': marshmallow.validate.Range(min=0), 'attribute': 'height2'})
  name: str = field(default="anonymous")


@dataclass
class City:
  name: Optional[str]
  buildings: List[Building] = field(default_factory=lambda: [])

# City.Schema contains a marshmallow schema class
city = City.Schema().load({
    "name": "Paris",
    "buildings": [
        {"name": "Eiffel Tower", "height":324}
    ]
})

print(City.Schema().dump(city))

building = Building(height=200, name="hello")
city = City(name='Paris', buildings=[building])
# print(Building.Schema().dump(building))
print(City.Schema().dump(city))

`unknown` does't take effect when nested

main code:

from dataclasses import dataclass, field
from typing import List, Optional

import marshmallow
import marshmallow_dataclass


@dataclass
class Building:
    # field metadata is used to instantiate the marshmallow field
    height: float = field(metadata={"validate": marshmallow.validate.Range(min=0)})
    name: str = field(default="anonymous")


@dataclass
class City:
    name: Optional[str]
    buildings: List[Building] = field(default_factory=list)


CitySchema = marshmallow_dataclass.class_schema(City)

city = CitySchema().load(
    {"name": "Paris", "aaa": "bbb", "buildings": [{"name": "Eiffel Tower", "height": 324, "ccc": "ddd"}]},
    unknown=marshmallow.EXCLUDE
)

print(city)

If I use {"name": "Paris", "aaa": "bbb", "buildings": [{"name": "Eiffel Tower", "height": 324}]}, unknown works well.

errer:

Traceback (most recent call last):
  File "clos/Untitled-2.py", line 25, in <module>
    unknown=marshmallow.EXCLUDE
  File "/home/hudingyuan/.local/lib/python3.6/site-packages/marshmallow/schema.py", line 714, in load
    data, many=many, partial=partial, unknown=unknown, postprocess=True
  File "/home/hudingyuan/.local/lib/python3.6/site-packages/marshmallow/schema.py", line 892, in _do_load
    raise exc
marshmallow.exceptions.ValidationError: {'buildings': {0: {'ccc': ['Unknown field.']}}}

Support for marshmallow validators

Currently, any @validates_schema decorators on a dataclass do not run when loading with a generated schema from that dataclass. I assume any field-level validations will also not be run (unless I've done something horribly wrong -- we couldn't find anything in the docs about running validators).

from dataclasses import dataclass
import marshmallow as mm
import marshmallow_dataclass as mmdc

@dataclass
class Foo:
  bar: str
  @mm.validates_schema
  def baz(self, data):
    print(data)
    raise mm.ValidationError('Some error', 'bar')

print(mmdc.class_schema(Foo)(strict=True).load({'bar': '1'}))                                                                                                                  

Expected: ValidationError raised

Actually prints:

UnmarshalResult(data=Foo(bar='1'), errors={})

Bug: not allow null in any values

from typing import Any
from typing import Dict

from marshmallow_dataclass import dataclass


@dataclass()
class A:
    data: Dict[str, Any]


A.Schema().load({
    'data': {'inner_data': None}
})

exception with from __future__ annotations

example:

from __future__ import annotations

import marshmallow_dataclass

@marshmallow_dataclass.dataclass
class C:
    s: str

result

AttributeError: 'str' object has no attribute '__dataclass_fields__'

I believe the problem is in _ _ init _ _.py:

    # Base types
    if typ in _native_to_marshmallow:
        return _native_to_marshmallow[typ](**metadata)

since type is 'str' instead of str, the base type isn't found

marshmallow-dataclass==6.0.0c1
python==3.7.2

Unexpected interaction between Union and Optional

Am trying to compose Union and Optional but am seeing several unexpected behaviors. Hopefully they are captured by the below examples. Can you please let me know what I'm missing?

Common setup for observations below:

from dataclasses import field
from typing import Optional, Union

from marshmallow_dataclass import dataclass
from marshmallow.validate import Equal


@dataclass
class C:
    _type: str = field(metadata=dict(validate=Equal('C')), default='C')


@dataclass
class B:
    _type: str = field(metadata=dict(validate=Equal('B')), default='B')


@dataclass
class A:
    x: Optional[Union[B, C]]

This fails:

raw = {'x': {'_type': 'C'}}
a = A.Schema().load(raw)

marshmallow.exceptions.ValidationError: {'x': {'_type': ['Must be equal to B.']}}.

This succeeds:

raw = {'x': {'_type': 'B'}}
a = A.Schema().load(raw)

The following produce identical results for the above usages:

  • x: Optional[Union[B, C]] = None
  • x: Union[B, C, None]
  • x: Union[B, C, None] = None

dict-field of str to another Dataclass

I want to decode this json :

{
    "ids": {
        "id124" : { "a":3 },
        "id254" : { "a":1 }
    }
}

and this json

{
    "ids": {
        "id132" : { "a":5 },
        "id1335":{ "a":2 },
        "id254" : { "a":1 }
    }
}

(so a dictionnary that maps string to a dataclass)

And I want this to fit in the same dataclass :

@dataclass
class DataItem:
    a: int

@dataclass
class Data:
    ids : typing.Dict[str, DataItem]

Unfortunately, this doesn't work, this will return marshmallow.exceptions.ValidationError: {'ids': ['Not a valid mapping type.']}

How do I do this? I managed to have lists, but not dicts.

Defining future import in same file as the dataclass prevents model schema load

Problem description:

If we have a dataclass
And using from __future__ import annotations, model_schema().load() is not working properly

Steps to reproduce

from __future__ import annotations  # removing this line fix the issue

import json
import unittest
from dataclasses import dataclass
from typing import Any, Dict

import marshmallow_dataclass


@dataclass
class Sample:
    name: str

    def as_json_string(self):
        return json.dumps(self, default=lambda o: o.__dict__)

    @classmethod
    def from_dict(cls, data: Dict[str, Any]) -> 'Sample':
        model_schema = marshmallow_dataclass.class_schema(cls)
        return model_schema().load(data)


class TestModel(unittest.TestCase):

    def test_json_to_object_to_json(self):
        test = Sample("Paris")
        test_to_json = json.loads(test.as_json_string())
        print(test_to_json)
        print(test)
        self.assertEqual(test, Sample.from_dict(test_to_json))


if __name__ == '__main__':
    unittest.main()
TypeError: 'str' is not a dataclass and cannot be turned into one.

Workaround
Do not use future

Add Support for Unions

I absolutely love your library, it's made my life so much easier. Thank you!

Can we add support for using marshmallow_union on Union fields?

Similar to how you support marshmallow_enum?

Cannot add marshmallow metadata to list items

Issue

I can't seem to find any way to apply validation to items of a list type object, and upon inspection of the field_for_schema implementation I don't think this is possible, as no metadata is forwarded to the schema creation for the list item type:

marshmallow_dataclass: init.py

239:  def field_for_schema(
240:          typ: type,
241:          default=marshmallow.missing,
242:          metadata: Mapping[str, Any] = None
243:  ) -> marshmallow.fields.Field:
...
303:      if origin in (list, List):
304:          list_elements_type = typing_inspect.get_args(typ, True)[0]
305:          return marshmallow.fields.List(
306:              field_for_schema(list_elements_type),  # <<< No metadata passed
307:              **metadata
308:          )

Current State

Let's say I have a User class and and want to maintain a list of email addresses for each user. Currently, the only way I'd be able to validate the IP addresses is with a callable that iterated over the list and manually invoked Marshmallow's validators:

from typing import List
from marshmallow.validate import Email
from marshmallow_dataclass import dataclass

@dataclass
class User:
    name: str
    emails: List[str] = field(metadata={
        'validate': lambda l: all(Email()(i) for i in l)
    })

...which, admittedly, is pretty simple, but it's more weakly representative of the validator's intent, particularly for tools that post-process the schema (i.e. - the fact that it's a list validator means that tools aren't able to recognize that the validation actually applies to the list item).

Proposal

I'd like to add a mechanism for explicitly passing metadata to the list item schema. I figure using a key in the metadata dict that doesn't conflict with Marshmallow schema kwargs would work just fine for this purpose:

from typing import List
from marshmallow.validate import Email
from marshmallow_dataclass import dataclass

@dataclass
class User:
    name: str
    emails: List[str] = field(metadata={
        'item_metadata': {
            'validate': Email()
        }
    })

If this sounds like something of interest, I'll gladly whip it up and put out a PR.

supporting generic types without type arguments, such as `dict`

Hi, variables is a dict. My dataclass NewReqPayload cannot be dumped.

from dataclasses import dataclass
import marshmallow_dataclass

@dataclass
class NewReqPayload:
    category: str
    type: str
    subtype: str
    short_description: str
    description: str
    requested_by: str
    assigned_to: str
    requested_due_date: str
    variables: dict

NewReqPayloadSchema = marshmallow_dataclass.class_schema(NewReqPayload)

req_payload = NewReqPayload(
        category="System Administration",
        type="Configuration Requests",
        subtype="Tools/Automation",
        short_description="jon short_description",
        description="jon description",
        requested_by="xiaojuwang",
        assigned_to="xiaojuwang",
        requested_due_date="2019-12-20",
        variables={"environment": "development"}
)

NewReqPayloadSchema().dump(req_payload)

This code raises TypeError: dict is not a dataclass and cannot be turned into one..

Assign to __schema__ instead of Schema; load and dump helpers?

It may make sense to assign the generated schema to __schema__ instead of Schema for consistency with dataclasses.dataclass, which only assigns dunder methods.

@dataclass
class Artist:
    name: str

Artist.__schema__  # => Schema class

To improve the DX, you could add dump and load helper methods.

import datetime as dt

from marshmallow_dataclass import dataclass, load

@dataclass
class Album:
    title: str
    release_date: dt.date

album = load(Album, {"title": "Hunky Dory", "release_date": "1971-12-17"})

Schema contructor arguments can be passed as keyword arguments.

albums = load(Album, input_data, many=True)

This could even open the door to conveniences like

albums = load(List[Album], input_data)  # calls Album.__schema__(many=True).load(input_data)

What do you think?

When type hint is not recognized (e.g. `pd.Timestamp`), we recursively try to convert into a dataclass which mutates the constructor throughout the codebase

Reproducible error:

import pandas as pd
import dataclasses

import marshmallow_dataclass

@dataclasses.dataclass
class Something:
    date_i_care_about: pd.Timestamp

print(f"here's a timestamp {pd.Timestamp('2017-01-01')}")

# this mutates the constructor for pd.Timestamp by recursively trying to convert to dataclass
marshmallow_dataclass.class_schema(Something)

print(f"here's a timestamp {pd.Timestamp('2017-01-01')}")

versions:
pandas 0.23.4
marshmallow 2.x
marshmallow_dataclass: 0.6.6

@lovasoa what is the intution behind trying to convert clazz to dataclass here after we fail to get the fields?

https://github.com/lovasoa/marshmallow_dataclass/blob/master/marshmallow_dataclass/__init__.py#L281

I feel that an error (or some kind of strict mode) would be more helpful

Allow modifying `_native_to_marshmallow`

This is a follow-up to #16, which was closed after introducing NewType. As explained in the following, the NewType-solution is not possible in my case, so I'd like to ask you to reconsider adding a way to make _native_to_marshmallow mutable. From the implementation it looks pretty simple to replace _native_to_marshmallow by a dynamic look-up to Marshmallow's Schema.TYPE_MAPPING. Then I could modify the latter and marshmallow_dataclass would pick up the changes.

I would like to "register" a fixed Marshmallow field that should be used whenever a type appears in a dataclass that is converted via marshmallow_dataclass:

@dataclass
class Schema:
    color: Color

should be equivalent to

@dataclass
class Schema:
    color: Color = field(metadata={  ... some lengthy definitions here ... })

Unfortunately, I cannot replace "Color" by a NewType, because it is defined in a lower-level library. I would like to avoid having two very similar types, one from the library, and another one for use in schemas.

"typing.Any is not a dataclass and cannot be turned into one"

Using 6.0.0rc2, my program can not start if I declare a dataclass that uses typing.Any:

from dataclasses import dataclass
from typing import Any
import marshmallow_dataclass

@dataclass
class Request:
    service: str
    method: str
    input: Any

RequestSchema = marshmallow_dataclass.class_schema(Request)

Attempting to start my program gives me the following error:

  File "<snip>/__init__.py", line 211, in class_schema
    raise TypeError(f"{getattr(clazz, '__name__', repr(clazz))} is not a dataclass and cannot be turned into one.")
TypeError: typing.Any is not a dataclass and cannot be turned into one.

default behavior doesn't work for non JSON serializable types (because default is used for both marshmallow's missing and default fields)

Marshmallow 2.x makes a distinction between default and missing parameters in it's Field class: default is used for serialization and missing for deserialization. However, the marshmallow_dataclass field_for_schema method sets the dataclass.field.default to both missing and default (here: https://github.com/lovasoa/marshmallow_dataclass/blob/master/marshmallow_dataclass/__init__.py#L268-L270). The problem is that missing should look more like an input to Field._deserialize and default should look more like an output of Field._deserialize.

Therefore, the following does not work like I would expect it to (and I suspect this is true for all non-JSON serializable types that marshmallow_dataclass supports):

from datetime import date
from dataclasses import dataclass, field
from marshmallow_dataclass import dataclass


@dataclass
class Params:
  date_i_need: date = field(default=date(2018,1,1))
  int_i_need: int = field(default=3)


if __name__ == '__main__':
    blob = {}
    #schema = Params.Schema(strict=True)
    #schema.load(blob)

    from marshmallow.fields import Date

    d = Date(missing=date(2018,1,1))
    d.deserialize({})

I get this error:

Traceback (most recent call last):
  File "date_play.py", line 15, in <module>
    schema.load(blob)
  File "/Users/evanw/SANDBOX/mallow/env/lib/python3.6/site-packages/marshmallow/schema.py", line 597, in load
    result, errors = self._do_load(data, many, partial=partial, postprocess=True)
  File "/Users/evanw/SANDBOX/mallow/env/lib/python3.6/site-packages/marshmallow/schema.py", line 720, in _do_load
    raise exc
marshmallow.exceptions.ValidationError: {'date_i_need': ['Not a valid date.']}

Allowing of init=False fields to appear in the schema

Can you please an argument to the class_schema method to allow appear fields with init=False in the schema?

This option will be very useful when using inheritance and default arguments.

And what do you think about adding to the schema class a suffix of 'Schema'

TypeError: Ellipsis is not a dataclass and cannot be turned into one.

from dataclasses import field
from decimal import Decimal
from typing import Tuple

from marshmallow_dataclass import dataclass


@dataclass(frozen=True)
class Signal:
    fields: Tuple[Decimal, ...] = field(default_factory=list)

This ellipsis means that all values in fields have Decimal type.

Behavior of marshmallow.post_load is dependent on function name

I discovered a discrepancy with how the post_load decorator behaves within a marshmallow dataclass. In Schema._invoke_processors the post loads make_data_class processor runs before or after the post_load defined in the dataclass based on whichever comes first alphabetically. Some example below illustrate this:

import marshmallow
import marshmallow_dataclass

@marshmallow_dataclass.dataclass
class Person:
	name: str

	@marshmallow.post_load
	def a(schema, data, **kwargs):
		data.name = data.name.capitalize()  # works if: data['name'] = data['name'].capitalize()
		return data

Person.Schema().load({'name': 'matt'})
>> AttributeError: 'dict' object has no attribute 'name'
import marshmallow
import marshmallow_dataclass

@marshmallow_dataclass.dataclass
class Person:
	name: str

	@marshmallow.post_load
	def z(schema, data, **kwargs):
		data.name = data.name.capitalize()
		return data

Person.Schema().load({'name': 'matt'})
>> Person(name='Matt')

The post_load in the first example does not create an instance of the dataclass itself before being invoked and feels more so like a pre_load because of this. The validation checks still seem to work properly, but it just feels a little weird to work with. I know the documentation does not include anything about pre/post loading, but I figured I'd document this in case anyone else runs into the same issue.

Example in readme not a valid

This example

import marshmallow
import marshmallow_dataclass


class UppercaseSchema(marshmallow.Schema):
    """A Schema that marshals data with uppercased keys."""

    def on_bind_field(self, field_name, field_obj):
        field_obj.data_key = (field_obj.data_key or field_name).upper()


class Sample:
    my_text: str
    my_int: int


SampleSchema = marshmallow_dataclass.class_schema(Sample, base_schema=UppercaseSchema)

Sample.Schema().dump(Sample(my_text="warm words", my_int=1))

not work with last version of marshmallow_dataclass.

Allow custom BaseSchema

Without marshmallow_dataclass:
I have my own BaseSchema, that (not only) converts all snake_case variable names to their camelCase alternatives when dumping. All my others schemas inherit from that.
I want to get rid of boilerplate schemas for my dataclasses and use this library, but i can't find a way to do so.

My idea:
With marshmallow_dataclass:
It would be great, if
PersonSchema = marshmallow_dataclass.class_schema(Person) had a parameter base_class=MyBaseClass

(for release candidate 3, rc3). correct @post_load handler params in `make_data_class`

from:

def _base_schema(clazz: type) -> Type[marshmallow.Schema]:
    class BaseSchema(marshmallow.Schema):
        @marshmallow.post_load
        def make_data_class(self, data):
            return clazz(**data)

    return BaseSchema

to: (just adds, **kwargs)

def _base_schema(clazz: type) -> Type[marshmallow.Schema]:
    class BaseSchema(marshmallow.Schema):
        @marshmallow.post_load
        def make_data_class(self, data, **kwargs):
            return clazz(**data)

    return BaseSchema

Create dataclasses with parameter

The following is the syntax for data class to use param like,
"init, repr, eq, order, unsafe_hash, frozen"

>>> import dataclasses
>>> 
>>> @dataclasses.dataclass(order=True, frozen=True)
... class Validate:
...     is_valid: bool
... 
>>> 

The below doesn't work today and give an exception.

>>> from marshmallow_dataclass import dataclass
>>> 
>>> @dataclass(order=True, frozen=True)
... class Validate:
...     is_valid: bool
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: dataclass() got an unexpected keyword argument 'order'

However as a workaround for now the following works instead,

>>> from marshmallow_dataclass import add_schema
>>> @add_schema
... @dataclasses.dataclass(order=True, frozen=True)
... class Validate:
...     is_valid: bool
... 
>>> 

IMO, would be good to have optional parameter present in the "marshmallow_dataclass.dataclass" decorator for future use case.

`None` not being assigned to `Union`

packages
marshmallow==3.6.1
marshmallow-dataclass==7.6.0
marshmallow-enum==1.5.1
marshmallow-union==0.1.15

from typing import List, Union

from marshmallow_dataclass import dataclass

import dataclasses

noned = lambda: dataclasses.field(
    metadata=dict(
        required=False,
        missing=None,
        default=None,
    )
)

@dataclass
class Inner:
    inner_var: int

@dataclass
class Test:
    req: int = None
    var: Union[int, Inner, List[Inner]] = noned()

I get the following error trying to use the class

/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/dataclasses.py:1019: in dataclass
    return wrap(cls)
/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/dataclasses.py:1011: in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)
/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/dataclasses.py:925: in _process_class
    _init_fn(flds,
/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/dataclasses.py:502: in _init_fn
    raise TypeError(f'non-default argument {f.name!r} '
E   TypeError: non-default argument 'var' follows default argument

It seems like default and missing None are not being assigned properly?

`base_schema` of nested dataclasses is ingored

The base_schema of the nested field is ignored.

Example:

from marshmallow import Schema, post_dump
from marshmallow_dataclass import dataclass


class DoubleSchema(Schema):
    @post_dump
    def double(self, data, **kwargs):
        return dict(double=data['number'] * 2)


@dataclass(base_schema=DoubleSchema)
class A:
    number: int


@dataclass
class B:
    a: A


print('is A.Schema correct?', issubclass(A.Schema, DoubleSchema))
print('is B.a.Schema correct?', isinstance(B.Schema._declared_fields['a'].schema, DoubleSchema))
print()

print('expected:')
print("    {'a': {'double': 2}}")
print('actual:')
print('   ', B.Schema().dump(B(A(1))))

stdout:

is A.Schema correct? True
is B.a.Schema correct? False

expected:
    {'a': {'double': 2}}
actual:
    {'a': {'number': 1}}

How to fix

--- marshmallow_dataclass/__init__.py   2019-11-25 14:59:36.146392038 +0200
+++ marshmallow_dataclass/__init__.py.fixed     2019-11-25 15:01:01.947898497 +0200
@@ -422,9 +422,12 @@ def field_for_schema(
 
+    # Nested marshmellow dataclass
+    nested_schema = getattr(typ, 'Schema', None)
+
     # Nested dataclasses
     forward_reference = getattr(typ, "__forward_arg__", None)
-    nested = forward_reference or class_schema(typ, base_schema=base_schema)
+    nested = nested_schema or forward_reference or class_schema(typ, base_schema=base_schema)
     return marshmallow.fields.Nested(nested, **metadata)

Support for attrs

I like to use the attrs module. Attrs functionality is a superset of dataclasses, though a few of the names are different (e.g. factory vs default_factory). How would you feel about having support for attrs in addition to dataclasses?

Dict keys / values are returned as they are regardless of the field

Hello,

Upon investigating an issue of a dataclass which has a Dict field of custom types, i found out that marshmallow (in the version pinned by marshmallow_dataclasses, 2.19.2) returns the key/value pairs as they are regardless of what Field we set for them.

An example:

from dataclasses import dataclass
import marshmallow
from marshmallow_dataclass import class_schema, _native_to_marshmallow

from typing import NewType, Dict, Any

T_Signature = bytes
Signature = NewType("Signature", T_Signature)

EMPTY_SIGNATURE = Signature(bytes(65))

class BytesField(marshmallow.fields.Field):
    """ Used for `bytes` in the dataclass, serialize to hex encoding"""

    def _serialize(self, value: bytes, attr: Any, obj: Any) -> str:
        return to_hex(value)

    def _deserialize(self, value: str, attr: Any, data: Any) -> bytes:
        return to_bytes(hexstr=value)

_native_to_marshmallow[Signature] = BytesField


@dataclass
class A:
    signatures: Dict[Signature, Signature]


a = A(signatures={
    b'123': b'456',
})

schema = class_schema(A)
print(schema().dump(a))

Here, you can see i am doing the following:

  1. Declaring a custom type Signature of type bytes
  2. Adding the type to marshmallow_dataclass._native_to_marshmallow dictionary with a custom Field
  3. The BytesField in this case would convert the bytes to a hex string and vice versa.
  4. Dumping a sample object.

The sample object is dumped as it was provided

MarshalResult(data={'signatures': {b'123': b'456'}}, errors={})

to_hex and to_bytes aren't even declared in this file so that should have error'ed but it doesn't because marshmallow isn't using the field but rather returning the values as they are.

Here's the code in marshmallow:
https://github.com/marshmallow-code/marshmallow/blob/ac9ff95432f929fb0fa2394c29b94d1f804a1504/src/marshmallow/fields.py#L281-L298

And here's the code for Dict:
https://github.com/marshmallow-code/marshmallow/blob/ac9ff95432f929fb0fa2394c29b94d1f804a1504/src/marshmallow/fields.py#L1083-L1102

As you can see, it doesn't use the fields passed down to them in marshmallow_dataclass dict code:

elif origin in (dict, Dict):
key_type, value_type = typing_inspect.get_args(typ, True)
return marshmallow.fields.Dict(
keys=field_for_schema(key_type),
values=field_for_schema(value_type),
**metadata
)

My suggestion here, is to provide your own "Dict" field implementation which is able to use the provided fields to serialize/deserialize keys according to their pre-defined field types.

What do you think?

Conda package? marshmallow 3.0?

Hi, and thanks for a nice package :)

A few quick questions - would it be worth pushing this to conda-forge to make it more discoverable? Would be nice to have it available through conda and not just pip.

Also, given that marshmallow 3.0 is bound to be out soon (the stable RCs are already out), would this package be compatible with it?

Thanks!

Installation command not working

Hi author, I have tried

pip3 install marshmallow-dataclass[enum,union]

but t seems not working, I think that we should change the command or fix the documentation so that user will understand it more.
Anyway, thanks for inventing this library, it helps me alot

non-required field causes TypeError on `load()` if missing from json

marshmallow-dataclass==0.6.6

A non-required field defined with marshmallow_field will cause a TypeError to be thrown in the make_dataclass() post-load processor if not present in the JSON object passed to load()

Minimal reproduction:

>>> from marshmallow_dataclass import dataclass 
>>> from typing import Optional 
>>> from marshmallow.fields import String
>>> from dataclasses import field 

>>> @dataclass 
>>> class Document(): 
>>>     content: Optional[str] = field(metadata=dict(marshmallow_field=String(), required=False)) 

>>> Document.Schema().load({})  # Should not raise an error  
TypeError: __init__() missing 1 required positional argument: 'content'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.