Giter Site home page Giter Site logo

yukihiko-shinoda / yaml-dataclass-config Goto Github PK

View Code? Open in Web Editor NEW
35.0 2.0 5.0 97 KB

This project helps you to import config file writen by YAML to Python dataclass.

License: MIT License

Python 100.00%
python yaml dataclass dataclasses type-safe configuration-file

yaml-dataclass-config's Introduction

YAML Data Class Config

Test Test Coverage Maintainability Code Climate technical debt Updates PyPI - Python Version PyPI - Downloads Twitter URL

This project helps you to import config file writen by YAML to Python Data Classes.

Advantage

  1. Type safe import from YAML to Data Classes
  2. Global access and easy unit testing

1. Type safe import from YAML to Data classes

When using pyyaml to import YAML, values be dict and list objects. Using dict or list object will cause such confuses:

  • Reference non exist properties for unexpected instance type
  • Typo of index or key name

To prevent these confuse, one of good way is to use object as model, and python has a good module Data Classes for this purpose.

2. Global access and easy unit testing

You will want to refer config as global because it's troublesome to pass config value as argument over and over like a bucket brigade.

However, when unit testing, if YAML file was loaded automatically on importing global definition, you will face problem that you can't replace config YAML file with the one for unit testing. YAML Data Class Config can divide timings between definition global instance and loading YAML file so you can replace YAML file for unit testing.

Quickstart

1. Install

pip install yamldataclassconfig

2. Prepare config YAML file

Put config.yml YAML Data class Config loads config.yml on Python execution directory by default.

property_a: 1
property_b: '2'
part_config:
  property_c: '2019-06-25 13:33:30'

3. Create config class

Anywhere is OK, for example, I prefer to place on myproduct/config.py

from dataclasses import dataclass, field
from datetime import datetime
from dataclasses_json import DataClassJsonMixin
from marshmallow import fields
from yamldataclassconfig.config import YamlDataClassConfig


@dataclass
class PartConfig(DataClassJsonMixin):
    property_c: datetime = field(metadata={'dataclasses_json': {
        'encoder': datetime.isoformat,
        'decoder': datetime.fromisoformat,
        'mm_field': fields.DateTime(format='iso')
    }})


@dataclass
class Config(YamlDataClassConfig):
    property_a: int = None
    property_b: str = None
    part_config: PartConfig = field(
        default=None,
        metadata={'dataclasses_json': {'mm_field': PartConfig}}
    )

4. Define as global

Also, anywhere is OK, for example, I prefer to place on myproduct/__init__.py

from myproduct.config import Config

CONFIG: Config = Config()

5. Call load before reference config value

from myproduct import CONFIG


def main():
    CONFIG.load()
    print(CONFIG.property_a)
    print(CONFIG.property_b)
    print(CONFIG.part_config.property_c)


if __name__ == '__main__':
    main()

How do I...

Fix path to yaml file independent on the Python execution directory?

override FILE_PATH property.

Ex:

from dataclasses import dataclass
from pathlib import Path

from yamldataclassconfig import create_file_path_field
from yamldataclassconfig.config import YamlDataClassConfig


@dataclass
class Config(YamlDataClassConfig):
    some_property: str = None
    # ...

    FILE_PATH: Path = create_file_path_field(Path(__file__).parent.parent / 'config.yml')

Switch target YAML config file to the one for unit testing?

When setup on unit testing, you can call Config.load() with argument.

Case when unittest:

from pathlib import Path
import unittest

from yourproduct import CONFIG

class ConfigurableTestCase(unittest.TestCase):
    def setUp(self):
        CONFIG.load(Path('path/to/yaml'))

Case when pytest:

from pathlib import Path
import pytest

from yourproduct import CONFIG

@pytest.fixture
def yaml_config():
    CONFIG.load(Path('path/to/yaml'))
    yield

def test_something(yaml_config):
    """test something"""

Use path to YAML config file as same as production when test?

fixturefilehandler can replace config.yml with tests/config.yml.dist easily. Please call all DeployerFactory.create with YamlConfigFilePathBuilder instance argument to create ConfigDeployer. Then, set target directory which config.yml should be placed into path_target_directory.

Case when unittest:

from pathlib import Path
import unittest
from fixturefilehandler.factories import DeployerFactory
from fixturefilehandler.file_paths import YamlConfigFilePathBuilder

from yourproduct import CONFIG


ConfigDeployer = DeployerFactory.create(YamlConfigFilePathBuilder(path_target_directory=Path(__file__).parent.parent))


class ConfigurableTestCase(unittest.TestCase):
    def setUp(self):
        ConfigDeployer.setup()
        CONFIG.load()

    def doCleanups(self):
        ConfigDeployer.teardown()

Case when pytest:

from pathlib import Path
import pytest
from fixturefilehandler.factories import DeployerFactory
from fixturefilehandler.file_paths import YamlConfigFilePathBuilder

from yourproduct import CONFIG


ConfigDeployer = DeployerFactory.create(YamlConfigFilePathBuilder(path_target_directory=Path(__file__).parent.parent))


@pytest.fixture
def yaml_config():
    ConfigDeployer.setup()
    CONFIG.load()
    yield
    ConfigDeployer.teardown()


def test_something(yaml_config):
    """test something"""

yaml-dataclass-config's People

Contributors

yukihiko-shinoda avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

yaml-dataclass-config's Issues

Array support

Hello, does it support yaml arrays ? If so, could you please add an example to do so in the README.md please ?

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Loading empty YAML file fails due to broken contract with Marshmallow

Example

# pytest-compatible file
import pathlib
from dataclasses import dataclass

from yamldataclassconfig import YamlDataClassConfig


def test_loads_empty_file(tmp_path: pathlib.Path):
    @dataclass
    class Config(YamlDataClassConfig):
        pass

    cfg_path = tmp_path / 'config.yml'
    cfg_path.touch()

    Config().load(cfg_path)

Expected result
Test finishes without error.

Actual result
The following error is reported.

../../.cache/pypoetry/virtualenvs/isa-factory-cloud-agent-jX_-wP6c-py3.8/lib/python3.8/site-packages/yamldataclassconfig/config.py:35: in load
    self.__dict__.update(self.__class__.schema().load(dictionary_config).__dict__)
../../.cache/pypoetry/virtualenvs/isa-factory-cloud-agent-jX_-wP6c-py3.8/lib/python3.8/site-packages/marshmallow/schema.py:722: in load
    return self._do_load(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <ConfigSchema(many=False)>, data = None

    def _do_load(
        self,
        data: (
            typing.Mapping[str, typing.Any]
            | typing.Iterable[typing.Mapping[str, typing.Any]]
        ),
        *,
        many: bool | None = None,
        partial: bool | types.StrSequenceOrSet | None = None,
        unknown: str | None = None,
        postprocess: bool = True,
    ):
        """Deserialize `data`, returning the deserialized result.
        This method is private API.
    
        :param data: The data to deserialize.
        :param many: Whether to deserialize `data` as a collection. If `None`, the
            value for `self.many` is used.
        :param partial: Whether to validate required fields. If its
            value is an iterable, only fields listed in that iterable will be
            ignored will be allowed missing. If `True`, all fields will be allowed missing.
            If `None`, the value for `self.partial` is used.
        :param unknown: Whether to exclude, include, or raise an error for unknown
            fields in the data. Use `EXCLUDE`, `INCLUDE` or `RAISE`.
            If `None`, the value for `self.unknown` is used.
        :param postprocess: Whether to run post_load methods..
        :return: Deserialized data
        """
        error_store = ErrorStore()
        errors = {}  # type: dict[str, list[str]]
        many = self.many if many is None else bool(many)
        unknown = (
            self.unknown
            if unknown is None
            else validate_unknown_parameter_value(unknown)
        )
        if partial is None:
            partial = self.partial
        # Run preprocessors
        if self._has_processors(PRE_LOAD):
            try:
                processed_data = self._invoke_load_processors(
                    PRE_LOAD, data, many=many, original_data=data, partial=partial
                )
            except ValidationError as err:
                errors = err.normalized_messages()
                result = None  # type: list | dict | None
        else:
            processed_data = data
        if not errors:
            # Deserialize data
            result = self._deserialize(
                processed_data,
                error_store=error_store,
                many=many,
                partial=partial,
                unknown=unknown,
            )
            # Run field-level validation
            self._invoke_field_validators(
                error_store=error_store, data=result, many=many
            )
            # Run schema-level validation
            if self._has_processors(VALIDATES_SCHEMA):
                field_errors = bool(error_store.errors)
                self._invoke_schema_validators(
                    error_store=error_store,
                    pass_many=True,
                    data=result,
                    original_data=data,
                    many=many,
                    partial=partial,
                    field_errors=field_errors,
                )
                self._invoke_schema_validators(
                    error_store=error_store,
                    pass_many=False,
                    data=result,
                    original_data=data,
                    many=many,
                    partial=partial,
                    field_errors=field_errors,
                )
            errors = error_store.errors
            # Run post processors
            if not errors and postprocess and self._has_processors(POST_LOAD):
                try:
                    result = self._invoke_load_processors(
                        POST_LOAD,
                        result,
                        many=many,
                        original_data=data,
                        partial=partial,
                    )
                except ValidationError as err:
                    errors = err.normalized_messages()
        if errors:
            exc = ValidationError(errors, data=data, valid_data=result)
            self.handle_error(exc, data, many=many, partial=partial)
>           raise exc
E           marshmallow.exceptions.ValidationError: {'_schema': ['Invalid input type.']}

Stack:

dataclasses-json==0.5.7
marshmallow==3.16.0
marshmallow-enum==1.5.1
pyyaml==6.0
yamldataclassconfig==1.5.0

It looks like Marchmallow's Schema.load expects data parameter to always be a dictionary 1. Loading empty file yields None which is passed further and results in what can be seen above.

If you want I may prepare a fix for this, just confirm that Expected result is really expected and it makes sense to correct that, please.

SystemError bad argument to internal function with PyYAML > 5.1

Hello, I have an issue with PyYAML version > 5.1. When importing YamlDataClassConfig.
from yamldataclassconfig.config import YamlDataClassConfig

Traceback (most recent call last): File "C:/Users/nxf46245/hwf_connector/src/hwf_connector/device_descriptor.py", line 10, in <module> from yamldataclassconfig.config import YamlDataClassConfig File "C:\Users\nxf46245\hwf_connector\venv\lib\site-packages\yamldataclassconfig\__init__.py", line 4, in <module> from yamldataclassconfig.config import * # noqa File "C:\Users\nxf46245\hwf_connector\venv\lib\site-packages\yamldataclassconfig\config.py", line 7, in <module> import yaml File "C:\Users\nxf46245\hwf_connector\venv\lib\site-packages\yaml\__init__.py", line 13, in <module> from .cyaml import * File "C:\Users\nxf46245\hwf_connector\venv\lib\site-packages\yaml\cyaml.py", line 7, in <module> from _yaml import CParser, CEmitter SystemError: c:\_work\14\s\objects\codeobject.c:121: bad argument to internal function

It works fine with PyYAML versions lower <= 5.1. I have installed the package from the pip repository.

Environment:
Windows 10 1909 x64
Python 3.8.0a4 x64
yamldataclassconfig 1.5.0

Cannot get List[Union[str, CustomClass]] to work

I'm trying to write a single class which can read the following two files:

#test-logentry.yml

Log:
  - Time: '1 s'
    Severity: 'Info'
    Message: 'this is a message'
  - Time: '1.5 s'
    Severity: 'Warning'
    Message: 'this is a warning'
#test-str.yml

Log:
  - 'this is a message'
  - 'this is a warning'

I'm using the following script:

from typing import List, Union
from pathlib import Path
from dataclasses import dataclass
from yamldataclassconfig.config import YamlDataClassConfig


@dataclass
class Logentry(YamlDataClassConfig):
    """
    """
    Time: str = None
    Severity: str = None
    Message: str = None


@dataclass
class Testcase(YamlDataClassConfig):
    """
    """
    Log: List[Union[str, Logentry]] = None
    #Log: List[Logentry] = None
    #Log: List[str] = None


_cpath = Path('test-.yml')
_osvr = Testcase()
_osvr.load(_cpath)
_osvr.FILE_PATH = _cpath
print(_osvr.FILE_PATH)
print('Log:', _osvr.Log)

Reading test-str.yml works as expected:

# python test.py 
test-str.yml
Log: ['this is a message', 'this is a warning']

Reading test-logentry.yml produces the following warnings:

# python test.py 
C:/msys64/mingw64/lib/python3.8/site-packages/dataclasses_json/mm.py:108: UserWarning: The type "dict" (value: "{'Time': '1 s', 'Severity': 'Info', 'Message': 'this is a message'}") is not in the list of possible types of typing.Union (dataclass: Testcase, field: Log). Value cannot be deserialized properly.
  warnings.warn(
C:/msys64/mingw64/lib/python3.8/site-packages/dataclasses_json/mm.py:108: UserWarning: The type "dict" (value: "{'Time': '1.5 s', 'Severity': 'Warning', 'Message': 'this is a warning'}") is not in the list of possible 
types of typing.Union (dataclass: Testcase, field: Log). Value cannot be deserialized properly.
  warnings.warn(
test-logentry.yml
Log: [{'Time': '1 s', 'Severity': 'Info', 'Message': 'this is a message'}, {'Time': '1.5 s', 'Severity': 'Warning', 'Message': 'this is a warning'}]

However, if Log: List[Logentry] = None is used, instead of the Union, reading test-logentry.yml does not produce any warning:

# python test.py 
test-logentry.yml
Log: [Logentry(FILE_PATH=WindowsPath('T:/vunit/osvb/OSVR/testunion/config.yml'), Time='1 s', Severity='Info', Message='this is a message'), Logentry(FILE_PATH=WindowsPath('T:/vunit/osvb/OSVR/testunion/config.yml'), Time='1.5 s', Severity='Warning', Message='this is a warning')]

I would appreciate any hint for solving this issue.

I'm running the tests on MSYS2 (MINGW64) using Python 3.8.8 and yamldataclassconfig 1.5.0.

Add support to attributes without default values

Problem

yaml-data-class-config doesn't currently support attributes without default values.

Cause

The attribute FILE_PATH in config.py that does not have a default value causes the error.

Possible solution

Remove the FILE_PATH attribute or add it after the config.yml is loaded so that it doesn't cause problems with mypy.

Example

config.yml:

property_a: 1
property_b: '2'

example.py:

from dataclasses import dataclass, field
from yamldataclassconfig.config import YamlDataClassConfig


@dataclass
class Config(YamlDataClassConfig):
    property_a: int
    property_b: str

Running mypy example.py throws the following errors:

problem.py:7: error: Attributes without a default cannot follow attributes with one
problem.py:8: error: Attributes without a default cannot follow attributes with one
Found 2 errors in 1 file (checked 1 source file)

One can go around this problem by adding Union[None, <type>] to every attribute but this is not ideal since one might want to have attributes that throw error if no value is provided.

Environment

  • Ubuntu 20.04.1 LTS x86_64
  • mypy==0.782
  • python==3.8.5

Thanks for the great library by the way!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.