Giter Site home page Giter Site logo

ecological's Introduction

https://travis-ci.org/jmcs/ecological.svg?branch=master Codacy Badge https://api.codacy.com/project/badge/Coverage/1ff45d0e1a5a40b8ad0569e3edb0539d

Ecological

Ecological combines PEP526 and environment variables to make the configuration of 12 factor apps easy.

Getting Started

Ecological automatically gets and converts environment variables according to the configuration class definition.

For example, imagine your application has a configurable (integer) Port and (boolean) Debug flag and a (string) Log Level, that is INFO by default, you could simply declare your configuration as:

class Configuration(ecological.Config):
    port: int
    debug: bool
    log_level: str = "INFO"

And then set the environment variables PORT, DEBUG and LOG_LEVEL. Ecological will automatically set the class properties from the environment variables with the same (but upper cased) name.

By default the values are set at the class definition type and assigned to the class itself (i.e. the class doesn't need to be instantiated). If needed this behavior can be changed (see the Autoloading section).

Tutorial

The tutorial can be used to get to know with the library's basic features interactively.

Typing Support

Ecological also supports some of the types defined in PEP484, for example:

class Configuration(ecological.Config):
    list_of_values: List[str]

Will automatically parse the environment variable value as a list.

Note

Please note that while this will ensure Configuration.list_of_values is a list it will not check that it contains only strings.

Prefixed Configuration

You can also decide to prefix your application configuration, for example, to avoid collisions:

class Configuration(ecological.Config, prefix='myapp'):
    home: str

In this case the home property will be fetched from the MYAPP_HOME environment property.

Nested Configuration

Ecological.Config also supports nested configurations, for example:

class Configuration(ecological.Config):
    integer: int

    class Nested(ecological.Config, prefix='nested'):
        boolean: bool

This way you can group related configuration properties hierarchically.

Advanced

Fine-grained Control

You can control some behavior of how the configuration properties are set.

It can be achieved by providing a ecological.Variable instance as the default value for an attribute or by specifying global options on the class level:

my_source = {"KEY1": "VALUE1"}

class Configuration(ecological.Config, transform=lambda v, wt: v, wanted_type=int, ...):
    my_var1: WantedType = ecological.Variable(transform=lambda v, wt: wt(v), source=my_source, ...)
    my_var2: str
    # ...

All possible options and their meaning can be found in the table below:

Option Class level Variable level Default Description
prefix yes no None A prefix that is uppercased and prepended when a variable name is derived from an attribute name.
variable_name yes yes Derived from attribute name and prefixed with prefix if specified; uppercased.

When specified on the variable level it states the exact name of the source variable that will be used.

When specified on the class level it is treated as a function that returns a variable name from the attribute name with the following signature:

def func(attribute_name: str, prefix: Optional[str] = None)

default no yes (no default) Default value for the property if it isn't set.
transform yes yes A source value is casted to the wanted_type In case of non-scalar types (+ scalar bool) the value is Python-parsed first.

A function that converts a value from the source to the value and wanted_type you expect with the following signature:

def func(source_value: str, wanted_type: Union[Type, str])

source yes yes os.environ Dictionary that the value will be loaded from.
wanted_type yes yes str

Desired Python type of the attribute's value.

On the variable level it is specified via a type annotation on the attribute: my_var_1: my_wanted_type.

However it can be also specified on the class level, then it acts as a default when the annotation is not provided:

class MyConfig(ecological.Config, wanted_type=int, ...)

The following rules apply when options are resolved:

  • when options are specified on both levels (variable and class), the variable ones take precedence over class ones,
  • when some options are missing on the variable level, their default values are taken from the class level,
  • it is not necessary to assign an ecological.Variable instance to change the behavior; it can still be changed on the class level (globally).

Autoloading

It is possible to defer/disable autoloading (setting) of variable values by specifying the autoload option on class definition.

On class creation (default)

When no option is provided values are loaded immediately on class creation and assigned to class attributes:

class Configuration(ecological.Config):
    port: int
# Values already read and set at this point.
# assert Configuration.port == <value-of-PORT-env-var>
Never

When this option is chosen, no autoloading happens. In order to set variable values, the Config.load method needs to be called explicitly:

class Configuration(ecological.Config, autoload=ecological.Autoload.NEVER):
    port: int
# Values not set at this point.
# Accessing Configuration.port would throw AttributeError.

Configuration.load()
# Values read and set at this point.
# assert Configuration.port == <value-of-PORT-env-var>
On object instance initialization

If it is preferred to load and store attribute values on the object instance instead of the class itself, the Autoload.OBJECT strategy can be used:

class Configuration(ecological.Config, autoload=ecological.Autoload.OBJECT):
    port: int
# Values not set at this point.

config = Configuration()
# Values read and set at this point on ``config``.
# assert config.port == <value-of-PORT-env-var>
# Accessing ``Configuration.port`` would throw AttributeError.

Caveats and Known Limitations

  • Ecological doesn't support (public) methods in Config classes

ecological's People

Contributors

jmcs avatar marcinzaremba avatar thilp avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

ecological's Issues

Provide examples

The documentation should provide some real example on how AutoConfig works.

Reliance on dataclasses throws errors in pip for Python3.9

Hello. I have used this library a handful of times over the years and it has been helpful to me.

Recently, however, on Python 3.9 I have seen failing builds due to the dependence on dataclasses. After installing ecological, all further pip commands throw errors like this:

Traceback (most recent call last):
  File "/install/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
  File "/install/lib/python3.9/site-packages/pip/_internal/cli/main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/install/lib/python3.9/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/install/lib/python3.9/site-packages/pip/_internal/cli/main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "/install/lib/python3.9/site-packages/pip/_internal/cli/cmdoptions.py", line 23, in <module>
    from pip._internal.cli.parser import ConfigOptionParser
  File "/install/lib/python3.9/site-packages/pip/_internal/cli/parser.py", line 12, in <module>
    from pip._internal.configuration import Configuration, ConfigurationError
  File "/install/lib/python3.9/site-packages/pip/_internal/configuration.py", line 20, in <module>
    from pip._internal.exceptions import (
  File "/install/lib/python3.9/site-packages/pip/_internal/exceptions.py", line 14, in <module>
    from pip._vendor.rich.console import Console, ConsoleOptions, RenderResult
  File "/install/lib/python3.9/site-packages/pip/_vendor/rich/console.py", line 55, in <module>
    from .pretty import Pretty, is_expandable
  File "/install/lib/python3.9/site-packages/pip/_vendor/rich/pretty.py", line 366, in <module>
    class Node:
  File "/install/lib/python3.9/site-packages/dataclasses.py", line 958, in dataclass
    return wrap(_cls)
  File "/install/lib/python3.9/site-packages/dataclasses.py", line 950, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)
  File "/install/lib/python3.9/site-packages/dataclasses.py", line 800, in _process_class
    cls_fields = [_get_field(cls, name, type)
  File "/install/lib/python3.9/site-packages/dataclasses.py", line 800, in <listcomp>
    cls_fields = [_get_field(cls, name, type)
  File "/install/lib/python3.9/site-packages/dataclasses.py", line 659, in _get_field
    if (_is_classvar(a_type, typing)
  File "/install/lib/python3.9/site-packages/dataclasses.py", line 550, in _is_classvar
    return type(a_type) is typing._ClassVar
AttributeError: module 'typing' has no attribute '_ClassVar'

When ecological is removed as a dependency (or copied and built without dataclasses) the error goes away.

My request is to change the pyproject.toml for this project to stop supporting python 3.6 and remove the dependency on dataclasses:

[tool.poetry.dependencies]
python = "^3.7"

There is a precedent for other Python libraries recently dropping support for Python3.6.

Alternatively, I know that with a setup.py, it's possible to specify a dependency like dataclasses only if using an earlier Python version. For instance, in a setup.py, the following would work:

    install_requires=["dataclasses>=0.7;python_version<'3.7'"]

I believe that the pyproject.toml for this project may be modified in the following way to achieve this:

dataclasses = {version = "0.6",  markers = "python_version < '3.7'"}

Failed if missing and/or Lazy Loading?

Hello,

First I want to say thanks for building this library. I've built many versions of this idea for years and I think this is cleaner than any of my attempts.

I have two things I'd like to include, though, and I wonder if you'd be in interested in them as pull requests. Both are potentially opinionated, though, so feel free to tell me if they don't fit here.

For the first, I often have variables I'd like to exist in the environment and if they don't I want to fail immediately. I usually raise exceptions when parsing these environment variables if they're not present. Thus, I was thinking about a required argument to your variable class that raises when the variable is not present or false-y in the environment.

For the second, and this isn't a big deal, but I often like to have variables lazy-loading, so they only get parsed from the environment on first access, instead of as class attributes that get parsed and loaded as soon as the class is imported.

I would be happy to contribute one or both of these as pull requests for review if you think they may have a place in this project.

As it is, I think this is a cool library, so thanks for publishing it and putting it on PyPi.

Allow reading the environment when instantiating the config class

Today, Ecological fills configuration classes, so you access your config via class fields. The configuration class is never instantiated. Your configuration is basically stored in a global variable initialized only once, at the very beginning of your program.

It would be awesome if we could ask Ecological to read the environment not at class creation time, but at instance creation time. I believe this would make it easier to test components relying on Ecological (see below) and also allow to "refresh" configurations during the process' lifetime.

Use Case

Today:

# I define my configuration in a single place, this is great.
class Config(ecological.AutoConfig):
    plugins: Sequence[str] = ()

# I want to read my configuration from multiple places, e.g. env & CLI.
# Whatever the source, the configuration fields are the same, so I reuse Config.
# Combining everything is the job of read_config:
def read_config() -> Type[Config]:
    args = docopt(...)  # for example
    if args.plugins:
        Config.plugins = (*Config.plugins, *args.plugins)
    return Config  # superfluous
# I understand I could just grab "Config" outside, it doesn't matter.

But I'm faced with a challenge when I try to write tests for read_config. I cannot do what comes naturally in Pytest:

def test_read_config(monkeypatch):
    monkeypatch.setenv("X", "Y")
    assert read_config().x == "Y"

because the environment was read "too early" for this test (when the module containing read_config was loaded).

I could avoid that problem by creating Config inside read_config:

def read_config():
    class Config(ecological.AutoConfig):
        ...
    # usual read_config logic here
    return Config

but this completely breaks static typing (which in my case is a huge reason to use Ecological):

def read_config() -> Type[Config]:  # boom: Config is not defined outside of read_config
    ...

# later
def process(conf: Config):  # boom: what is Config?
    level = conf.level  # as your IDE, I no longer have any idea about the fields of Config

How this could look like

Imho something that only minimally changes Ecological's interface would be best. For example:

class Config(ecological.InstanceConfig, prefix=...):  # notice the different superclass
    ...  # nothing else changes
# the environment is not read yet

# later
config = Config()  # the environment is read now
config = Config()  # the environment is read again

I would then have regular Config instances which type is well-known to every other part of my program, and that can be created at will in tests.

pypi / source distribution

@jmcs I'm testing out a dependency-management tool for the nix language / package manager, and it failed when it couldn't find a source distribution for ecological on pypi.

Can you look into uploading one? :)

Complex type hints break AutoConfig

While the following code works:

class Configuration(ecological.AutoConfig):
    integer_list: List
    dictionary: Dict

The following doesn't:

class Configuration(ecological.AutoConfig):
    integer_list: List[int]
    dictionary: Dict[str, str]

Reorganize tests

Currently all tests are functional tests (from user perspective, ecological.Config is subclassed and then end results are checked). However, pure unit tests are lacking despite reported high coverage. It would be nice to improve this situation.

  • Separate current tests as functional ones
  • Write more unit tests (details to come)

Migrate to Poetry

Or equivalent to avoid the issue of having requirements specified in two places like Pipfile and setup.py and any other possible difficulty that comes with using setuptools.

Todo:

  • consolidate setup.py and Pipfile to a corresponding pyproject.toml
  • fix tox deprecations along the way
  • do the needful with .travis.yml
The [tox:travis] section is deprecated in favor of the "python" key of the [travis] section.
Matching undeclared envs is deprecated. Be sure all the envs that Tox should run are declared in the tox config.

Change default value handling

If the environmental variable is not set and a default is set, the default should be returned as is, instead of being parsed by the transform function.

os.environb suppport?

While trying to sort out storing/retrieving binary values in an env var on Heroku, I noticed that ecological only used os.environ and not also os.environb.

I don't know if it's useful to add; just wanted to let you know in case you hadn't considered it. It didn't matter in our case; the Heroku CLI didn't support setting the binary value in the first place, so I fell back on using base64 which worked equivalently for both environ and environb.

Better support for boolean values

When specifying true instead of True as a value for an environment variable typed as bool in Ecological, the program crashes with an obscure error.

Expect behavior:

  • ideal: true (as well as TRUE and other likely variations) is handled identically to True (and similarly for False);
  • acceptable: The error message warns that only valid Python expressions are accepted as values.

I don't know if this is or should be limited to boolean values.

Replay

$ python3 -m venv /tmp/venv
$ source /tmp/venv/bin/activate
(venv) $ python3 -m pip install ecological
...
Successfully installed ecological-1.6.0
# test.py
import ecological

class Config(ecological.AutoConfig, prefix="test"):
    hi: bool = False

print(f"Config.hi = {Config.hi!r}")
(venv) $ python3 test.py  # all is well
Config.hi = False
(venv) $ env TEST_HI=True python3 test.py  # all is well
Config.hi = True
(venv) $ env TEST_HI=true python3 test.py  # "true" instead of "True"
Traceback (most recent call last):
  File "/tmp/venv/lib/python3.7/site-packages/ecological/autoconfig.py", line 135, in get
    value = self.transform(raw_value, wanted_type)
  File "/tmp/venv/lib/python3.7/site-packages/ecological/autoconfig.py", line 90, in cast
    if isinstance(representation, str)
  File "…/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 91, in literal_eval
    return _convert(node_or_string)
  File "…/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 90, in _convert
    return _convert_signed_num(node)
  File "…/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 63, in _convert_signed_num
    return _convert_num(node)
  File "…/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 55, in _convert_num
    raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: <_ast.Name object at 0x7fbf363839e8>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    class Config(ecological.AutoConfig, prefix="test"):
  File "/tmp/venv/lib/python3.7/site-packages/ecological/autoconfig.py", line 196, in __new__
    value = attribute.get(attribute_type)
  File "/tmp/venv/lib/python3.7/site-packages/ecological/autoconfig.py", line 137, in get
    raise ValueError(f"Invalid configuration for '{self.name}': {e}.")
ValueError: Invalid configuration for 'TEST_HI': malformed node or string: <_ast.Name object at 0x7fbf363839e8>.

Support typing.NewType

Thank you for this great library!

In programs that use a lot of simple data types (strings, integers) for different purposes (account IDs, names, domains, addresses, etc.), I like to use NewType to avoid mixing all of these and clarify the documentation. Thus I may have AccountId, Name, Domain, Address, etc. as newtypes.

It would be great if I could use these types in my configuration class, but it looks like Ecological fallbacks to strings when it doesn't know the type:

>>> I = NewType("I", int)
>>> os.environ["I"] = "2"
>>> class Config(ecological.Autoconfig):
...   i: I
>>> Config.i
"2"

This is not a huge inconvenient (and it does the right thing for newtypes of str anyway), but if it's feasible, it would be the cherry on top. If you are open to this feature, I would even be interested in proposing a PR soon, let me know what you think! 🙂

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.