jmcs/ecological

Ecological combines PEP526 and environment variables to make the configuration of 12 factor apps easy.

ecological's Issues

Support typing.NewType

Thank you for this great library!

In programs that use a lot of simple data types (strings, integers) for different purposes (account IDs, names, domains, addresses, etc.), I like to use NewType to avoid mixing all of these and clarify the documentation. Thus I may have AccountId, Name, Domain, Address, etc. as newtypes.

It would be great if I could use these types in my configuration class, but it looks like Ecological fallbacks to strings when it doesn't know the type:

>>> I = NewType("I", int)
>>> os.environ["I"] = "2"
>>> class Config(ecological.Autoconfig):
...   i: I
>>> Config.i
"2"

This is not a huge inconvenient (and it does the right thing for newtypes of str anyway), but if it's feasible, it would be the cherry on top. If you are open to this feature, I would even be interested in proposing a PR soon, let me know what you think! 🙂

Migrate to Poetry

Or equivalent to avoid the issue of having requirements specified in two places like Pipfile and setup.py and any other possible difficulty that comes with using setuptools.

Todo:

• consolidate setup.py and Pipfile to a corresponding pyproject.toml
• fix tox deprecations along the way
• do the needful with .travis.yml

The [tox:travis] section is deprecated in favor of the "python" key of the [travis] section.
Matching undeclared envs is deprecated. Be sure all the envs that Tox should run are declared in the tox config.

Change default value handling

If the environmental variable is not set and a default is set, the default should be returned as is, instead of being parsed by the transform function.

Document typing to types relationship

Document how Ecological maps typing "types" to real types.

Complex type hints break AutoConfig

While the following code works:

class Configuration(ecological.AutoConfig):
    integer_list: List
    dictionary: Dict

The following doesn't:

class Configuration(ecological.AutoConfig):
    integer_list: List[int]
    dictionary: Dict[str, str]

Test package both on Python 3.6 and 3.7 on CI

Make python 3.6.1 types optional

Some typing types like Counter where only added on python 3.6.1. Ecological should work even if they are not available.

os.environb suppport?

While trying to sort out storing/retrieving binary values in an env var on Heroku, I noticed that ecological only used os.environ and not also os.environb.

I don't know if it's useful to add; just wanted to let you know in case you hadn't considered it. It didn't matter in our case; the Heroku CLI didn't support setting the binary value in the first place, so I fell back on using base64 which worked equivalently for both environ and environb.

Reorganize tests

Currently all tests are functional tests (from user perspective, ecological.Config is subclassed and then end results are checked). However, pure unit tests are lacking despite reported high coverage. It would be nice to improve this situation.

• Separate current tests as functional ones
• Write more unit tests (details to come)

Failed if missing and/or Lazy Loading?

Hello,

First I want to say thanks for building this library. I've built many versions of this idea for years and I think this is cleaner than any of my attempts.

I have two things I'd like to include, though, and I wonder if you'd be in interested in them as pull requests. Both are potentially opinionated, though, so feel free to tell me if they don't fit here.

For the first, I often have variables I'd like to exist in the environment and if they don't I want to fail immediately. I usually raise exceptions when parsing these environment variables if they're not present. Thus, I was thinking about a required argument to your variable class that raises when the variable is not present or false-y in the environment.

For the second, and this isn't a big deal, but I often like to have variables lazy-loading, so they only get parsed from the environment on first access, instead of as class attributes that get parsed and loaded as soon as the class is imported.

I would be happy to contribute one or both of these as pull requests for review if you think they may have a place in this project.

As it is, I think this is a cool library, so thanks for publishing it and putting it on PyPi.

Provide examples

The documentation should provide some real example on how AutoConfig works.

Better support for boolean values

When specifying true instead of True as a value for an environment variable typed as bool in Ecological, the program crashes with an obscure error.

Expect behavior:

• ideal: true (as well as TRUE and other likely variations) is handled identically to True (and similarly for False);
• acceptable: The error message warns that only valid Python expressions are accepted as values.

I don't know if this is or should be limited to boolean values.

Replay

$ python3 -m venv /tmp/venv
$ source /tmp/venv/bin/activate
(venv) $ python3 -m pip install ecological
...
Successfully installed ecological-1.6.0

# test.py
import ecological

class Config(ecological.AutoConfig, prefix="test"):
    hi: bool = False

print(f"Config.hi = {Config.hi!r}")

(venv) $ python3 test.py  # all is well
Config.hi = False
(venv) $ env TEST_HI=True python3 test.py  # all is well
Config.hi = True
(venv) $ env TEST_HI=true python3 test.py  # "true" instead of "True"
Traceback (most recent call last):
  File "/tmp/venv/lib/python3.7/site-packages/ecological/autoconfig.py", line 135, in get
    value = self.transform(raw_value, wanted_type)
  File "/tmp/venv/lib/python3.7/site-packages/ecological/autoconfig.py", line 90, in cast
    if isinstance(representation, str)
  File "…/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 91, in literal_eval
    return _convert(node_or_string)
  File "…/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 90, in _convert
    return _convert_signed_num(node)
  File "…/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 63, in _convert_signed_num
    return _convert_num(node)
  File "…/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 55, in _convert_num
    raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: <_ast.Name object at 0x7fbf363839e8>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    class Config(ecological.AutoConfig, prefix="test"):
  File "/tmp/venv/lib/python3.7/site-packages/ecological/autoconfig.py", line 196, in __new__
    value = attribute.get(attribute_type)
  File "/tmp/venv/lib/python3.7/site-packages/ecological/autoconfig.py", line 137, in get
    raise ValueError(f"Invalid configuration for '{self.name}': {e}.")
ValueError: Invalid configuration for 'TEST_HI': malformed node or string: <_ast.Name object at 0x7fbf363839e8>.

pypi / source distribution

@jmcs I'm testing out a dependency-management tool for the nix language / package manager, and it failed when it couldn't find a source distribution for ecological on pypi.

Can you look into uploading one? :)

Allow reading the environment when instantiating the config class

Today, Ecological fills configuration classes, so you access your config via class fields. The configuration class is never instantiated. Your configuration is basically stored in a global variable initialized only once, at the very beginning of your program.

It would be awesome if we could ask Ecological to read the environment not at class creation time, but at instance creation time. I believe this would make it easier to test components relying on Ecological (see below) and also allow to "refresh" configurations during the process' lifetime.

Use Case

Today:

# I define my configuration in a single place, this is great.
class Config(ecological.AutoConfig):
    plugins: Sequence[str] = ()

# I want to read my configuration from multiple places, e.g. env & CLI.
# Whatever the source, the configuration fields are the same, so I reuse Config.
# Combining everything is the job of read_config:
def read_config() -> Type[Config]:
    args = docopt(...)  # for example
    if args.plugins:
        Config.plugins = (*Config.plugins, *args.plugins)
    return Config  # superfluous
# I understand I could just grab "Config" outside, it doesn't matter.

But I'm faced with a challenge when I try to write tests for read_config. I cannot do what comes naturally in Pytest:

def test_read_config(monkeypatch):
    monkeypatch.setenv("X", "Y")
    assert read_config().x == "Y"

because the environment was read "too early" for this test (when the module containing read_config was loaded).

I could avoid that problem by creating Config inside read_config:

def read_config():
    class Config(ecological.AutoConfig):
        ...
    # usual read_config logic here
    return Config

but this completely breaks static typing (which in my case is a huge reason to use Ecological):

def read_config() -> Type[Config]:  # boom: Config is not defined outside of read_config
    ...

# later
def process(conf: Config):  # boom: what is Config?
    level = conf.level  # as your IDE, I no longer have any idea about the fields of Config

How this could look like

Imho something that only minimally changes Ecological's interface would be best. For example:

class Config(ecological.InstanceConfig, prefix=...):  # notice the different superclass
    ...  # nothing else changes
# the environment is not read yet

# later
config = Config()  # the environment is read now
config = Config()  # the environment is read again

I would then have regular Config instances which type is well-known to every other part of my program, and that can be created at will in tests.

Reliance on dataclasses throws errors in pip for Python3.9

Hello. I have used this library a handful of times over the years and it has been helpful to me.

Recently, however, on Python 3.9 I have seen failing builds due to the dependence on dataclasses. After installing ecological, all further pip commands throw errors like this:

Traceback (most recent call last):
  File "/install/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
  File "/install/lib/python3.9/site-packages/pip/_internal/cli/main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/install/lib/python3.9/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/install/lib/python3.9/site-packages/pip/_internal/cli/main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "/install/lib/python3.9/site-packages/pip/_internal/cli/cmdoptions.py", line 23, in <module>
    from pip._internal.cli.parser import ConfigOptionParser
  File "/install/lib/python3.9/site-packages/pip/_internal/cli/parser.py", line 12, in <module>
    from pip._internal.configuration import Configuration, ConfigurationError
  File "/install/lib/python3.9/site-packages/pip/_internal/configuration.py", line 20, in <module>
    from pip._internal.exceptions import (
  File "/install/lib/python3.9/site-packages/pip/_internal/exceptions.py", line 14, in <module>
    from pip._vendor.rich.console import Console, ConsoleOptions, RenderResult
  File "/install/lib/python3.9/site-packages/pip/_vendor/rich/console.py", line 55, in <module>
    from .pretty import Pretty, is_expandable
  File "/install/lib/python3.9/site-packages/pip/_vendor/rich/pretty.py", line 366, in <module>
    class Node:
  File "/install/lib/python3.9/site-packages/dataclasses.py", line 958, in dataclass
    return wrap(_cls)
  File "/install/lib/python3.9/site-packages/dataclasses.py", line 950, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)
  File "/install/lib/python3.9/site-packages/dataclasses.py", line 800, in _process_class
    cls_fields = [_get_field(cls, name, type)
  File "/install/lib/python3.9/site-packages/dataclasses.py", line 800, in <listcomp>
    cls_fields = [_get_field(cls, name, type)
  File "/install/lib/python3.9/site-packages/dataclasses.py", line 659, in _get_field
    if (_is_classvar(a_type, typing)
  File "/install/lib/python3.9/site-packages/dataclasses.py", line 550, in _is_classvar
    return type(a_type) is typing._ClassVar
AttributeError: module 'typing' has no attribute '_ClassVar'

When ecological is removed as a dependency (or copied and built without dataclasses) the error goes away.

My request is to change the pyproject.toml for this project to stop supporting python 3.6 and remove the dependency on dataclasses:

[tool.poetry.dependencies]
python = "^3.7"

There is a precedent for other Python libraries recently dropping support for Python3.6.

Alternatively, I know that with a setup.py, it's possible to specify a dependency like dataclasses only if using an earlier Python version. For instance, in a setup.py, the following would work:

    install_requires=["dataclasses>=0.7;python_version<'3.7'"]

I believe that the pyproject.toml for this project may be modified in the following way to achieve this:

dataclasses = {version = "0.6",  markers = "python_version < '3.7'"}