jmcs / ecological Goto Github PK
View Code? Open in Web Editor NEWEcological combines PEP526 and environment variables to make the configuration of 12 factor apps easy.
License: MIT License
Ecological combines PEP526 and environment variables to make the configuration of 12 factor apps easy.
License: MIT License
Some typing types like Counter
where only added on python 3.6.1. Ecological
should work even if they are not available.
When specifying true
instead of True
as a value for an environment variable typed as bool
in Ecological, the program crashes with an obscure error.
Expect behavior:
true
(as well as TRUE
and other likely variations) is handled identically to True
(and similarly for False
);I don't know if this is or should be limited to boolean values.
$ python3 -m venv /tmp/venv
$ source /tmp/venv/bin/activate
(venv) $ python3 -m pip install ecological
...
Successfully installed ecological-1.6.0
# test.py
import ecological
class Config(ecological.AutoConfig, prefix="test"):
hi: bool = False
print(f"Config.hi = {Config.hi!r}")
(venv) $ python3 test.py # all is well
Config.hi = False
(venv) $ env TEST_HI=True python3 test.py # all is well
Config.hi = True
(venv) $ env TEST_HI=true python3 test.py # "true" instead of "True"
Traceback (most recent call last):
File "/tmp/venv/lib/python3.7/site-packages/ecological/autoconfig.py", line 135, in get
value = self.transform(raw_value, wanted_type)
File "/tmp/venv/lib/python3.7/site-packages/ecological/autoconfig.py", line 90, in cast
if isinstance(representation, str)
File "…/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 91, in literal_eval
return _convert(node_or_string)
File "…/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 90, in _convert
return _convert_signed_num(node)
File "…/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 63, in _convert_signed_num
return _convert_num(node)
File "…/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 55, in _convert_num
raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: <_ast.Name object at 0x7fbf363839e8>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 3, in <module>
class Config(ecological.AutoConfig, prefix="test"):
File "/tmp/venv/lib/python3.7/site-packages/ecological/autoconfig.py", line 196, in __new__
value = attribute.get(attribute_type)
File "/tmp/venv/lib/python3.7/site-packages/ecological/autoconfig.py", line 137, in get
raise ValueError(f"Invalid configuration for '{self.name}': {e}.")
ValueError: Invalid configuration for 'TEST_HI': malformed node or string: <_ast.Name object at 0x7fbf363839e8>.
While trying to sort out storing/retrieving binary values in an env var on Heroku, I noticed that ecological only used os.environ
and not also os.environb
.
I don't know if it's useful to add; just wanted to let you know in case you hadn't considered it. It didn't matter in our case; the Heroku CLI didn't support setting the binary value in the first place, so I fell back on using base64 which worked equivalently for both environ
and environb
.
The documentation should provide some real example on how AutoConfig
works.
Document how Ecological
maps typing "types" to real types.
Currently all tests are functional tests (from user perspective, ecological.Config
is subclassed and then end results are checked). However, pure unit tests are lacking despite reported high coverage. It would be nice to improve this situation.
While the following code works:
class Configuration(ecological.AutoConfig):
integer_list: List
dictionary: Dict
The following doesn't:
class Configuration(ecological.AutoConfig):
integer_list: List[int]
dictionary: Dict[str, str]
Today, Ecological fills configuration classes, so you access your config via class fields. The configuration class is never instantiated. Your configuration is basically stored in a global variable initialized only once, at the very beginning of your program.
It would be awesome if we could ask Ecological to read the environment not at class creation time, but at instance creation time. I believe this would make it easier to test components relying on Ecological (see below) and also allow to "refresh" configurations during the process' lifetime.
Today:
# I define my configuration in a single place, this is great.
class Config(ecological.AutoConfig):
plugins: Sequence[str] = ()
# I want to read my configuration from multiple places, e.g. env & CLI.
# Whatever the source, the configuration fields are the same, so I reuse Config.
# Combining everything is the job of read_config:
def read_config() -> Type[Config]:
args = docopt(...) # for example
if args.plugins:
Config.plugins = (*Config.plugins, *args.plugins)
return Config # superfluous
# I understand I could just grab "Config" outside, it doesn't matter.
But I'm faced with a challenge when I try to write tests for read_config
. I cannot do what comes naturally in Pytest:
def test_read_config(monkeypatch):
monkeypatch.setenv("X", "Y")
assert read_config().x == "Y"
because the environment was read "too early" for this test (when the module containing read_config
was loaded).
I could avoid that problem by creating Config inside read_config
:
def read_config():
class Config(ecological.AutoConfig):
...
# usual read_config logic here
return Config
but this completely breaks static typing (which in my case is a huge reason to use Ecological):
def read_config() -> Type[Config]: # boom: Config is not defined outside of read_config
...
# later
def process(conf: Config): # boom: what is Config?
level = conf.level # as your IDE, I no longer have any idea about the fields of Config
Imho something that only minimally changes Ecological's interface would be best. For example:
class Config(ecological.InstanceConfig, prefix=...): # notice the different superclass
... # nothing else changes
# the environment is not read yet
# later
config = Config() # the environment is read now
config = Config() # the environment is read again
I would then have regular Config instances which type is well-known to every other part of my program, and that can be created at will in tests.
Or equivalent to avoid the issue of having requirements specified in two places like Pipfile
and setup.py
and any other possible difficulty that comes with using setuptools
.
Todo:
setup.py
and Pipfile
to a corresponding pyproject.toml
tox
deprecations along the way.travis.yml
The [tox:travis] section is deprecated in favor of the "python" key of the [travis] section.
Matching undeclared envs is deprecated. Be sure all the envs that Tox should run are declared in the tox config.
Hello. I have used this library a handful of times over the years and it has been helpful to me.
Recently, however, on Python 3.9 I have seen failing builds due to the dependence on dataclasses
. After installing ecological
, all further pip commands throw errors like this:
Traceback (most recent call last):
File "/install/bin/pip", line 5, in <module>
from pip._internal.cli.main import main
File "/install/lib/python3.9/site-packages/pip/_internal/cli/main.py", line 9, in <module>
from pip._internal.cli.autocompletion import autocomplete
File "/install/lib/python3.9/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
from pip._internal.cli.main_parser import create_main_parser
File "/install/lib/python3.9/site-packages/pip/_internal/cli/main_parser.py", line 8, in <module>
from pip._internal.cli import cmdoptions
File "/install/lib/python3.9/site-packages/pip/_internal/cli/cmdoptions.py", line 23, in <module>
from pip._internal.cli.parser import ConfigOptionParser
File "/install/lib/python3.9/site-packages/pip/_internal/cli/parser.py", line 12, in <module>
from pip._internal.configuration import Configuration, ConfigurationError
File "/install/lib/python3.9/site-packages/pip/_internal/configuration.py", line 20, in <module>
from pip._internal.exceptions import (
File "/install/lib/python3.9/site-packages/pip/_internal/exceptions.py", line 14, in <module>
from pip._vendor.rich.console import Console, ConsoleOptions, RenderResult
File "/install/lib/python3.9/site-packages/pip/_vendor/rich/console.py", line 55, in <module>
from .pretty import Pretty, is_expandable
File "/install/lib/python3.9/site-packages/pip/_vendor/rich/pretty.py", line 366, in <module>
class Node:
File "/install/lib/python3.9/site-packages/dataclasses.py", line 958, in dataclass
return wrap(_cls)
File "/install/lib/python3.9/site-packages/dataclasses.py", line 950, in wrap
return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)
File "/install/lib/python3.9/site-packages/dataclasses.py", line 800, in _process_class
cls_fields = [_get_field(cls, name, type)
File "/install/lib/python3.9/site-packages/dataclasses.py", line 800, in <listcomp>
cls_fields = [_get_field(cls, name, type)
File "/install/lib/python3.9/site-packages/dataclasses.py", line 659, in _get_field
if (_is_classvar(a_type, typing)
File "/install/lib/python3.9/site-packages/dataclasses.py", line 550, in _is_classvar
return type(a_type) is typing._ClassVar
AttributeError: module 'typing' has no attribute '_ClassVar'
When ecological
is removed as a dependency (or copied and built without dataclasses
) the error goes away.
My request is to change the pyproject.toml for this project to stop supporting python 3.6 and remove the dependency on dataclasses:
[tool.poetry.dependencies]
python = "^3.7"
There is a precedent for other Python libraries recently dropping support for Python3.6.
Alternatively, I know that with a setup.py
, it's possible to specify a dependency like dataclasses
only if using an earlier Python version. For instance, in a setup.py
, the following would work:
install_requires=["dataclasses>=0.7;python_version<'3.7'"]
I believe that the pyproject.toml for this project may be modified in the following way to achieve this:
dataclasses = {version = "0.6", markers = "python_version < '3.7'"}
Hello,
First I want to say thanks for building this library. I've built many versions of this idea for years and I think this is cleaner than any of my attempts.
I have two things I'd like to include, though, and I wonder if you'd be in interested in them as pull requests. Both are potentially opinionated, though, so feel free to tell me if they don't fit here.
For the first, I often have variables I'd like to exist in the environment and if they don't I want to fail immediately. I usually raise exceptions when parsing these environment variables if they're not present. Thus, I was thinking about a required
argument to your variable class that raises when the variable is not present or false-y in the environment.
For the second, and this isn't a big deal, but I often like to have variables lazy-loading, so they only get parsed from the environment on first access, instead of as class attributes that get parsed and loaded as soon as the class is imported.
I would be happy to contribute one or both of these as pull requests for review if you think they may have a place in this project.
As it is, I think this is a cool library, so thanks for publishing it and putting it on PyPi.
@jmcs I'm testing out a dependency-management tool for the nix language / package manager, and it failed when it couldn't find a source distribution for ecological on pypi.
Can you look into uploading one? :)
If the environmental variable is not set and a default is set, the default should be returned as is, instead of being parsed by the transform function.
Thank you for this great library!
In programs that use a lot of simple data types (strings, integers) for different purposes (account IDs, names, domains, addresses, etc.), I like to use NewType to avoid mixing all of these and clarify the documentation. Thus I may have AccountId, Name, Domain, Address, etc. as newtypes.
It would be great if I could use these types in my configuration class, but it looks like Ecological fallbacks to strings when it doesn't know the type:
>>> I = NewType("I", int)
>>> os.environ["I"] = "2"
>>> class Config(ecological.Autoconfig):
... i: I
>>> Config.i
"2"
This is not a huge inconvenient (and it does the right thing for newtypes of str
anyway), but if it's feasible, it would be the cherry on top. If you are open to this feature, I would even be interested in proposing a PR soon, let me know what you think! 🙂
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.