martinthoma/cfg_load

Load config files the right way

Hi 👋

I'm Martin and I am currently based in Germany working as a Lead Developer / Python Backend Dev.

• 🤝 Interested in open source and free software projects.
• 📫 You can reach me best via e-mail to [email protected]

If you like my work, you can support me:

---

Blog posts

• ReDoS — Denial of Service by RegEx
• 4 Alternatives to the Standard Interactive Python Shell
• Designing Beautiful Command-Line Applications With Python
• The State of PyPI — 2022 Edition
• 5 Ways to get Enums in Python

Programming Languages and Technology

cfg_load's Issues

Allow adding custom parsers for ENV variables

With YAML, one can add lists and dictionaries as configuration values.

Environment variables can only contain strings.

I, as a user, want to overwrite all possible values with an environment variable and still have the same type.

ModuleNotFoundError: No module named 'tzlocal' on dependency "mpu"

Hello! A fresh install with poetry (python package manager) is giving an error of ModuleNotFoundError when the function cfg.load() is called.

Somehow the module tzlocal is not getting installed?
Is this with how cfg_load declares its deps? or is it on the mpu dep side?

Trace:

Traceback (most recent call last):
  File "/home/martin/Documentos/Programming/Python/Projetos/test-project/test_project/__init__.py", line 6, in <module>
    cfg.load("/home/martin/Documentos/Programming/Python/Projetos/steam-api-explore/cfg-template.yaml")
  File "/home/martin/Documentos/PythonEnvironments/steam-api-explore-oyX2e_1N-py3.8/lib/python3.8/site-packages/cfg_load/__init__.py", line 66, in load
    meta = mpu.io.get_file_meta(filepath)
  File "/home/martin/Documentos/PythonEnvironments/steam-api-explore-oyX2e_1N-py3.8/lib/python3.8/site-packages/mpu/io.py", line 366, in get_file_meta
    meta["last_access_datetime"] = get_access_datetime(filepath)
  File "/home/martin/Documentos/PythonEnvironments/steam-api-explore-oyX2e_1N-py3.8/lib/python3.8/site-packages/mpu/io.py", line 344, in get_access_datetime
    import tzlocal
ModuleNotFoundError: No module named 'tzlocal'

I have included a minimal pyproject for you to test.

pyproject.toml

[tool.poetry]
name = "test-project"
version = "0.1.0"
description = ""
authors = ["Martin <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.8"
cfg-load = "*"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Sample file:

__version__ = '0.1.0'

import cfg_load as cfg

if __name__ == '__main__':
    cfg.load("/home/martin/Documentos/Programming/Python/Projetos/steam-api-explore/cfg-template.yaml")

Add to_dict()

Sometimes it is handy to just have a dict. For this reason, the Configuration objects should have a to_dict() method

Document list of available converters

https://github.com/MartinThoma/cfg_load/blob/master/cfg_load/__init__.py#L366

Add times as metadata

• When was the file read?
• when was it last modified?
• When was it created?

Allow validation of configuration

• Keys that have to exist
• strings minimum / maximum length; defined set
• floats: min / max
• relative: One key has to be bigger than another

Configuration file validator

Please add a CLI tool in this package that allows direct validation of configuration files.

Add _META

The key config['_META'] should contain the filepath and the datetime (in UTC) of the loaded configuration.

Can cfg_load be used by PyCharm to load a configuration

I'm new to configuration files and really like your setup.

Can or should a cfg_load file be modified to be opened or run with PyCharm or Docker?

Still learning how this works.

How can I specify end line in INI file

How to specify EOL such as \n or \n\r in ini files?

Add 'policy' to _load_remote

Currently, _load_remote does not have the policy implemented (here). The default is always used.

_path magic is not applied in lists

Add load_if_possible policies to remote.load

The load_if_possible policy should always try to load the remote file. If it fails, the local one is used.

Create nicer readme for pypi.org

https://pypi.org/project/cfg-load/

Add cfg.pprint()

For easy inspection in REPL, a cfg.pprint() method would be nice.

Add parameter to not load remote files

Add git hash

In case the config file is in a git repository, add the hash.

Add str_empty_is_none converter

If the string is empty, then convert it to None

Make config immutable

Overwriting the configuration during the application is likely to lead to an error. It should be done with caution. Thus it might be desirable to at least log a message if the configuration is changed.

Add recursive boolean option to update

I have a nested dictionary that i'd like updated to be empty when I update with a user config. Ideally the top level attribute would be set to {} and then base_cfg.update(user_cfg, recursive=False) would nuke the whole dict with the user value of {}

Requires mpu[io]

Update mpu

See MartinThoma/mpu@5746b5b

Define how to deal with ENV variables

Add support for TOML

Please add support for Tom's Obvious, Minimal Language. Realizing it's much like INI files, support is almost there. TOML is sufficiently different from INI that there are dedicated TOML libraries in Python and ConfigParser does not explicitly represent that is handles TOML.

https://en.wikipedia.org/wiki/TOML

Allow override configuration with command line args [enhancement]

Nice work to streamline config files! What I usually do is load the configuration file, but allow the command line arguments to override the settings. The advantage is that the conf file does not need changing for every small experiment.

The hierarchy stays the same, for example in the (yaml) conf file:

sound:
    volume_left: 50

Could be exec'd with modified settings like so:
myfile.py --sound.volume_left 80

If you like the idea, I can implement the feature.