janehmueller/python-json-config

A config library for python

python-json-config's Issues

Properly set required/optional access in Config Nodes that are created at a later point in time (e.g. upsert)

Pass the value of the member strict_access to children spawned at a later point in time

Enable pickling of Config

Pickle tries to call __getstate__ which fails since it is not handled in the custom __getattr__ method.

Use this solution.

Enable config iterations

It would be great if it were possible to iterate over the configuration.

For example:

for key, value in config.items():
    print(f'{key} --> {value}')

Return config after env merge

Add a return self statement at the end of the merge_with_env_variables function

When upserting a new value in a new subfield, the subfield is not added

E.g., we add value x to path [foo, bar] but foo does not exist yet. The subpath foo is not added and an error is thrown.

Use msgpack for binary serialization

It would be great if one could also serialize the config to a binary format that can also be read by other languages such as Java or C++. The msgpack library is ideal for this purpose. For example, there is a msgpack library in all common programming languages, which makes communication easy.

Therefore I propose to add two instance methods to the config object called to_bytes and from_bytes which use msgpack to serialize and deserialize the configuration from and to a sequence of bytes.

Revise ENV merging

Instead of merging the current config with the ENV variables by doing:

builder = ConfigBuilder()
# you can also just pass a single prefix (builder.merge_with_env_variables("MYPROJECT")
builder.merge_with_env_variables(["MYPROJECT", "MYPYTHONPROJECTS"])
config = builder.parse_config({"server.host": "0.0.0.0"})

It should be possible to do it within the builder patter like this:

builder = ConfigBuilder()
builder.parse_config({"server.host": "0.0.0.0"}).merge_with_env_variables("MYPROJECT").merge_with_env_variables("MYPYTHONPROJECTS")

JSONSchema support

Instead of configuring the CofigBuilder directly in the code, it should be possible to pass a JSONSchema file which describes the configuration. This tutorial describes how JSONSchema works.

This file should define the schema of the configuration. Then the corresponding validators and converters can be used while reading the configuration.

Check if key exists in config

It would be good to have the possibility to check if a certain configuration parameter is defined within the configuration or not. So basically something like this should be possible:

if 'backend.media.type' in config:
    return True
else:
    return False

Return as dict

Unfortunately, it is often the case that other libraries only accept dictionaries as configuration input. For this reason, it would be a nice feature to be able to access all or part of the configuration as a normal dictionary.

Submodule for validators and converters

There should be two submodules that already provide a number of validators and converters for certain objects.

For example, validators for IP addresses, port numbers, etc.

Merge with ENV variables

It would be nice if one could specify the prefix or list of prefixes for environment variables that should be merged into the configuration. For example to merge all environment variables starting with "FOO_" into the configuration. If a key already exists, the corresponding value will be overwritten, it doesn't exist, it will be added.

A possible implementation might look like this:

    def _merge_with_env_variables(self, prefix: Union[str, List[str]]):
        """
        This method takes all environment variables that start with the specified prefixes,
        and merges them into the config. If the key is not already present in the config,
        it will be added, if the key is present in the config it will be overwritten

        :param prefix: a prefix or list of prefixes for the env variables to merge
        """
        prefixes = []
        if isinstance(prefix, str):
            prefixes = [prefix]

        for key in os.environ.keys():
            for prefix in prefixes:
                if key.startswith(prefix):
                    value = os.environ[key]
                    key = key.replace(prefix, '').lower().replace('_', '.')
                    # TODO: Merge key and value into the configurtation
                    self.config.update(key, value)

Handling of optional values

If you define a key for which certain checks are to be performed, it is implicitly assumed that this key is always present. However, this assumption does not hold in all cases. For example it is quite possible that a key is optional. So if it is defined, then certain checks should be done, but if it is missing, then this is OK. In this case no error should be thrown. For example in the following case:

config.validate_field_type('backend.username', str)

it would be OK if the key is missing. All in all it should be possible to define which keys are optional.

Switch to pytest and add flake 8 linting

Better error reporting

There is currently no way to specify what exactly failed while parsing the config. In the optimal case, every validator should be able to define a message to show the user what exactly went wrong.

For example: "There was an error while validating the host field, 127.0.0." is not a valid IP address."

Add new keys to the config

It needs to be possible to attach new keys to config during runtime.

So for example:
config['foo.bar'] = 'xyz' and/or config.foo.bar = 'xyz' or config.add('foo.bar', 'xyz')

Chain validator functions

It would be great if one would be able to execute several validator functions in succession. So instead of having something like:

builder.validate_field_value('server.port', function_1)

it would be great if one could do:

builder.validate_field_value('server.port', [function_1, function_2])

... so that the specified validators get executed one after another.

Environment variable prefix should only match those variables with an underscore after the prefix

E.g. with prefix TEST the following should happen

• match TEST_VALUE
• not match TESTSUITE_VERSION

Currently it matches both

Serialization methods (JSON)

It would be great if a parsed configuration could also be transformed back into a dictionary or json string format. For example like: config.dump() or config.to_json()

Handle environment variables that overwrite config values with underscores in their names.

Currently if a field has an underscore in its name (e.g., log_file), the field can't be overwritten via environment variables. This can be enabled by escaping these underscores with another underscore. (In this case setting the variable PREFIX_LOG__FILE)

Validation of optional fields throws errors when the field does not exist

This was an oversight in #8

Use proper python 3.7 for travis and cache pip dependencies

For python 3.7 to work with travis the build needs to run on xenial (Ubuntu 16.04) instead of trusty (Ubuntu 14.04) and require sudo. (reference).

The travis file should have the following

python:
  - "3.6"
matrix:
  include:
    - python: 3.7
      dist: xenial
      sudo: true

Also add caching for pip dependencies:

cache: pip