Giter Site home page Giter Site logo

environs's Introduction

environs: simplified environment variable parsing

Latest version Build Status marshmallow 3 compatible

environs is a Python library for parsing environment variables. It allows you to store configuration separate from your code, as per The Twelve-Factor App methodology.

Contents

Features

  • Type-casting
  • Read .env files into os.environ (useful for local development)
  • Validation
  • Define custom parser behavior
  • Framework-agnostic, but integrates well with Flask and Django

Install

pip install environs

Basic usage

With some environment variables set...

export GITHUB_USER=sloria
export MAX_CONNECTIONS=100
export SHIP_DATE='1984-06-25'
export TTL=42
export ENABLE_LOGIN=true
export GITHUB_REPOS=webargs,konch,ped
export GITHUB_REPO_PRIORITY="webargs=2,konch=3"
export COORDINATES=23.3,50.0
export LOG_LEVEL=DEBUG

Parse them with environs...

from environs import Env

env = Env()
env.read_env()  # read .env file, if it exists
# required variables
gh_user = env("GITHUB_USER")  # => 'sloria'
secret = env("SECRET")  # => raises error if not set

# casting
max_connections = env.int("MAX_CONNECTIONS")  # => 100
ship_date = env.date("SHIP_DATE")  # => datetime.date(1984, 6, 25)
ttl = env.timedelta("TTL")  # => datetime.timedelta(0, 42)
log_level = env.log_level("LOG_LEVEL")  # => logging.DEBUG

# providing a default value
enable_login = env.bool("ENABLE_LOGIN", False)  # => True
enable_feature_x = env.bool("ENABLE_FEATURE_X", False)  # => False

# parsing lists
gh_repos = env.list("GITHUB_REPOS")  # => ['webargs', 'konch', 'ped']
coords = env.list("COORDINATES", subcast=float)  # => [23.3, 50.0]

# parsing dicts
gh_repos_priorities = env.dict(
    "GITHUB_REPO_PRIORITY", subcast_values=int
)  # => {'webargs': 2, 'konch': 3}

Supported types

The following are all type-casting methods of Env:

  • env.str
  • env.bool
  • env.int
  • env.float
  • env.decimal
  • env.list (accepts optional subcast and delimiter keyword arguments)
  • env.dict (accepts optional subcast_keys, subcast_values and delimiter keyword arguments)
  • env.json
  • env.datetime
  • env.date
  • env.time
  • env.timedelta (assumes value is an integer in seconds)
  • env.url
  • env.uuid
  • env.log_level
  • env.path (casts to a pathlib.Path)
  • env.enum (casts to any given enum type specified in type keyword argument, accepts optional ignore_case keyword argument)

Reading .env files

# .env
DEBUG=true
PORT=4567

Call Env.read_env before parsing variables.

from environs import Env

env = Env()
# Read .env into os.environ
env.read_env()

env.bool("DEBUG")  # => True
env.int("PORT")  # => 4567

Reading a specific file

By default, Env.read_env will look for a .env file in current directory and (if no .env exists in the CWD) recurse upwards until a .env file is found.

You can also read a specific file:

from environs import Env

with open(".env.test", "w") as fobj:
    fobj.write("A=foo\n")
    fobj.write("B=123\n")

env = Env()
env.read_env(".env.test", recurse=False)

assert env("A") == "foo"
assert env.int("B") == 123

Handling prefixes

# export MYAPP_HOST=lolcathost
# export MYAPP_PORT=3000

with env.prefixed("MYAPP_"):
    host = env("HOST", "localhost")  # => 'lolcathost'
    port = env.int("PORT", 5000)  # => 3000

# nested prefixes are also supported:

# export MYAPP_DB_HOST=lolcathost
# export MYAPP_DB_PORT=10101

with env.prefixed("MYAPP_"):
    with env.prefixed("DB_"):
        db_host = env("HOST", "lolcathost")
        db_port = env.int("PORT", 10101)

Variable expansion

# export CONNECTION_URL=https://${USER:-sloria}:${PASSWORD}@${HOST:-localhost}/
# export PASSWORD=secret
# export YEAR=${CURRENT_YEAR:-2020}

from environs import Env

env = Env(expand_vars=True)

connection_url = env("CONNECTION_URL")  # =>'https://sloria:secret@localhost'
year = env.int("YEAR")  # =>2020

Validation

# export TTL=-2
# export NODE_ENV='invalid'
# export EMAIL='^_^'

from environs import Env
from marshmallow.validate import OneOf, Length, Email

env = Env()

# simple validator
env.int("TTL", validate=lambda n: n > 0)
# => Environment variable "TTL" invalid: ['Invalid value.']


# using marshmallow validators
env.str(
    "NODE_ENV",
    validate=OneOf(
        ["production", "development"], error="NODE_ENV must be one of: {choices}"
    ),
)
# => Environment variable "NODE_ENV" invalid: ['NODE_ENV must be one of: production, development']

# multiple validators
env.str("EMAIL", validate=[Length(min=4), Email()])
# => Environment variable "EMAIL" invalid: ['Shorter than minimum length 4.', 'Not a valid email address.']

Deferred validation

By default, a validation error is raised immediately upon calling a parser method for an invalid environment variable. To defer validation and raise an exception with the combined error messages for all invalid variables, pass eager=False to Env. Call env.seal() after all variables have been parsed.

# export TTL=-2
# export NODE_ENV='invalid'
# export EMAIL='^_^'

from environs import Env
from marshmallow.validate import OneOf, Email, Length, Range

env = Env(eager=False)

TTL = env.int("TTL", validate=Range(min=0, max=100))
NODE_ENV = env.str(
    "NODE_ENV",
    validate=OneOf(
        ["production", "development"], error="NODE_ENV must be one of: {choices}"
    ),
)
EMAIL = env.str("EMAIL", validate=[Length(min=4), Email()])

env.seal()
# environs.EnvValidationError: Environment variables invalid: {'TTL': ['Must be greater than or equal to 0 and less than or equal to 100.'], 'NODE_ENV': ['NODE_ENV must be one of: production, development'], 'EMAIL': ['Shorter than minimum length 4.', 'Not a valid email address.']}

env.seal() validates all parsed variables and prevents further parsing (calling a parser method will raise an error).

URL schemes

env.url() supports non-standard URL schemes via the schemes argument.

REDIS_URL = env.url(
    "REDIS_URL", "redis://redis:6379", schemes=["redis"], require_tld=False
)

Serialization

# serialize to a dictionary of simple types (numbers and strings)
env.dump()
# {'COORDINATES': [23.3, 50.0],
# 'ENABLE_FEATURE_X': False,
# 'ENABLE_LOGIN': True,
# 'GITHUB_REPOS': ['webargs', 'konch', 'ped'],
# 'GITHUB_USER': 'sloria',
# 'MAX_CONNECTIONS': 100,
# 'MYAPP_HOST': 'lolcathost',
# 'MYAPP_PORT': 3000,
# 'SHIP_DATE': '1984-06-25',
# 'TTL': 42}

Defining custom parser behavior

# export DOMAIN='http://myapp.com'
# export COLOR=invalid

from furl import furl


# Register a new parser method for paths
@env.parser_for("furl")
def furl_parser(value):
    return furl(value)


domain = env.furl("DOMAIN")  # => furl('https://myapp.com')


# Custom parsers can take extra keyword arguments
@env.parser_for("choice")
def choice_parser(value, choices):
    if value not in choices:
        raise environs.EnvError("Invalid!")
    return value


color = env.choice("COLOR", choices=["black"])  # => raises EnvError

Usage with Flask

# myapp/settings.py

from environs import Env

env = Env()
env.read_env()

# Override in .env for local development
DEBUG = env.bool("FLASK_DEBUG", default=False)
# SECRET_KEY is required
SECRET_KEY = env.str("SECRET_KEY")

Load the configuration after you initialize your app.

# myapp/app.py

from flask import Flask

app = Flask(__name__)
app.config.from_object("myapp.settings")

For local development, use a .env file to override the default configuration.

# .env
DEBUG=true
SECRET_KEY="not so secret"

Note: Because environs depends on python-dotenv, the flask CLI will automatically read .env and .flaskenv files.

Usage with Django

environs includes a number of helpers for parsing connection URLs. To install environs with django support:

pip install environs[django]

Use env.dj_db_url, env.dj_cache_url and env.dj_email_url to parse the DATABASE_URL, CACHE_URL and EMAIL_URL environment variables, respectively.

For more details on URL patterns, see the following projects that environs is using for converting URLs.

Basic example:

# myproject/settings.py
from environs import Env

env = Env()
env.read_env()

# Override in .env for local development
DEBUG = env.bool("DEBUG", default=False)
# SECRET_KEY is required
SECRET_KEY = env.str("SECRET_KEY")

# Parse database URLs, e.g.  "postgres://localhost:5432/mydb"
DATABASES = {"default": env.dj_db_url("DATABASE_URL")}

# Parse email URLs, e.g. "smtp://"
email = env.dj_email_url("EMAIL_URL", default="smtp://")
EMAIL_HOST = email["EMAIL_HOST"]
EMAIL_PORT = email["EMAIL_PORT"]
EMAIL_HOST_PASSWORD = email["EMAIL_HOST_PASSWORD"]
EMAIL_HOST_USER = email["EMAIL_HOST_USER"]
EMAIL_USE_TLS = email["EMAIL_USE_TLS"]

# Parse cache URLS, e.g "redis://localhost:6379/0"
CACHES = {"default": env.dj_cache_url("CACHE_URL")}

For local development, use a .env file to override the default configuration.

# .env
DEBUG=true
SECRET_KEY="not so secret"

For a more complete example, see django_example.py in the examples/ directory.

Why...?

Why envvars?

See The 12-factor App section on configuration.

Why not os.environ?

While os.environ is enough for simple use cases, a typical application will need a way to manipulate and validate raw environment variables. environs abstracts common tasks for handling environment variables.

environs will help you

  • cast envvars to the correct type
  • specify required envvars
  • define default values
  • validate envvars
  • parse list and dict values
  • parse dates, datetimes, and timedeltas
  • parse expanded variables
  • serialize your configuration to JSON, YAML, etc.

Why another library?

There are many great Python libraries for parsing environment variables. In fact, most of the credit for environs' public API goes to the authors of envparse and django-environ.

environs aims to meet three additional goals:

  1. Make it easy to extend parsing behavior and develop plugins.
  2. Leverage the deserialization and validation functionality provided by a separate library (marshmallow).
  3. Clean up redundant API.

See this GitHub issue which details specific differences with envparse.

License

MIT licensed. See the LICENSE file for more details.

environs's People

Contributors

alexpirine avatar augpro avatar bvanelli avatar c-w avatar dependabot-preview[bot] avatar dependabot-support avatar dependabot[bot] avatar deronnax avatar epicserve avatar gnarvaja avatar gregoiredx avatar gthank avatar gvialetto avatar hukkinj1 avatar hvtuananh avatar kamforka avatar kochankovid avatar nvtkaszpir avatar parnassius avatar pre-commit-ci[bot] avatar rcuza avatar ribeaud avatar rjcohn avatar sabdouni avatar sloria avatar tomgrin10 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

environs's Issues

Add schema API?

Proposed API

env = Env(BOOLEAN_VAR=bool, LIST_VAR=dict(cast=list, subcast=int))
env('BOOLEAN_VAR')

Two proxied variables

Suppose I'm trying to construct a variable by concatenating two proxied variables...

VAR_A=foo
VAR_B=bar

VAR_C={{VAR_A}}_{{VAR_B}

Environs will fail to parse the two proxies and claim that the a joined variable is not set

>>> import environs
>>> env = envions.Env()
>>> env.read_env(...)
>>> env('VAR_C') 
environs.EnvError:  Environment variable "VAR_A}}_{{VAR_B" not set

error parsing URLs for docker-compose style hostnames

Hello,

I have a parsing issue using env.url for docker-compose type URLs.

For example, if I have a service called vault that I want to reach from another service, I'm passing it an environment variable like so: VAULT_ADDRESS=http://vault:8200

This doesn't work:

from environs import Env

environment = Env(eager=False)
environment.read_env()


class Settings:
    vault_address: ParseResult = environment.url("VAULT_ADDRESS")


environment.seal()

Error:
environs.EnvValidationError: Environment variables invalid: {'VAULT_ADDRESS': ['Not a valid URL.']}

This works:

from environs import Env
from urllib.parse import ParseResult, urlparse

environment = Env(eager=False)

@environment.parser_for("url2")
def url2_parser(value):
    return urlparse(value)

environment.read_env()


class Settings:
    vault_address: ParseResult = environment.url2("VAULT_ADDRESS")


environment.seal()
  • Tested on 7.3.0 and 7.4.0

dump() is lazy?

> from environs import Env
> env = Env()
> env.read_env()
> loaded_env = env.dump()
> 
> loaded_env
> {}
> 
> mytest = env('MYVAR')
> loaded_env = env.dump()
> loaded_env
> {'MYVAR': my value}
> 

So loaded_env remains empty, even if there are valid entries in .env
After a variable is accessed via env('MYVAR), it appears in the dump() result...

Why not in the first case?

Chamber support

I'm not sure if this is a good idea or not but it might make things simpler if there was a hook for loading chamber secrets into environs if you're using chamber.

Allow opening other files, not just .env

Currently environs read only hard-coded .env files, and only path is available.
Yet more and more tools allow custom .env files.

I believe environs should add parameter like file='.env' to be able to override defaults.

How about an environs-cli?

Just some random thoughts ... but it might be cool if there was an environs-cli that could generate an empty .env file so you could easily the settings you need to set when creating a new project. Maybe the output could be something like the following. Inspiration comes from, https://github.com/lincolnloop/goodconf.

DEBUG=off # Default (bool)
INTERNAL_IPS=127.0.0.1,172.18.0.1,localhost,0.0.0.0  # Default (list)
ALLOWED_HOSTS=127.0.0.1  # Default (list)
DOMAIN='127.0.0.1:8000'  # Default (string)
EMAIL_HOST=smtp.mailtrap.io # Dault (string)
EMAIL_HOST_USER=  # No Default (string)
EMAIL_HOST_PASSWORD=  # No Default (string)
EMAIL_PORT=25  # Default (string)  
SECRET_KEY=''  # No Default (string)
DATABASE_URL=''  # No Default (string)
AWS_ACCESS_KEY_ID=''  # No Default (string)
AWS_SECRET_ACCESS_KEY=''  # No Default (string)
BITLY_ACCESS_TOKEN=''  # No Default (string)
SENTRY_DSN=''  # No Default (url)

Change (or at least document) surprising behavior parsing env var when explicitly set to empty string

Environs users I'm supporting just hit this:

cat config/foo.env 
MYINT="0"
MYFLOAT="0.0"
MYBOOL="False"
MYSTR=""

➜ . config/foo.envpython3
...
>>> from environs import Env
>>> env = Env()
>>> env.int("MYINT")
0
>>> env.float("MYFLOAT")
0.0
>>> env.bool("MYBOOL")
False
>>> env.str("MYSTR")
<marshmallow.missing>

What? Expected empty string, as explicitly set:

>>> from os import environ
>>> environ["MYSTR"]
''

This violates reasonable assumptions that the type of the value returned by env.foo is always foo, as is otherwise consistent with env.bool, env.int, env.float, etc.:

>>> isinstance(env.str("MYSTR"), str)
False

In this use case, the user should not need to know what a marshmallow.missing is, as it is an implementation detail, and should get back the explicitly set value of empty string rather than needing special logic that knows how to deal with marshmallow.missing.

If the current behavior cannot be changed e.g. for compatibility reasons, it should at least be documented, as this seems like a common use case.

Happy to submit a PR if that would be of interest. Thanks!

Add .url() parser

I think URLs deserve first-class handling in environs, using urllib.parse.urlparse as the parser and throwing the corresponding exception when the value is formatted incorrectly.

I know I can plug it in myself. But URLs are built-in Python functionality and framework-agnostic, I don't see how they are different from, say, timedeltas.

Reading from multiple .env files with overriding

Hey so I was expecting this to work:

# .env
FOO=bar
# dev.env
FOO=baz
env = Env()
env.read_env('.env')
env.read_env('dev.env')

assert env('FOO') != 'bar' and env('FOO') == 'baz', 'dev.env did not override .env'

Any reason we don't like this behavior?

Implement `__contains__` in `Env`

Feature request

Would be nice if we could do

env = Env()
if "SOME_ENV_VAR" in env:
    ....

to check if an env variable exists. I.e. implement the __contains__ dunder method.

The benefit of this compared to "SOME_ENV_VAR" in os.environ is that it could be used in a with env.prefixed("PREFIX"): statement.

IDE autocomplete support

HI!
Thanks for the package, looks amazing, but what about IDE autocomplete / IntelliSense / hints?
The IDE (I'm using PyCharm) does not prompt existing methods of the Env class instance (see picture below).

Screenshot from 2019-09-20 11-15-20

For me, this is an important drawback of this package.
Will it be decided?
Thanks!

Dependabot couldn't fetch all your path-based dependencies

Dependabot couldn't fetch one or more of your project's path-based Python dependencies. The affected dependencies were './setup.py.

To use path-based dependancies with Dependabot the paths must be relative and resolve to a directory in this project's source code.

You can mention @dependabot in the comments below to contact the Dependabot team.

Read .env files

Proposed API

env.read_env()  # searches for a .env in the file hierarchy, parses it into os.environ

Considerations

This is already implemented by envparse and django-environ (it's hard to tell, but I think original credit goes to the author of this gist: https://gist.github.com/bennylope/2999704).

I'm hesitant to just copy and paste the implementation from one of these sources into this library. I'm more inclined to release a separate library and just import that library here. That way, other libraries could reuse this code. Also, users may want to read .env files without having to use environs, envparse, etc.

Possibility to disable proxied variables

First of all: thanks for the library! Makes env-handling much easier :)

Nevertheless I am facing an issue with proxied variables, we want to allow env-vars having a content like {{EXAMPLE}}, but unfortunately environs tries then to get the content of an env-var EXAMPLE.

I found this described in the documentation as "Proxied variables" and also took a look at the code, unfortunately it is not possible to disable this and that means for us we have to switch for these env-vars back to good-old-os.environs.

If it is fine for you I can try to create a small PR, I would propose the following two solutions:

  1. Global setting, e. g. provide a property proxy_variables in the Env() that is by default true and could be set to false
  2. Named argument proxy_variable when getting an env, e. g. env("SMTP_LOGIN", proxy_variable=False) (of course by-default also true)

Validation only happens one field at a time

Currently, with the following code:

from environs import Env
env = Env()
var1 = env("VAR1")
var2 = env("VAR2")

If both env vars are missing, environs will simply display:

environs.EnvError: Environment variable "VAR1" not set

And only when that's set will it display the second error.

It'd be great to have it validate as far as possible all the required environment variables, perhaps with a manual flagging of when to start and stop such a process:

from environs import Env
env = Env(eager=False)
var1 = env("VAR1")
var2 = env("VAR2")
var3 = env.int("VAR3")
env.read()

environs.EnvError: Environment variables invalid: {"VAR1": "missing", "VAR2": "missing", "VAR3": "wrong type"]. 

Support for env var substitutions in env.str and env.dict and other composite types

I found some use cases where doing env var substitution inside the VALUE of another envvar it's very helpful.

For example:

DB_URI=postgres://${PGUSER:-gnarvaja}:${PGPASS}@myserver/mydb
PGPASS=awesome

And being able to read the variable like this would be awesome:

>>> env.str("DB_URI", substitute_envs=True)
postgres://gnarvaja:awesome@myserver/mydb

The main use case is when you have the user/pass as individuals secrets (perhaps handled by an operator like https://github.com/zalando/postgres-operator) but you need a URI with parts that don't need to be secret.

Another use case is a workaround when composite variables like dict become too complex. For example:

INIT_KARGS="name=foo,script_params=--http-header Referer=https://radiocut.fm"

If you do env.dict("INIT_KARGS") you get ValueError: too many values to unpack because of the "=" in the second value. So in this case you can do:

INIT_KARGS="name=foo,script_params=--http-header ${HTTP_HEADER}"
HTTP_HEADER="Referer=https://radiocut.fm"
>>> env.dict("INIT_KARGS", substitute_envs=True)
{"name": "foo", "script_params": "--http-header Referer=https://radiocut.fm"}

We can use a syntax like this for the references to the environments variables: ${VARIABLE[:-default][:ttype]}. For example ${YEAR:-2020:tint}. (the :- is how you specify defaults in bash, the third :t parameter is for specifying the type is not str).

Here is the code I'm using in my project (a bit ugly):

envvar_matcher = re.compile(r'\$\{([A-Za-z0-9_]+)(:-[^\}:]*)?(:t[^\}]*)?\}')


def _replace_envs(value):
    ret = ""
    prev_start = 0
    for m in envvar_matcher.finditer(value):
        env_name = m.group(1)
        env_default = m.group(2)
        env_type = m.group(3)
        if env_type is None:
            method = env.str
        else:
            method_name = env_type[2:]
            method = getattr(env, method_name)
        if env_default is None:
            env_value = method(env_name)
        else:
            env_default = env_default[2:]
            env_value = method(env_name, default=env_default)

        if m.start() == 0 and m.end() == len(value):
            return env_value
        ret += value[prev_start:m.start()] + env_value
        prev_start = m.end()
    ret += value[prev_start:]
    return ret

If you agree with this new feature, I can try to do a PR. The new feature will be activated only if explicitly using substitute_envs=True as parameter.

Exposing derived settings via Env() Objects.

Hi,

I would like to know if it is possible to add new varaibles to the Env() object after reading the file.

For example, lets say I have the screen HEIGHT and WIDTH and RESOLUTION = HEIGHT * WIDTH. This is a simple example, but there could also be cases where the output from some function could be assigned.

Now in the code, I can access env("RESOLUTION") instead of having to do the multiplication again and again.

Thanks for the great library and for considering my request.

Make Env methods self-documenting

I would like to preface this issue by saying that I really love this library, and use it a lot in my projects 😀

I personally see great importance in self-documenting code, and a large part of self-documenting code is being able to view
a function's arguments and their types in an IDE when using an API, without looking at its docs.
As this library has no official API reference (which is totally fine!), I think its crucial that the function signatures are clear to the user.

I see 2 choices for achieving this:

  1. Adding a .pyi file. (this means that every change to Env's API means a change in the .pyi file too)
  2. Rewrite Env's methods so that they have a clear signature that IDEs can parse.

I personally think the 1st option is easier, but wanted to know your opinion before submitting a PR.

Dependabot couldn't fetch all your path-based dependencies

Dependabot couldn't fetch one or more of your project's path-based Python dependencies. The affected dependencies were './setup.py.

To use path-based dependancies with Dependabot the paths must be relative and resolve to a directory in this project's source code.

You can mention @dependabot in the comments below to contact the Dependabot team.

Don't throw IOError for missing .env file

First of all thanks very much for this beautiful library, I'm using it to replace a few others as well as a lot of manual type-casting.

I would like to suggest making read_env() only opportunistically search for a .env file, and do nothing (or perhaps log a warning) if no file is found. This is the way django-environ, envparse, and python-dotenv all work.

I think it makes sense to support .env files in situations where there is no better way to manage environment variables, such as running an application locally, but not require them when running in production under, say, Docker or a traditional process manager.

I can of course work around this with a simple try/catch, but a change like this would make the library even nicer to use.

env.log_level("LOG_LEVEL") with LOG_LEVEL=info returns "logging.info" function

The current code of log_level handler, just does getattr(value, logging) to get the value, so if you put the log level in lowercase you don't get an error but instead the module funcion is returned.

I made a fix that applies .upper() to the value and also validates the module member getattr(logging, value) is int for extra validation.

PR #138

Set default value once and reuse

Coming from django-environ, I thought env(varname, "defaultval") would set a default to be automatically used on subsequent calls to env(varname). There should be a way to set a default value for an environment var, to be overrided or not on subsequent calls. Something like env.setdefault('varname', "originaldefault") would do; then if varname is not defined in environment, susbsequent calls to env("varname") would output "originaldefault" instead of raising the usual exception.

Remove usage of implicit Optional

A past version of this PEP allowed type checkers to assume an optional type when the default value is None, as in this code:

def handle_employee(e: Employee = None): ...

This would have been treated as equivalent to:

def handle_employee(e: Optional[Employee] = None) -> None: ...

This is no longer the recommended behavior. Type checkers should move towards requiring the optional type to be made explicit.

https://www.python.org/dev/peps/pep-0484/

We currently use implicit Optionals. mypy still allows them but may change this in the future.

Preprocess functions can't handle marshmallow.missing

The preprocess functions of Env.list, Env.dict and Env.json can't handle a marshmallow.missing.

This can be reproduced by e.g.

env = Env(eager=False)
env.list("NON_EXISTING_ENVVAR")

The above will raise an unhandled unexpected exception.

Haven't looked into the problem in too much detail, but I'm guessing the solution is to either skip the preprocess function if marshmallow.missing, or make the spec of preprocess functions so that they take a Union[str, marshmallow._Missing, <insert-the-fields-proper-type>] as input.

Prefixed Method Does Not Clear self._prefix On Error

version: 4.1.3

Problem

When using the following code, the private variable self._prefix is not cleared on errors:

with env.prefixed(PREFIX):
    do_something()

If do_something() raises an error, then session attempts to run with env.prefixed() again it will behave like it is a nested with. This can happen in an interactive shell or if exceptions are caught.

Details

You can do the following to reproduce.

$ export PREFIX_PNG="path/to/my.png"
$ ipython
/usr/local/Cellar/ipython/7.5.0/libexec/lib/python3.7/site-packages/IPython/core/interactiveshell.py:925: UserWarning: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
  warn("Attempting to work in a virtualenv. If you encounter problems, please "
Python 3.7.3 (default, Mar 27 2019, 09:23:15)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from environs import Env

In [2]: env = Env()

In [3]: with env.prefixed("PREFIX_"):
   ...:     print(env.str("PNG"))
   ...:
path/to/my.png

In [4]: with env.prefixed("PREFIX_"):
   ...:     raise
   ...:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-6c7273e50e26> in <module>
      1 with env.prefixed("PREFIX_"):
----> 2     raise
      3

RuntimeError: No active exception to reraise

In [5]: with env.prefixed("PREFIX_"):
   ...:     print(env.str("PNG"))
   ...:
---------------------------------------------------------------------------
EnvError                                  Traceback (most recent call last)
<ipython-input-5-60d974fcda59> in <module>
      1 with env.prefixed("PREFIX_"):
----> 2     print(env.str("PNG"))
      3

~/dev/virtualenvs/tmp-da5b0d36b01c294/lib/python3.7/site-packages/environs.py in method(self, name, default, subcast, **kwargs)
     51         if raw_value is ma.missing and field.missing is ma.missing:
     52             raise EnvError(
---> 53                 'Environment variable "{}" not set'.format(proxied_key or parsed_key)
     54             )
     55         if raw_value or raw_value == "":

EnvError: Environment variable "PREFIX_PREFIX_PNG" not set

env.dump() only shows pre-loaded variables

I just started using environs, so let me know if I'm not understanding the best way to use it, but what I expect env.dump() to do is dump all the variables from .env. It appears that it only dumps variables which have already been called on (i.e. env('FLASK_APP')), and only then will it appear in the output of env.dump(). Is there/could there be a way to just show all the contents of a .env file in a dict without explicitly loading them? Or maybe I'm missing the correct usage of this?

Multiline values question

Hello,

I haven't noticed any explicit mentions of it but does environs support multiline values for env vars in .env?
For envparse, for example, I had to do nasty tricks to make it work (like use env_var='many\nlines\nconnected\nwith\nexplicit\nline\nseparator') which may be PITA if there's a lot of lines and/or special chars.

API is a bit strange

I know the title is a bit weird as well but bear with me, I'll explain where I'm coming from.

Instances of classes usually hold and manage state related to their name. The main class in this library is called Env. That suggests to me that an instance of Env holds and manages the state of the environment. Now, let's look at some code:

>>> from environs import Env
>>> import os
>>> env = Env()
>>> env('FOO')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/user/project/env/lib64/python3.7/site-packages/environs/__init__.py", line 74, in method
    raise EnvValidationError('Environment variable "{}" not set'.format(source_key), [message])
environs.EnvValidationError: Environment variable "FOO" not set
>>> os.environ['FOO'] = 'bar'
>>> env('FOO')
'bar'

To me that is quite unexpected. It looks like manipulating os.environ changes the state of env, that's not really nice behavior.

I do know that this is actually not how the library works. Env does not actually hold and manage any state related to the environment. The only state it holds is the parsing state, as in the variables it has parsed so far. Let's look at some more code (continuing where I left off):

>>> env.dump()
{'FOO': 'bar'}
>>> os.environ['FOO'] = 'lorem'
>>> env.dump()
{'FOO': 'bar'}
>>> env('FOO')
'lorem'
>>> env.dump()
{'FOO': 'lorem'}

dump seems to return all variables that have been parsed so far, including their values. But after changing the environment it still has the old value, quite unexpected again, it does keep some state. Only after parsing again, the new value shows up in the dump.

Another confusing behavior is that of read_env which does not suggest any side effects but it does change the current environment. Granted, it does not override anything but it's still not ideal behavior.

Some of the confusion can probably be fixed with better documentation.

However, the reason I started writing this is another one. I was hoping that this library could help me parse the environment of another system, specifically a Docker container. But while it has all the wonderful tools I can't do that without polluting my own environment, and read_env only accepts file system paths while the underlying library would happily accept file-like objects as well.

Add subcast for key in dict

Hello, I haven't found a clear explanation in the doc but from what I understand from the source code in _preprocess_dict, subcast parameter only impacts the value of the duct, not the key. It would be nice to have two different parameters for subcasting. Like subcast_key and subcast_value.

Happy to submit PR If you think this is a good improvement!

Typehint return values of type-cast methods

Motivation

In projects of large scale, it can be INFINITELY useful if given e.g.

SOME_VALUE = env.int("SOME_ENVVAR")

a type checker can deduce that SOME_VALUE is of type int.

Impediments

The above is currently very difficult to achieve because env.int (and most other type-cast methods, just using this as an example) can return types other than int, and in ways that are really hard for static code analysis to understand. The possible return types that I'm aware of are:

  1. int - The usual case.
  2. None - The case of env.int("NON_EXISTING_ENVVAR", default=None)
  3. str - When eager=False and the read envvar doesn't deserialize to an int
  4. marshmallow._Missing - When eager=False and the envvar is missing.

Type annotating the return type always as Union[int, None, str, marshmallow._Missing] is not useful at all, because it means whenever you are accessing that value, you will have to make asserts/casts to prove the type checker which one of the types the value actually is.

Optimally IMO we would only have cases 1 and 2 which should be quite straightforward to type annotate. We could tell the type checker that the return value is always int unless the value of default is None, in which case the return value is Union[int, None]. Static analysis should be able to understand this.

Cases 3 and 4 are a lot more difficult, because the eager=False declaration doesn't happen on the env.int() line so I'm not sure if there is syntax to make type checkers understand it.

Possible solutions

  1. Move eager=False functionality to a subclass that can be type annotated separately and less strictly.
  2. When using eager=False, make values of missing/invalid envvars a "null value" of the correct type. For instance make env.int("NON_EXISTING_ENVVAR") return a 0, or similarly make env.str failure cases return an empty string "". In these error cases the value is meaningless anyways, but making a value of correct type simplifies typing (making typing exactly the same as eager=True mode).
  3. Keep things as they are, and typehint according to eager=True mode only. Communicate in the docs that users of eager=False mode should never access read environment variables before env.seal() is called, or even after the call if it raises.
  4. Consider eager=False mode in its current form more important than type checking. Don't do anything here, but perhaps make a strictly typed fork.

import environs fails on Ubuntu 16.04.6 LTS with Python 3.5.2

user@2fcfee34b4bd:~$ virtualenv -p python3 venv
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/user/venv/bin/python3
Also creating executable in /home/user/venv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
user@2fcfee34b4bd:~$ . venv/bin/activate
(venv) user@2fcfee34b4bd:~$ pip install environs
Collecting environs
  Downloading environs-7.2.0-py2.py3-none-any.whl (11 kB)
Collecting python-dotenv
  Downloading python_dotenv-0.12.0-py2.py3-none-any.whl (17 kB)
Collecting marshmallow>=2.7.0
  Downloading marshmallow-3.5.0-py2.py3-none-any.whl (45 kB)
     |################################| 45 kB 257 kB/s
Installing collected packages: python-dotenv, marshmallow, environs
Successfully installed environs-7.2.0 marshmallow-3.5.0 python-dotenv-0.12.0
(venv) user@2fcfee34b4bd:~$ python
Python 3.5.2 (default, Oct  8 2019, 13:06:37)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import environs
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/venv/lib/python3.5/site-packages/environs/__init__.py", line 31, in <module>
    Subcast = typing.Union[typing.Type, typing.Callable[..., _T]]
  File "/usr/lib/python3.5/typing.py", line 552, in __getitem__
    dict(self.__dict__), parameters, _root=True)
  File "/usr/lib/python3.5/typing.py", line 512, in __new__
    for t2 in all_params - {t1} if not isinstance(t2, TypeVar)):
  File "/usr/lib/python3.5/typing.py", line 512, in <genexpr>
    for t2 in all_params - {t1} if not isinstance(t2, TypeVar)):
  File "/usr/lib/python3.5/typing.py", line 1077, in __subclasscheck__
    if super().__subclasscheck__(cls):
  File "/home/user/venv/lib/python3.5/abc.py", line 225, in __subclasscheck__
    for scls in cls.__subclasses__():
TypeError: descriptor '__subclasses__' of 'type' object needs an argument
>>>

Proxy env var to prefixed env vars

When I proxy a non-prefixed env var to a prefixed one and try to load them an error is thrown.

Example:

.env

FLASK_ENV=development
APP_ENV={{FLASK_ENV}}

settings.py

from environs import Env

env = Env()
env.read_env()


with env.prefixed('APP_'):
    ENV = env.str('ENV')

Output:

Traceback (most recent call last):
  File "settings.py", line 8, in <module>
    ENV = env.str('ENV')
  File "/.../venv/lib/python3.6/site-packages/environs.py", line 47, in method
    'Environment variable "{}" not set'.format(proxied_key or parsed_key)
environs.EnvError: Environment variable "FLASK_ENV" not set

Is it intentional to only allow proxying variables within the prefix scope?

Obviously there are many possible workarounds to fix the error, however it feels unintuitive to restrict proxying to a prefix scope.

Error when parsing empty list with subcast

Hello,

The env.list can't handle an empty list when using subcast
It raises an exception environs.EnvValidationError: Environment variable "LIST" invalid: {0: ['Not a valid integer.']}
This can be reproduced by e.g

    os.environ["LIST"] = ""
    env = Env()
    LIST = env.list("LIST", [], subcast=int)

If you don't use subcast it works just fine by e.g

    os.environ["LIST"] = ""
    env = Env()
    LIST = env.list("LIST", [])

Maybe one solution is if the env variable is falsy (empty or doesn't exist) env.list should return the default value if defined

Happy to submit a PR if that would be of interest. Thanks!

Integration with argparse

I have a use-case where I need to support both command-line arguments, as well as environment variables. This pattern ends up looking like this:

def parse_env():
    env = Env()
    return {
        'my_option': env.str('MY_OPTION', default='value'),
        'positional': env.str('POSITIONAL')
        'many_inputs': env.list('MANY_INPUTS')
    }

def parse_cli():
    parser = argparse.ArgumentParser()
    parser.add_argument('-m', '--my-option', default='value')
    parser.add_argument(dest='positional')
    parser.add_argument(dest='many_inputs', nargs='+')
    return vars(parser.parse_args())

if __name__ == '__main__':
    if os.environ.get('USE_ENV'):
        args = parse_env()
    else:
        args = parse_cli()

    do_something(**args)

Is there a better pattern for this? There's a lot of duplicate logic, making sure both variables are the same and defaults match. Could this be a feature request to add this to argparse, maybe with a subclass?

I'm thinking something like this:

from environs import EnvArgumentParser
parser = EnvArgumentParser(
    env_prefix='MY_PREFIX_'  # this would be neat
)
parser.add_argument('-m', '--my-option', default='value', env_name='MY_OPTION')
parser.add_argument(dest='positional', env_name='CUSTOM_NAME')
parser.add_argument(dest='many_inputs', nargs='+')  # default env_name to MANY_INPUTS

Thoughts? If you think this would be useful for this library, I can try working on it. I don't know the internals of argparse very well though.

Add env.database_url

Add a database URL parser method that validates that the input envvar is a valid database URL.

Error message within prefixed() context

The error message does not prepend the prefix to error message output, when an error occurs in an env.prefixed() context. The code below illustrates the issue. The environment variable is named TEST_VAR, but the error message is showing VAR.

######@shim-dev:~/workspace/shim (environ-adoption) 
$ export TEST_VAR=-1
######@shim-dev:~/workspace/shim (environ-adoption) 
$ ipython
Python 3.6.5 (default, Aug 15 2019, 19:51:45) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.7.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from environs import Env                                                                                                                                                                                                                                                     

In [2]: env = Env()                                                                                                                                                                                                                                                                  

In [3]: with env.prefixed("TEST_"): 
   ...:     invalid_test = env.int("VAR", validate=lambda n: n > 0) 
   ...:                                                                                                                                                                                                                                                                              
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
~/.pyenv/versions/3.6.5/lib/python3.6/site-packages/environs.py in method(self, name, default, subcast, **kwargs)
     55         try:
---> 56             value = field.deserialize(value)
     57         except ma.ValidationError as error:

~/.pyenv/versions/3.6.5/lib/python3.6/site-packages/marshmallow/fields.py in deserialize(self, value, attr, data)
    265         output = self._deserialize(value, attr, data)
--> 266         self._validate(output)
    267         return output

~/.pyenv/versions/3.6.5/lib/python3.6/site-packages/marshmallow/fields.py in _validate(self, value)
    205         if errors:
--> 206             raise ValidationError(errors, **kwargs)
    207 

ValidationError: ['Invalid value.']

The above exception was the direct cause of the following exception:

EnvError                                  Traceback (most recent call last)
<ipython-input-3-be1d17efe506> in <module>
      1 with env.prefixed("TEST_"):
----> 2     invalid_test = env.int("VAR", validate=lambda n: n > 0)
      3 

~/.pyenv/versions/3.6.5/lib/python3.6/site-packages/environs.py in method(self, name, default, subcast, **kwargs)
     56             value = field.deserialize(value)
     57         except ma.ValidationError as error:
---> 58             raise EnvError('Environment variable "{}" invalid: {}'.format(name, error.args[0])) from error
     59         else:
     60             self._values[parsed_key] = value

EnvError: Environment variable "VAR" invalid: ['Invalid value.']

In [4]:  

No module named 'environs'

Platform: WSL Ubuntu18.04
Python: Python3.8

~ cd Test
➜  Test  python3.8 -m venv .venv
➜  Test source .venv/bin/activate
(.venv) ➜  Test pip install environs
Collecting environs
  Using cached https://files.pythonhosted.org/packages/b3/4f/99d49bf40e46d86992b4dd5ceb6e23530c5a06fbc4322007a9b4810ad6c1/environs-7.3.1-py2.py3-none-any.whl
Collecting python-dotenv (from environs)
  Using cached https://files.pythonhosted.org/packages/7f/ee/e0cd2d8ba548e4c3e8c9e70d76e423b3e8b8e4eec351f51292d828c735d2/python_dotenv-0.12.0-py2.py3-none-any.whl
Collecting marshmallow>=2.7.0 (from environs)
  Using cached https://files.pythonhosted.org/packages/d8/0b/bb43d7610e71d87e8537025841c248471dbf938c676d32b8f94c82148c04/marshmallow-3.5.1-py2.py3-none-any.whl
Installing collected packages: python-dotenv, marshmallow, environs
Successfully installed environs-7.3.1 marshmallow-3.5.1 python-dotenv-0.12.0
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(.venv) ➜  Test  python
Python 3.8.2 (default, Feb 26 2020, 02:56:10)
[GCC 7.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import environs
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'environs'
>>>

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.