mozey / config Goto Github PK
View Code? Open in Web Editor NEWManage env vars with flat key/value files
License: MIT License
Manage env vars with flat key/value files
License: MIT License
Some projects might use the configu
command to generate the pkg/config
.
Making all users of a project install the command just to toggle env is a bit harsh.
This issue is about simplifying the bash script for toggling env, so it does not depend on the command, for example:
# Helper func to toggle env with github.com/mozey/config
conf() {
# Set environment variables from file
# https://stackoverflow.com/a/20909045/639133
UNAME_STR=$(uname)
if [ "$UNAME_STR" = 'Linux' ]; then
export "$(grep -v '^#' .env | xargs -d '\n')"
elif [ "$UNAME_STR" = 'FreeBSD' ] || [ "$UNAME_STR" = 'Darwin' ]; then
export "$(grep -v '^#' .env | xargs -0)"
fi
export APP_DIR=${APP_DIR}
# Print application env
printenv | sort | grep --color -E "APP_|AWS_"
}
TODO
Disadvantages
ardanlabs/conf: "Package conf provides support for using environmental variables and command line arguments for configuration"
See docs: "This package takes a struct value and parses it for both the environment and flags. It supports several tags to customize the flag options"
Make the env flag optional, e.g. by default use config.json
for simple projects
Create an example of using a base64 string, e.g.
// configBase64 must be compiled with ldflags,
// read config from env if this is not set
var configBase64 string
func main() {
// Config
if configBase64 != "" {
err := config.SetEnvBase64(configBase64)
if err != nil {
log.Error().Stack().Err(err).Msg("")
return os.Exit(1)
}
}
conf := config.New()
// ...
}
TODO Option to lock base64 values only, even if empty, i.e. don't read from env?
The --prefix
flag (with default APP_
) namespaces the env vars to avoid collisions.
For some variables overriding is the point, and the namespace must be optional, for example AWS_PROFILE
. There's already some code that deals with this specific case.
However, the following command...
configu -env dev -key AWS_PROFILE -value aws-local
...will error with
key for env dev must strart with prefix APP_
And this command...
configu -env dev -key AWS_PROFILE -value aws-local --prefix AWS_
...will error with
AWS_DIR env not set
Consider adding an --ignore-prefix
flag?
This cmd
./config -key APP_foO -value xxx
Should create the key APP_FOO
Refactor errors, all errors must be defined in pkg/cmdconfig/errors.go
Sometimes it's useful to pass env to command(s) inline. Without setting the env in the parent shell.
This is similar to -csv
but instead of comma the delimiter should be a space. Maybe allow passing in an optional -delim
flag?
See direnv/direnv: "augments existing shells with a new feature that can load and unload environment variables depending on the current directory"
Also see discussion: Zed should support direnv
Config files must have a flat structure, e.g. {KEY: VALUE}
. Both key and value must be strings.
This issue is for adding chain-able type conversion functions, without changing the existing API.
For example, given a config.dev.json
like this
{
"APP_A": "This is a string",
"APP_B": "123",
"APP_C": "true",
"APP_D": "123.45",
}
The existing API returns a string when an eponymous method for each key is called
conf, err = config.LoadFile("dev")
...
var s string
s = conf.A()
To parse a type, first call the conf.Fn{{Key}}
method to start a chain
var s string
s = conf.FnA().String()
var err error
var i64 int64
i64, err = conf.FnB().Int64()
var b bool
b, err = conf.FnC().Bool()
var f64 float64
f64, err = conf.FnD().Float64()
// etc...
In the examples above, err != nil
if the specified type could not be parsed.
Some keys (if they have the syntax PREFIX_TEMPLATE_
) use other keys (or parameters) to generate values, for example
APP_BUZ="Buzz"
APP_TEMPLATE_FIZ="Fizz{{.Buz}}"
Would generate this code
// ExecTemplateFiz fills APP_TEMPLATE_FIZ with given params
func (c *Config) ExecTemplateFiz() string {
t := template.Must(template.New("templateFiz").Parse(c.templateFiz))
b := bytes.Buffer{}
_ = t.Execute(&b, map[string]interface{}{
"Buz": c.buz,
})
return b.String()
}
Or, if no key is defined for APP_BUZ
, assume it is a parameter
// ExecTemplateFiz fills APP_TEMPLATE_FIZ with given params
func (c *Config) ExecTemplateFiz(buz string) string {
t := template.Must(template.New("templateFiz").Parse(c.templateFiz))
b := bytes.Buffer{}
_ = t.Execute(&b, map[string]interface{}{
"Buz": buz,
})
return b.String()
}
Review the toggling env, compare to solutions posted here:
Usually it's a good idea to have the same keys across config files. That makes it easier to diff the files. If the key doesn't apply to the env then use the empty value.
This cmd might be invoked like this
./config -all -key APP_FOO -value xxx
There should be a way for the ${GOPATH}/go/bin/configu
command that is installed with go install to print the tag of the release that was installed?
If there are no template properties, APP_TEMPLATE_*
, then the template.go
file must not be created
When compiling base64 config into a binary, it might be useful if the config is encrypted.
The decryption key can then be made available in a file, e.g.
# JSON
$APP_DIR/config-creds.json
# ENV
$APP_DIR/.env-creds
# etc
The structure of the creds file is the same as config files
// JSON
{
"APP_SECRET": "xxx"
}
# env
APP_SECRET=xxx
The benefit on doing this is that the binary can be distributed publicly, while the description key is distributed separately
Related issues
.env files are often used to record and set key value config. They support comments, and can be sourced without a special function to toggle env, for example. However, this method won't unset undefined variables matching the prefix (e.g. APP_
)
Syntax rules applying to .env
files:
When generating package code, error if any keys match a reserved prefix as per the key naming conventions
Money type using go-money
APP_DIR is required by configu
, doesn't make sense for it to unset this.
Originally the command assumed APP_DIR would be listed in config.ENV.json
, and set it in the config file if missing. That behavior was removed, causing this bug.
configu
commandMention this in the README
If config file exists, but is empty, the conf function errors with
command not found
Updating the config file with empty json {}
resolves the error
Return errors.WithStack(err)
instead of calling panic.
Configure zerolog console writer and use log.Error().Stack().Err(err)
Env vars starting with APP_TEMPLATE_* (see naming conventions) can be executed. This is usually done by calling the corresponding .ExecTemplate*
method on *config.Config
A flag for using the binary to do this would be useful in scripts, e.g.
configu -exec APP_TEMPLATE_X param1 param2
"The YAML 1.2 specification was published in 2009. Its primary focus was making YAML a strict superset of JSON." See 1.2.2 spec
YAML files take precedence, e.g.
config.dev.yaml
config.dev.json
This feature would be useful for adding comments to config files. Config files must still be flat key value pairs
Refactor to make type conversion functions the default, i.e.
// Search on this to find all code that references the APP_FOO env var
conf.Foo()
// Remove this, makes it difficult to find all usages of the var
conf.FnFoo()
// Then do this
conf.Foo().String()
conf.Foo().Int64()
// Instead of
conf.Foo() // Not a string, starts a chain
This is a breaking change since code must be refactored, but the config file format stays the same
Following instructions to easily rename your Git default branch from master to main
Rename
cd ${PRO_PATH}/config
git branch -m master main
git push -u origin main
Then change default branch on GitHub settings
Settings > General > Default branch > Switch to another branch > "main" > Update
Delete master branch
View all branches > Delete master
Delete origin/master in working copy
git fetch --all --prune
# Would make sense if `git fetch` updates HEAD but it doesn’t
# https://stackoverflow.com/a/29405808/639133
# Only run the following command if you got the message below:
# "refs/remotes/origin/HEAD has become dangling"
git remote set-head origin -a
For updating local clones, as per link above, only run the following commands
git checkout master
git branch -m master main
git fetch
git branch --unset-upstream
git branch -u origin/main
git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main
# Delete origin/master in working copy as above
The configu
cmd should not set or unset APP_DIR
. It has to be set by the user. Preferably from context, e.g. use value of $(pwd)
before executing the command
Similar to APP_TEMPLATE
, consider another special prefix: APP_X
(extension)
If there is a sub-dir path matching the value, then the config file in that dir will be included.
Keys in the extended config must not duplicate keys in the main config
Related issues
When adding new keys, or changing default values, it would be useful to set a value in all config files in APP_DIR
Probably requires the following changes
conf.bat
to toggle configThe all flag is not working with .env files, maybe an issue with the wildcard filter?
See this comment
As per this comment, create a GetEnv function on the config helper. It must return all env as map[string]string
spf13/cobra "is built on a structure of commands, arguments & flags"
Note that the key/value pairs are sorted before writing them back to the file.
To fix this would require keeping context for each pair? E.g. conf.Comments = map[string]string
Another option would be to disable sorting, and replace the pair in-place
Install with webi?
Also consider this: "our shell scripts begin and end with { and } and run in strict mode...", refactor conf.sh
accordingly?
Config is read from env config.New
(usually),
or a file config.LoadFile
(useful when testing),
or a ldflag variable config.SetEnvBase64
(useful when distributing single file executables).
The functions above set private variables on the config package. They can be changed by setter functions, but won't pick up changes made in the parent process. i.e. os.Getenv
is called once for each variable.
As per the wikipedia entry for environment variables: "An environment variable is a dynamic-named value that can affect the way running processes will behave on a computer"
It might be useful to create a Live
(or Refresh
) function? For example
// Live retrieves the new value of the environment variable if set
func (fn *Fn) Live() (value string) {
newValue, err := os.LookupEnv(fn.keyPrefix)
if err != nil {
return newValue
}
return fn.input
}
Pass in -base64
and optionally -env
flags to encode specified config file as a base64 string and print it
If the .env
file contains lines with empty values
APP_FOO=
APP_BAR=bar
The value is incorrectly set to the next line like this
APP_FOO=APP_BAR=bar
Workaround is to use quotes around an empty string
APP_FOO=""
APP_BAR=bar
However, it would be better if the code was aware of this, and behaved better. Fix by making the value regex check for empty string followed by a newline?
Create constructor funcs for untyped values, e.g. for ft.Bool
func BoolFromAny(val any) Bool {
// Use type switch with logic similar to Bool.UnmarshalJSON
// https://go.dev/tour/methods/16
}
Consider the logic for strings in Bool.UnmarshalJSON, it follows the definition for truthy described here, that is: "all values are truthy unless they are defined as falsy". The values for falsy in this packge do not correspond to JavaScript. Add "no", "disabled", and "off" as additional falsy values?
Also consider checkbox inputs: "a checkbox's value is only included in the submitted data if the checkbox is currently checked. If it is, then the value of the checkbox's value attribute is reported as the input's value, or on if no value is set"
For example, currently not specifying a value produces an error
configu -key APP_VERSION
...missing value for key APP_VERSION
Instead, rather print the value for the given key, e.g.
configu -key APP_VERSION
2021-01-01#abcdef
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.