The config from mozey

Simplify toggling env

Some projects might use the configu command to generate the pkg/config.

Making all users of a project install the command just to toggle env is a bit harsh.

This issue is about simplifying the bash script for toggling env, so it does not depend on the command, for example:

# Helper func to toggle env with github.com/mozey/config
conf() {
	# Set environment variables from file
	# https://stackoverflow.com/a/20909045/639133
	UNAME_STR=$(uname)
	if [ "$UNAME_STR" = 'Linux' ]; then
		export "$(grep -v '^#' .env | xargs -d '\n')"
	elif [ "$UNAME_STR" = 'FreeBSD' ] || [ "$UNAME_STR" = 'Darwin' ]; then
		export "$(grep -v '^#' .env | xargs -0)"
	fi

	export APP_DIR=${APP_DIR}

	# Print application env
	printenv | sort | grep --color -E "APP_|AWS_"
}

TODO

Error handling in case ".env" file does not exist
Support for ".json" files?
Default env is dev, first arg overrides

Disadvantages

Does not unset keys that have been removed from the config file

Comparison with ardanlabs/conf

ardanlabs/conf: "Package conf provides support for using environmental variables and command line arguments for configuration"

See docs: "This package takes a struct value and parses it for both the environment and flags. It supports several tags to customize the flag options"

Make env flag optional

Make the env flag optional, e.g. by default use config.json for simple projects

Base64 example

Create an example of using a base64 string, e.g.

// configBase64 must be compiled with ldflags,
// read config from env if this is not set
var configBase64 string

func main() {
	// Config
	if configBase64 != "" {
		err := config.SetEnvBase64(configBase64)
		if err != nil {
			log.Error().Stack().Err(err).Msg("")
			return os.Exit(1)
		}
	}
	conf := config.New()

	// ...
}

TODO Option to lock base64 values only, even if empty, i.e. don't read from env?

Flexible prefix

The --prefix flag (with default APP_) namespaces the env vars to avoid collisions.

For some variables overriding is the point, and the namespace must be optional, for example AWS_PROFILE. There's already some code that deals with this specific case.

However, the following command...

configu -env dev -key AWS_PROFILE -value aws-local

...will error with

key for env dev must strart with prefix APP_

And this command...

configu -env dev -key AWS_PROFILE -value aws-local --prefix AWS_

...will error with

AWS_DIR env not set

Consider adding an --ignore-prefix flag?

Common cause for errors

Refactor errors, all errors must be defined in pkg/cmdconfig/errors.go

Optionally specify separator for use with csv flag

Sometimes it's useful to pass env to command(s) inline. Without setting the env in the parent shell.

This is similar to -csv but instead of comma the delimiter should be a space. Maybe allow passing in an optional -delim flag?

Comparison with direnv

See direnv/direnv: "augments existing shells with a new feature that can load and unload environment variables depending on the current directory"

Also see discussion: Zed should support direnv

Type conversion functions

Config files must have a flat structure, e.g. {KEY: VALUE}. Both key and value must be strings.

This issue is for adding chain-able type conversion functions, without changing the existing API.

For example, given a config.dev.json like this

{
  "APP_A": "This is a string",
  "APP_B": "123",
  "APP_C": "true",
  "APP_D": "123.45",
}

The existing API returns a string when an eponymous method for each key is called

conf, err = config.LoadFile("dev")
...
var s string
s = conf.A()

To parse a type, first call the conf.Fn{{Key}} method to start a chain

var s string
s = conf.FnA().String()

var err error
var i64 int64
i64, err = conf.FnB().Int64()

var b bool
b, err = conf.FnC().Bool()

var f64 float64
f64, err = conf.FnD().Float64()
// etc...

In the examples above, err != nil if the specified type could not be parsed.

Generate template.go

Some keys (if they have the syntax PREFIX_TEMPLATE_) use other keys (or parameters) to generate values, for example

APP_BUZ="Buzz"
APP_TEMPLATE_FIZ="Fizz{{.Buz}}"

Would generate this code

// ExecTemplateFiz fills APP_TEMPLATE_FIZ with given params
func (c *Config) ExecTemplateFiz() string {
	t := template.Must(template.New("templateFiz").Parse(c.templateFiz))
	b := bytes.Buffer{}
	_ = t.Execute(&b, map[string]interface{}{
		"Buz": c.buz,
	})
	return b.String()
}

Or, if no key is defined for APP_BUZ, assume it is a parameter

// ExecTemplateFiz fills APP_TEMPLATE_FIZ with given params
func (c *Config) ExecTemplateFiz(buz string) string {
	t := template.Must(template.New("templateFiz").Parse(c.templateFiz))
	b := bytes.Buffer{}
	_ = t.Execute(&b, map[string]interface{}{
		"Buz": buz,
	})
	return b.String()
}

Review toggle func

Review the toggling env, compare to solutions posted here:

Cmd to set key value in all config files

Usually it's a good idea to have the same keys across config files. That makes it easier to diff the files. If the key doesn't apply to the env then use the empty value.

This cmd might be invoked like this

./config -all -key APP_FOO -value xxx

Flag to print version

There should be a way for the ${GOPATH}/go/bin/configu command that is installed with go install to print the tag of the release that was installed?

No code in template.go

If there are no template properties, APP_TEMPLATE_*, then the template.go file must not be created

Simpler test lib

~~The quicktest package is leaner than testify (less dependencies) with mostly the same functionality?~~

Encrypted config

When compiling base64 config into a binary, it might be useful if the config is encrypted.

The decryption key can then be made available in a file, e.g.

# JSON
$APP_DIR/config-creds.json
# ENV
$APP_DIR/.env-creds
# etc

The structure of the creds file is the same as config files

// JSON
{
  "APP_SECRET": "xxx"
}

# env
APP_SECRET=xxx

The benefit on doing this is that the binary can be distributed publicly, while the description key is distributed separately

Related issues

#28

.env files are often used to record and set key value config. They support comments, and can be sourced without a special function to toggle env, for example. However, this method won't unset undefined variables matching the prefix (e.g. APP_)

Syntax rules applying to .env files:

Each line in an env file must be in VAR=VAL format.
Lines beginning with # are processed as comments and ignored.
Blank lines are ignored.
There is no special handling of quotation marks. This means that they are part of the VAL.
Lines not matching the VAR=VAL are ignored

Error on reserved prefix when generating package code

When generating package code, error if any keys match a reserved prefix as per the key naming conventions

Money type

Money type using go-money

Don't print command to unset APP_DIR

APP_DIR is required by configu, doesn't make sense for it to unset this.

Originally the command assumed APP_DIR would be listed in config.ENV.json, and set it in the config file if missing. That behavior was removed, causing this bug.

APP_DIR must not be listed in the config file. You might want to move the project folder, and that should not make the config file invalid
APP_DIR must be set by the toggle helper script to the current directory. This implies you have to be in the same dir as the config file when running the configu command

Notes re. twelve factor apps

Mention this in the README

Error when config file is blank

If config file exists, but is empty, the conf function errors with

command not found

Updating the config file with empty json {} resolves the error

Return all errors

Return errors.WithStack(err) instead of calling panic.

Configure zerolog console writer and use log.Error().Stack().Err(err)

Flag to execute template var

Env vars starting with APP_TEMPLATE_* (see naming conventions) can be executed. This is usually done by calling the corresponding .ExecTemplate* method on *config.Config

A flag for using the binary to do this would be useful in scripts, e.g.

configu -exec APP_TEMPLATE_X param1 param2

YAML Support

"The YAML 1.2 specification was published in 2009. Its primary focus was making YAML a strict superset of JSON." See 1.2.2 spec

~~YAML files take precedence, e.g.~~

Try to read config.dev.yaml

If above doesn't exist, try to read config.dev.json

~~Error if neither file exists~~

This feature would be useful for adding comments to config files. Config files must still be flat key value pairs

Type conversion functions by default

Refactor to make type conversion functions the default, i.e.

// Search on this to find all code that references the APP_FOO env var
conf.Foo() 

// Remove this, makes it difficult to find all usages of the var
conf.FnFoo() 

// Then do this 
conf.Foo().String()
conf.Foo().Int64()

// Instead of
conf.Foo() // Not a string, starts a chain

This is a breaking change since code must be refactored, but the config file format stays the same

Rename master branch to main

Following instructions to easily rename your Git default branch from master to main

Rename

cd ${PRO_PATH}/config
git branch -m master main
git push -u origin main

Then change default branch on GitHub settings

Settings > General > Default branch > Switch to another branch > "main" > Update

Delete master branch

View all branches > Delete master

Delete origin/master in working copy

git fetch --all --prune

# Would make sense if `git fetch` updates HEAD but it doesn’t
# https://stackoverflow.com/a/29405808/639133
# Only run the following command if you got the message below:
# "refs/remotes/origin/HEAD has become dangling"
git remote set-head origin -a

For updating local clones, as per link above, only run the following commands

git checkout master
git branch -m master main
git fetch
git branch --unset-upstream
git branch -u origin/main
git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main
# Delete origin/master in working copy as above

Breaking changes for v1

How does go-oauth2/oauth2 import from "/v4", it's a tag not a branch?

List of changes

#44
etc?

User must set APP_DIR

The configu cmd should not set or unset APP_DIR. It has to be set by the user. Preferably from context, e.g. use value of $(pwd) before executing the command

Extended config

~~Similar to APP_TEMPLATE, consider another special prefix: APP_X (extension)~~

~~If there is a sub-dir path matching the value, then the config file in that dir will be included.~~

Keys in the extended config must not duplicate keys in the main config

Related issues

#37

Set value for all .config..json

When adding new keys, or changing default values, it would be useful to set a value in all config files in APP_DIR

Support Windows

Probably requires the following changes

Print Windows commands if OS detected
conf.bat to toggle config

All flag wildcard bug

The all flag is not working with .env files, maybe an issue with the wildcard filter?

Investigate regex replace performance vs text/template

See this comment

GetEnv func for config helper

As per this comment, create a GetEnv function on the config helper. It must return all env as map[string]string

Windows permission bug

On Windows, when setting a key value, for example

configu -key APP_SOMETHING -value xxx

The config file config.dev.json has the Read-only attribute set

Use spf13/cobra instead of flag standard library

spf13/cobra "is built on a structure of commands, arguments & flags"

Set value strips comments from .env files

Note that the key/value pairs are sorted before writing them back to the file.

To fix this would require keeping context for each pair? E.g. conf.Comments = map[string]string

Another option would be to disable sorting, and replace the pair in-place

Webi installer

Install with webi?

Also consider this: "our shell scripts begin and end with { and } and run in strict mode...", refactor conf.sh accordingly?

Lookup current value of an environment variable

Config is read from env config.New (usually),
or a file config.LoadFile (useful when testing),
or a ldflag variable config.SetEnvBase64 (useful when distributing single file executables).

The functions above set private variables on the config package. They can be changed by setter functions, but won't pick up changes made in the parent process. i.e. os.Getenv is called once for each variable.

As per the wikipedia entry for environment variables: "An environment variable is a dynamic-named value that can affect the way running processes will behave on a computer"

It might be useful to create a Live (or Refresh) function? For example

// Live retrieves the new value of the environment variable if set
func (fn *Fn) Live() (value string) {
  newValue, err := os.LookupEnv(fn.keyPrefix)
  if err != nil {
    return newValue
  }
  return fn.input
}

CMD Flag to encode config file as base64

Pass in -base64 and optionally -env flags to encode specified config file as a base64 string and print it

Empty value bug with .env files

If the .env file contains lines with empty values

APP_FOO=
APP_BAR=bar

The value is incorrectly set to the next line like this

APP_FOO=APP_BAR=bar

Workaround is to use quotes around an empty string

APP_FOO=""
APP_BAR=bar

However, it would be better if the code was aware of this, and behaved better. Fix by making the value regex check for empty string followed by a newline?

Constructor funcs for untyped values

Create constructor funcs for untyped values, e.g. for ft.Bool

func BoolFromAny(val any) Bool {
  // Use type switch with logic similar to Bool.UnmarshalJSON
  // https://go.dev/tour/methods/16
}

Consider the logic for strings in Bool.UnmarshalJSON, it follows the definition for truthy described here, that is: "all values are truthy unless they are defined as falsy". The values for falsy in this packge do not correspond to JavaScript. Add "no", "disabled", and "off" as additional falsy values?

Also consider checkbox inputs: "a checkbox's value is only included in the submitted data if the checkbox is currently checked. If it is, then the value of the checkbox's value attribute is reported as the input's value, or on if no value is set"

Get flag to print value for given key

For example, currently not specifying a value produces an error

configu -key APP_VERSION
...missing value for key APP_VERSION

Instead, rather print the value for the given key, e.g.

configu -key APP_VERSION
2021-01-01#abcdef

mozey / config Goto Github PK

config's People

Contributors

Watchers

Forkers

config's Issues

Recommend Projects

Recommend Topics

Recommend Org