koding/multiconfig

Load configuration from multiple sources in Go

Multiconfig

Load configuration from multiple sources. Multiconfig makes loading/parsing from different configuration sources an easy task. The problem with any app is that with time there are many options how to populate a set of configs. Multiconfig makes it easy by dynamically creating all necessary options. Checkout the example below to see it in action.

Features

Multiconfig is able to read configuration automatically based on the given struct's field names from the following sources:

• Struct tags
• TOML file
• JSON file
• YAML file
• Environment variables
• Flags

Install

go get github.com/koding/multiconfig

Usage and Examples

Lets define and struct that defines our configuration

type Server struct {
	Name    string `required:"true"`
	Port    int    `default:"6060"`
	Enabled bool
	Users   []string
}

Load the configuration into multiconfig:

// Create a new DefaultLoader without or with an initial config file
m := multiconfig.New()
m := multiconfig.NewWithPath("config.toml") // supports TOML, JSON and YAML

// Get an empty struct for your configuration
serverConf := new(Server)

// Populated the serverConf struct
err := m.Load(serverConf) // Check for error
m.MustLoad(serverConf)    // Panic's if there is any error

// Access now populated fields
serverConf.Port // by default 6060
serverConf.Name // "koding"

Run your app:

# Sets default values first which are defined in each field tag value. 
# Starts to load from config.toml
$ app

# Override any config easily with environment variables, environment variables
# are automatically generated in the form of STRUCTNAME_FIELDNAME
$ SERVER_PORT=4000 SERVER_NAME="koding" app

# Or pass via flag. Flags are also automatically generated based on the field
# name
$ app -port 4000 -users "gopher,koding"

# Print dynamically generated flags and environment variables:
$ app -help
Usage of app:
  -enabled=true: Change value of Enabled.
  -name=Koding: Change value of Name.
  -port=6060: Change value of Port.
  -users=[ankara istanbul]: Change value of Users.

Generated environment variables:
   SERVER_NAME
   SERVER_PORT
   SERVER_ENABLED
   SERVER_USERS

License

The MIT License (MIT) - see LICENSE for more details

multiconfig's Issues

Change -debug=true to -debug

Boolean vars are expecting an explicit true otherwise

What is the priority of loading values?

In which order does the lib load values? Does the config file override command line flag?

Add CamelCase support to TomlReader

Would be nice for consistency.

PS. Is this still maintained? We really like it but there's a few niggles here and there we'd like to change, like adding a name tag to make this kind of camelcase / snake_case be more easily configured via the config struct.

Add nested struct support

Nested structs field should be retrieved with an addionital _ for env's or - for flags. Example:

env:

CONFIG_POSTGRES_DBNAME
CONFIG_POSTGRES_URL

flag:

-postgres-dbname
-postgres-url

Not able to specify configuration file on command line

Unless I'm missing something obvious, I can't find a way to specify configuration file on command line like
app -conf config.toml

Using the flags package on top of multiconfig breaks the helpfile and produces a "flag provided but not defined".

Access control mechanism for tags

Default values for slices of embedded structs are not working

Similar to #13, but the problem is if you have a slice of a struct, the default values aren't getting applied. I took a stab at trying to figure out what's going on here, but didn't really get anywhere productive.

How to change flag usage string, documentation

Hi.. thanks for a great package. I didn't see how to change flag usage output. It seems hard wired as "Change value of ___" Is this an open question or did I just miss how to do it?

When working with glog,It gets flag provided but not defined: -alsologtostderr

Because of the flagSet.Parse(args) ignored the flags defined by glog,I added flag.VisitAll before it.

Works fine.

    flag.VisitAll(func(ff *flag.Flag) {
        f.flagSet.Var(ff.Value, ff.Name, ff.Usage)
    })

Test

Strange behavior when working with pointer field

Test code

package main

import "github.com/koding/multiconfig"
import "os"
import "time"

type duration struct {
    time.Duration
}

func (d *duration) UnmarshalText(text []byte) error {
    var err error
    d.Duration, err = time.ParseDuration(string(text))
    return err
}

type A struct {
    //B *duration `default:"10s"`

    // works as expected
    B *duration `toml:"b"`
}

func main() {
    a := &A{}
    multiconfig.MustLoadWithPath(os.Args[1], a)
    println(a.B.String())
}

When specify argument value through env or cli args for pointer type, will cause multiconfig compain "not support type: ptr"

Using "go test" flags causes multiconfig to exit with a non-zero status

The issue appears to be caused by "go test" passing its flags (with some reformatting, like -v turning into -test.v) to the test code and multiconfig is concerned because it is unaware of those flags.

I've pushed a branch with code that demonstrates the issue: https://github.com/four2five/gotest_multiconfig_example

validator support pointer to struct

required tag is not working if its related field is inside a struct which is referenced using pointer in parent struct.

type Parent struct {
    C *Child
}

type Child {
    SomeField `toml:"some_field" required:"true"`
}

Add tests for flagLoader

How could I use CamelCase?

It is hard to use,

    m := multiconfig.New()
    m.MustLoad(&config)
    l := multiconfig.EnvironmentLoader{
        CamelCase: true,
    }
    l.Load(&config)

Distinct behaviors of `-flag false` and `-flag=false`

For boolean flag values, there is what seems to me inconsistent behavior.

Reusing the example from the README:

package main

import (
    "fmt"
    "github.com/koding/multiconfig"
)

type Server struct {
    Enabled bool
}

func main() {
    m := multiconfig.New()
    serverConf := new(Server)
    m.MustLoad(serverConf)

    fmt.Println(serverConf.Enabled)
}

Here are a few runs of the app:

$ app         
false
$ app -enabled
true
$ app -enabled=true
true
$ app -enabled=false
false
$ app -enabled true
true
$ app -enabled false
true

I would expect the last one to be false.

README.md points to non-existent file

GOT

The MIT License (MIT) - see LICENSE.md for more details

WANT

The MIT License (MIT) - see [LICENSE](/LICENSE) for more details

Toml cannot be used map

#config.toml 
title       = "blabla"
[servers]
    [servers.a]
    ip      = "1.1.1.1"
    dc      = "abc"
    [servers.b]
    ip      = "2.2.2.2"
    dc      = "cba"


#main.go
type (
	WebSiteConfig struct {
		Title string
		Servers map[string]Server
	}
)

type Server struct {
		IP string
		DC string
}

func Load() *WebSiteConfig {
	config := new(WebSiteConfig)
	multiconfig.NewWithPath(LogPath). MustLoad(config)

	return config
}


#error 
panic: not struct

goroutine 1 [running]:
github.com/fatih/structs.strctVal(0xc0001c4500, 0xc0001b88f8, 0x16, 0xc0001c4500, 0xc0001b88f8)
        /Users/Xwei/go/src/github.com/fatih/structs/structs.go:437 +0x12f
github.com/fatih/structs.(*Field).FieldOk(0xc0002122d0, 0xc0002143c2, 0x1, 0xc0001ddc80, 0xc0001caa10)
        /Users/Xwei/go/src/github.com/fatih/structs/field.go:129 +0x98
github.com/fatih/structs.(*Field).Field(0xc0002122d0, 0xc0002143c2, 0x1, 0x157ad93)
        /Users/Xwei/go/src/github.com/fatih/structs/field.go:109 +0x3f
github.com/koding/multiconfig.(*EnvironmentLoader).processField(0xc0001c02a0, 0x1596ef3, 0xd, 0xc0002122d0, 0x157ad93, 0x7, 0x15fa320, 0xc0001b93b0, 0x0, 0x0)
        /Users/Xwei/go/src/github.com/koding/multiconfig/env.go:61 +0x164
github.com/koding/multiconfig.(*EnvironmentLoader).Load(0xc0001c02a0, 0x15c80e0, 0xc0001b88d0, 0x0, 0x0)

is it right? thanks

code:
func main() {
var cfg = struct {
Name string ``
Age int `default:"30"`
}{"Lily", 29}
multiconfig.MustLoadWithPath("config.toml", &cfg)
fmt.Println(cfg)
}
config:
name = "Tom"`

---

the result is "{Tom 30}"，I expected cfg's Age is 29, I think cfg's Age will be 30 when not initialized

env usage help prints env vars in random order

from env.go

iteration over a map is random. Need to copy into the []string, sort, then then use that for range.

// PrintEnvs prints the generated environment variables to the std out.
func (e *EnvironmentLoader) PrintEnvs(s interface{}) {
    strct := structs.New(s)
    strctMap := strct.Map()
    prefix := e.getPrefix(strct)

    for key, val := range strctMap {
        field := strct.Field(key)

        e.printField(prefix, field, key, val)
    }
}

Validator does not return a list of validation errors but returns first validation failure.

Validator returns first error during validation process, It would be nice to have all the validation errors to be returned at once.

The error does not return if the file name does not end on json, yaml etc.

configLoader := multiconfig.NewWithPath("1111")
err := configLoader.Load(c)

Add `default value` support for tags

Default values for embedded structs are not working

bug : flag: help requested

code:
m := multiconfig.NewWithPath(configPath)
config := new(common.Configs)
m.MustLoad(config)

app -help
Usage of app:
...
...
flag: help requested

This panic arises func (d *DefaultLoader) MustLoad(conf interface{})

Want to know why

Accessing non-option arguments

How do I access non-option command line arguments?

For example, running ./app -opt foo bar baz won't complain about bar and baz, but I also don't see any methods in multiconfig giving me access to these. flag.Args() doesn't work (acts as if flag hasn't been used); os.Args (expectedly) yields the entire command line.

-help panicking with Go 1.7

After upgrading to Go 1.7 (and 1.7.1), -help gives a panic:

$ brew switch go 1.6.3
Cleaning /usr/local/Cellar/go/1.6.3
Cleaning /usr/local/Cellar/go/1.7
Cleaning /usr/local/Cellar/go/1.7.1
3 links created for /usr/local/Cellar/go/1.6.3
$ go build
$ ./gosonic -help
Usage of ./gosonic:
  -dbpath
        Change value of DbPath. (default ./devDb)
  -disabledownsampling
        Change value of DisableDownsampling.
  -disablevalidation
        Change value of DisableValidation. (default true)
...

Generated environment variables:
   GOSONIC_PORT
   GOSONIC_MUSICFOLDER
   GOSONIC_DBPATH
...

$ brew switch go 1.7.1
Cleaning /usr/local/Cellar/go/1.6.3
Cleaning /usr/local/Cellar/go/1.7
Cleaning /usr/local/Cellar/go/1.7.1
3 links created for /usr/local/Cellar/go/1.7.1
$ go build
$ ./gosonic -help
Usage of ./gosonic:
panic: reflect: call of reflect.Value.Interface on zero Value

goroutine 1 [running]:
panic(0x5c8280, 0xc4202c0360)
    /usr/local/Cellar/go/1.7.1/libexec/src/runtime/panic.go:500 +0x1a1
reflect.valueInterface(0x0, 0x0, 0x0, 0x28744d88401, 0x0, 0xd6e84)
    /usr/local/Cellar/go/1.7.1/libexec/src/reflect/value.go:912 +0x204
reflect.Value.Interface(0x0, 0x0, 0x0, 0x11bcb, 0x6089a0)
    /usr/local/Cellar/go/1.7.1/libexec/src/reflect/value.go:907 +0x44
github.com/deluan/gosonic/vendor/github.com/fatih/structs.(*Field).Value(0xc4202d0360, 0xf547, 0x5dc700)
    /Users/deluan/Development/go/src/github.com/deluan/gosonic/vendor/github.com/fatih/structs/field.go:31 +0x40
github.com/deluan/gosonic/vendor/github.com/koding/multiconfig.(*fieldValue).String(0xc4202d0360, 0x5ef2a0, 0xc4202d0360)
    /Users/deluan/Development/go/src/github.com/deluan/gosonic/vendor/github.com/koding/multiconfig/flag.go:151 +0x2f
flag.isZeroValue(0xc4202ab180, 0xc420297f00, 0x7, 0xc4202c0180)
    /usr/local/Cellar/go/1.7.1/libexec/src/flag/flag.go:393 +0x12d
flag.(*FlagSet).PrintDefaults.func1(0xc4202ab180)
    /usr/local/Cellar/go/1.7.1/libexec/src/flag/flag.go:467 +0x1d5
flag.(*FlagSet).VisitAll(0xc4202bcea0, 0xc420053ab8)
    /usr/local/Cellar/go/1.7.1/libexec/src/flag/flag.go:323 +0x67
flag.(*FlagSet).PrintDefaults(0xc4202bcea0)
    /usr/local/Cellar/go/1.7.1/libexec/src/flag/flag.go:476 +0x46
github.com/deluan/gosonic/vendor/github.com/koding/multiconfig.(*FlagLoader).Load.func1()
    /Users/deluan/Development/go/src/github.com/deluan/gosonic/vendor/github.com/koding/multiconfig/flag.go:67 +0x118
flag.(*FlagSet).usage(0xc4202bcea0)
    /usr/local/Cellar/go/1.7.1/libexec/src/flag/flag.go:828 +0x5c
flag.(*FlagSet).parseOne(0xc4202bcea0, 0x617be0, 0x1, 0xc4202d40f0)
    /usr/local/Cellar/go/1.7.1/libexec/src/flag/flag.go:870 +0xb4c
flag.(*FlagSet).Parse(0xc4202bcea0, 0xc420094070, 0x1, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.7.1/libexec/src/flag/flag.go:915 +0x60
github.com/deluan/gosonic/vendor/github.com/koding/multiconfig.(*FlagLoader).Load(0xc4202bb900, 0x584160, 0xc420121810, 0xc420139ee0, 0x0)
    /Users/deluan/Development/go/src/github.com/deluan/gosonic/vendor/github.com/koding/multiconfig/flag.go:82 +0x25f
github.com/deluan/gosonic/conf.LoadFromFlags()
    /Users/deluan/Development/go/src/github.com/deluan/gosonic/conf/configuration.go:33 +0x79
main.main()
    /Users/deluan/Development/go/src/github.com/deluan/gosonic/main.go:15 +0x2b

For reference, my code is available here: https://github.com/deluan/gosonic/blob/master/conf/configuration.go

[]string serializing seems not to work.

Implement config validation mechanism

Embedding with environment variables?

Is this the expected behaviour when using embedded struct?

I was hoping to get CONFIG_PASSWORD and CONFIG_USERNAME.
If i try using EnvironmentLoader with Prefix and EnvPrefix i only get the prefix on the most outer config.

Generated environment variables:
   CONFIG_PARENTCONFIG_PASSWORD
   CONFIG_USERNAME

example program:

package main

import (
    "log"

    "github.com/koding/multiconfig"
)

type ParentConfig struct {
    Password string
}

type Config struct {
    ParentConfig
    Username string
}

func main() {
    conf := &Config{}
    multiconfig.MustLoad(conf)

    log.Println(conf)
}

Comma separated lists in YAML

Hi

I've just discovered a minor flag in the YAML implementation.

In my yaml file, I have:

somekey: a,b,c,d,e,f

and in my struct, I have:

type test struct {
SomeKey []string
}

The YAML is officially valid, but causes a parse error

add full support for time.Duration

Support for time.Duration was added in #28, but its usage is inconsistent between
the different loaders. While it is possible to specify a duration as 10s as an ENV variable,
the same is not possible e.g. for TOML files. Here, you'd have to set the same duration as 10000000000.

Output help message

Currently we only output the dynamically generated flags and environments if something goes wrong. There is no manual way of outputing the flags. For example if we only use the env loader, there is again no way to output the help message, because currently the help message is only defined inside the flag load.

There a two ways to do it.
#1. Create a new interface

We create a new interface which every loader could implement:

type Helper interface {
   Help() string
}

If this is implemented via a loader, we'll have another function that will print those:

func PrintHelp(helper ...Helper) {}
func PrintHelpWriter(w io.Writer, helper ...Helper) {}

//usage
multiconfig.PrintHelp(flagLoader, envLoader)

This is totally optional, but at least it would imense helpful as we could generate those dynamically, as we do for envs, flags, etc...
#2. Extend the Loader interface

This is will add a new method to the existing Loader interface:

type Loader interface {
    Load(s interface{}) error
    Help() string
}

After that, if loader.Load(s) fails it will output the help messages automatically (because the loader has access to the Help() method now. Or we could manually invoke to output the help messages.

Opinions ?

panic: not struct

@cihangir commit 15a9baf seems to have introduced a bug.

panic: not struct

goroutine 1 [running]:
panic(0x8ff0a0, 0xc820202b90)
    /usr/lib/go/src/runtime/panic.go:481 +0x3e6
github.com/fatih/structs.strctVal(0x8d8520, 0xc8201cb3b0, 0x0, 0x0, 0x0)
    /home/jonaz/go/src/github.com/fatih/structs/structs.go:439 +0x138
github.com/fatih/structs.(*Field).FieldOk(0xc8201dc6c0, 0xc82020295c, 0x4, 0x3, 0xa0a840)
    /home/jonaz/go/src/github.com/fatih/structs/field.go:129 +0x107
github.com/fatih/structs.(*Field).Field(0xc8201dc6c0, 0xc82020295c, 0x4, 0xad1120)
    /home/jonaz/go/src/github.com/fatih/structs/field.go:109 +0x39
github.com/koding/multiconfig.(*EnvironmentLoader).processField(0xc8201fe880, 0xadd680, 0xc, 0xc8201dc6c0, 0xad1120, 0xa, 0x8fdba0, 0xc8201d2090, 0x0, 0x0)
    /home/jonaz/go/src/github.com/koding/multiconfig/env.go:61 +0x1da
github.com/koding/multiconfig.(*EnvironmentLoader).Load(0xc8201fe880, 0x8d2160, 0xc8201cb3b0, 0x0, 0x0)
    /home/jonaz/go/src/github.com/koding/multiconfig/env.go:45 +0x1bd
github.com/koding/multiconfig.multiLoader.Load(0xc8201f2fc0, 0x4, 0x4, 0x8d2160, 0xc8201cb3b0, 0x0, 0x0)
    /home/jonaz/go/src/github.com/koding/multiconfig/multiloader.go:14 +0xc6
github.com/koding/multiconfig.(*multiLoader).Load(0xc8201fe8a0, 0x8d2160, 0xc8201cb3b0, 0x0, 0x0)
    <autogenerated>:7 +0xc1
github.com/stampzilla/stampzilla-go/nodes/stampzilla-server/notifications.(*router).Load(0xc8201d20f0, 0xb3b9a0, 0x12, 0x0, 0x0)
    /home/jonaz/go/src/github.com/stampzilla/stampzilla-go/nodes/stampzilla-server/notifications/router.go:49 +0x210

multiconfig is used here:
https://github.com/stampzilla/stampzilla-go/blob/master/nodes/stampzilla-server/notifications/router.go#L49

If you need more than that please say and i'll put together a runnable example.

DefaultLoader/FlagLoader is unusable

A common case for production is you need to use a command line parameter to specify the path of config file, then read and parse it. But that parameter option is apparently not included in FlagLoader's FlagSet, which will result in

flag provided but not defined

error.

Need to remove it from DefaultLoader.