containous/staert

Merge your configuration sources

Stært

Stært is a Go library for loading and merging a program configuration structure from many sources.

Overview

Stært was born in order to merge two sources of configuration (Flæg, TOML). Now it also supports Key-Value Store.

We developed Flæg and Stært in order to simplify configuration maintenance on Træfik.

Features

• Load your configuration structure from many sources
• Keep your configuration structure values unchanged if no overwriting (support defaults values)
• Three native sources :
  • Command line arguments using Flæg package
  • TOML config file using TOML package
  • Key-Value Store using libkv and mapstructure packages
• An interface to add your own sources
• Handle pointers field :
  • You can give a structure of default values for pointers
  • Same comportment as Flæg
• Stært is command oriented
  • It use flaeg.Command
  • Same comportment as Flæg commands
  • Stært supports only one command (the root-command)
  • Flæg allows you to use many commands
  • Only Flæg will be used if a sub-command is called. (because the configuration type could be different from one command to another)
  • You can add meta-data "parseAllSources" -> "true" to a sub-command if you want to parse all sources (it requires the same configuration type on the sub-command and the root-command)

Getting Started

The configuration

It works on your own configuration structure, like this one :

package example

import (
	"fmt"
	"github.com/containous/flaeg"
	"github.com/containous/staert"
	"os"
)

// Configuration is a struct which contains all different type to field
type Configuration struct {
	IntField     int                      `description:"An integer field"`
	StringField  string                   `description:"A string field"`
	PointerField *PointerSubConfiguration `description:"A pointer field"`
}

// PointerSubConfiguration is a SubStructure Configuration
type PointerSubConfiguration struct {
	BoolField  bool    `description:"A boolean field"`
	FloatField float64 `description:"A float field"`
}

Let's initialize it:

func main() {
	// Init with default value
	config := &Configuration{
		IntField:    1,
		StringField: "init",
		PointerField: &PointerSubConfiguration{
			FloatField: 1.1,
		},
	}
	// Set default pointers value
	defaultPointersConfig := &Configuration{
		PointerField: &PointerSubConfiguration{
			BoolField:  true,
			FloatField: 99.99,
		},
	}
	//...
}

The command

Stært uses flaeg.Command structure, like this:

// Create command
command := &flaeg.Command{
	Name:"example",
	Description:"This is an example of description",
	Config:config,
	DefaultPointersConfig:defaultPointersConfig,
	Run: func() error {
		fmt.Printf("Run example with the config :\n%+v\n", config)
		fmt.Printf("PointerField contains:%+v\n", config.PointerField)
		return nil
	}
}

Use stært with sources

Initialize Stært:

s := staert.NewStaert(command)

Initialize TOML source:

toml := staert.NewTomlSource("example", []string{"./toml/", "/any/other/path"})

Initialize Flæg source:

f := flaeg.New(command, os.Args[1:])

Add sources

Add TOML and Flæg sources:

s.AddSource(toml)
s.AddSource(f)

NB: You can change order, so that, Flæg configuration will overwrite TOML one.

Load your configuration

Just call LoadConfig function:

loadedConfig, err := s.LoadConfig();
if err != nil {
	// oops
}
// do what you want with `loadedConfig`
// or call run function

You can call Run

Run function will call run() from the command:

if err := s.Run(); err != nil {
	//OOPS
}

NB: If you didn't call LoadConfig() before, your function run() will use your original configuration.

Let's run example

TOML file ./toml/example.toml:

IntField = 2
[PointerField]

We can run the example program using following CLI arguments:

$ ./example --stringfield=owerwrittenFromFlag --pointerfield.floatfield=55.55
Run example with the config :
&{IntField:2 StringField:owerwrittenFromFlag PointerField:0xc82000ec80}
PointerField contains:&{BoolField:true FloatField:55.55}

Full example

Tagoæl is a trivial example which shows how Stært can be use. This funny GoLang program takes its configuration from both TOML and Flæg sources to display messages.

$ ./tagoael -h
tagoæl is an enhanced Hello World program to display messages with
an advanced configuration mechanism provided by Flæg & Stært.

flæg:   https://github.com/containous/flaeg
stært:  https://github.com/containous/staert
tagoæl: https://github.com/debovema/tagoael


Usage: tagoael [--flag=flag_argument] [-f[flag_argument]] ...     set flag_argument to flag(s)
   or: tagoael [--flag[=true|false| ]] [-f[true|false| ]] ...     set true/false to boolean flag(s)

Flags:
        -c, --commandlineoverridesconfigfile               Whether configuration from command line overrides configuration from configuration file or not. (default "true")
        --configfile                                       Configuration file to use (TOML). (default "tagoael")
        -i, --displayindex                                 Whether to display index of each message (default "false")
        -m, --messagetodisplay                             Message to display (default "HELLO WOLRD")
        -n, --numbertodisplay                              Number of messages to display (default "1000")
        -h, --help                                         Print Help (this message) and exit

Thank you @debovema for this work :)

KvStore

As with Flæg and TOML sources, the configuration structure can be loaded from a Key-Value Store. The package libkv provides connection to many KV Store like Consul, Etcd or Zookeeper.

The whole configuration structure is stored, using architecture like this pattern:

• Key: <prefix1>/<prefix2>/.../<fieldNameLevel1>/<fieldNameLevel2>/.../<fieldName>
• Value: <value>

It handles:

• All mapstructure features(bool, int, ... , Squashed Embedded Sub struct, Pointer).
• Maps with pattern : .../<MapFieldName>/<mapKey> -> <mapValue> (Struct as key not supported)
• Slices (and Arrays) with pattern : .../<SliceFieldName>/<SliceIndex> -> <value>

Note: Hopefully, we provide the function StoreConfig to store your configuration structure ;)

KvSource

KvSource implements Source:

type KvSource struct {
	store.Store
	Prefix string // like this "prefix" (without the /)
}

Initialize

It can be initialized like this:

kv, err := staert.NewKvSource(backend store.Backend, addrs []string, options *store.Config, prefix string)

LoadConfig

You can directly load data from the KV Store into the config structure (given by reference)

config := &ConfigStruct{} // Here your configuration structure by reference
err := kv.Parse(config)
// do what you want with `config`

Add to Stært sources

You can add this source to Stært, as with other sources:

s.AddSource(kv)

StoreConfig

You can also store your whole configuration structure into the KV Store:

// We assume that `config` is initialized
err := kv.StoreConfig(config)

Contributing

1. Fork it!
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -am 'Add some feature'
4. Push to the branch: git push origin my-new-feature
5. Submit a pull request :D

staert's Issues

[feature request] Want to parse TOML when sub-command called

We assume that rootCmd and subCmd are two *flaeg.Command with same Config struct.

//init source flaeg with rootCmd as root command
f := flaeg.New(rootCmd, os.Args[1:])
//add subcommand subCmd
f.AddCommand(subCmd)

//init toml source
toml := staert.NewTomlSource("configfile", []string{"."})

//init stært with rootCmd
s := staert.NewStaert(rootCmd)

//add sources
s.AddSource(toml)
s.AddSource(f)

s.LoadConfig()

So far, when the subcommand subCmd is called, only args from flæg are parsed.
Would like to parse the TOML source (and maybe any others) as well when a subCommand is called.

Fix hint : Adding subcommands to staert if those subcommands have the same Config struct ?

mapstructure's decoder doesn't zero fields by default

This has been responsible for traefik/traefik#2153

The longer analysis is available in Traefik's issue, but in short, when loading a stored object from the KV store, it's important to zero the destination fields before loading it up, otherwise, mapstructure will just partially overwrite data, leaving possible extra data present.

In the case of Traaefik, this means a JSON array stored as bytes[] could come back with junk at the end if the new array was shorter than the previous one.

Support environment variables source

Is there any plans to add Environment variables as a source?

pseudo code :

s := staert.NewStaert(command)
env := staert.NewEnvSource("PREFIX_")
s.AddSource(env)

TomlSource missing custom parsers

func (ts *TomlSource) Parse(cmd *flaeg.Command) (*flaeg.Command, error) uses flaeg to set pointer value.
fleag.Parse(cmd) requiers custom parsers to work

Tests fail on windows

Tests are failing on windows because of path and separator differences :

c:\proj\gospace\src\github.com\containous\staert\staert_test.go:1534: input /etc/test
		expected /etc/test
		 got C:\etc\test

KvSource ETCD fails

I tried to parse a KvSource working with ETCD using Staert.LoadConfig() (which calls KvSource.Parse())
It prints errors on maps, slices or arrays fields :

4 error(s) decoding:
* 'DefaultEntryPoints': source data must be an array or slice, got string
* 'EntryPoints' expected a map, got 'string'
* 'Etcd' expected a map, got 'string'
* 'Web' expected a map, got 'string'

Probably due to the KV Store which stores a key/value for each directory 😢