Giter Site home page Giter Site logo

properties's Introduction

Travis CI Status License GoDoc

Overview

Please run git pull --tags to update the tags. See below why.

properties is a Go library for reading and writing properties files.

It supports reading from multiple files or URLs and Spring style recursive property expansion of expressions like ${key} to their corresponding value. Value expressions can refer to other keys like in ${key} or to environment variables like in ${USER}. Filenames can also contain environment variables like in /home/${USER}/myapp.properties.

Properties can be decoded into structs, maps, arrays and values through struct tags.

Comments and the order of keys are preserved. Comments can be modified and can be written to the output.

The properties library supports both ISO-8859-1 and UTF-8 encoded data.

Starting from version 1.3.0 the behavior of the MustXXX() functions is configurable by providing a custom ErrorHandler function. The default has changed from panic to log.Fatal but this is configurable and custom error handling functions can be provided. See the package documentation for details.

Read the full documentation on GoDoc

Getting Started

import (
	"flag"
	"github.com/magiconair/properties"
)

func main() {
	// init from a file
	p := properties.MustLoadFile("${HOME}/config.properties", properties.UTF8)

	// or multiple files
	p = properties.MustLoadFiles([]string{
			"${HOME}/config.properties",
			"${HOME}/config-${USER}.properties",
		}, properties.UTF8, true)

	// or from a map
	p = properties.LoadMap(map[string]string{"key": "value", "abc": "def"})

	// or from a string
	p = properties.MustLoadString("key=value\nabc=def")

	// or from a URL
	p = properties.MustLoadURL("http://host/path")

	// or from multiple URLs
	p = properties.MustLoadURL([]string{
			"http://host/config",
			"http://host/config-${USER}",
		}, true)

	// or from flags
	p.MustFlag(flag.CommandLine)

	// get values through getters
	host := p.MustGetString("host")
	port := p.GetInt("port", 8080)

	// or through Decode
	type Config struct {
		Host    string        `properties:"host"`
		Port    int           `properties:"port,default=9000"`
		Accept  []string      `properties:"accept,default=image/png;image;gif"`
		Timeout time.Duration `properties:"timeout,default=5s"`
	}
	var cfg Config
	if err := p.Decode(&cfg); err != nil {
		log.Fatal(err)
	}
}

Installation and Upgrade

$ go get -u github.com/magiconair/properties

License

2 clause BSD license. See LICENSE file for details.

ToDo

  • Dump contents with passwords and secrets obscured

Updated Git tags

13 Feb 2018

I realized that all of the git tags I had pushed before v1.7.5 were lightweight tags and I've only recently learned that this doesn't play well with git describe ๐Ÿ˜ž

I have replaced all lightweight tags with signed tags using this script which should retain the commit date, name and email address. Please run git pull --tags to update them.

Worst case you have to reclone the repo.

#!/bin/bash
tag=$1
echo "Updating $tag"
date=$(git show ${tag}^0 --format=%aD | head -1)
email=$(git show ${tag}^0 --format=%aE | head -1)
name=$(git show ${tag}^0 --format=%aN | head -1)
GIT_COMMITTER_DATE="$date" GIT_COMMITTER_NAME="$name" GIT_COMMITTER_EMAIL="$email" git tag -s -f ${tag} ${tag}^0 -m ${tag}

I apologize for the inconvenience.

Frank

properties's People

Contributors

adityavallabh avatar aliras1 avatar apesternikov avatar doxsch avatar eclipseo avatar haroon-sheikh avatar harsimranmaan avatar kmala avatar liubog2008 avatar maage avatar magiconair avatar mkjor avatar oconnormi avatar pascaldekloe avatar rkhokhla avatar sriv avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

properties's Issues

Trailing comments are not preserved

The README states:

Comments and the order of keys are preserved.

However, comments at the end of a properties file are not preserved. See #44 for a failing test.

Note that the example properties file shown in wikipedia has a trailing comment, so trailing comments are certainly valid.

A special, degenerate case of trailing comments is a properties file consisting of comments alone.

GetInt should return default val instead of panic

In GetInt function should return default value instead of panic

// GetInt parses the expanded value as an int if the key exists.
// If key does not exist or the value cannot be parsed the default
// value is returned. If the value does not fit into an int the
// function panics with an out of range error.
func (p *Properties) GetInt(key string, def int) int {
	v, err := p.getInt64(key)
	if err != nil {
		return def
	}
	return intRangeCheck(key, v)
}

more:

panic(fmt.Sprintf("Value %d for key %s out of range", v, key))

have symbol equal in the value property

Hi I ran into this issue from eBay fabio, i want to set property 'registry.consul.register.tags'. i.e. "key1=val1" as one of the tag, but it has this kind of error:
[FATAL] 1.1.5. circular reference

is that a way to escape equal symbol. thanks

Support typed setter functions

I'm not sure if I'm missing something, but it seems you can only set properties to strings. I know you can read other types, but what if I was to SetInt()?

LoadAll significantly increases code size

Compare the two:

The following compiles to 6.6MB w/ go1.22.5 on osx (arm64)

package main

import (
	"github.com/magiconair/properties"
)

func main() {
	_, err := properties.LoadFile("./config.properties", properties.UTF8)
	if err != nil {
		panic(err)
	}
}

The following compiles to 3.7MB:

package main

import (
	"github.com/magiconair/properties"
)

func main() {
	loader := &properties.Loader{Encoding: properties.UTF8}
	_, err := loader.LoadFile("./config.properties")
	if err != nil {
		panic(err)
	}
}

The reason for this is because properties.LoadFile funnels through Loader.LoadAll, which can load properties files from URLs, so net/http and everything gets pulled in.

Drop gocheck dependency

It is not necessary and makes Debian packaging more complex since they unbundle the vendored dependencies.

Bugs LoadFile will lose the first '\' when config file's value is '\1\n\t%p\t%r\t%u\t%d\t%t\t%e\t%T\t%S\t%U\t%E\t\t'

example.conf

log_line_prefix='\1\n\t%p\t%r\t%u\t%d\t%t\t%e\t%T\t%S\t%U\t%E\t\t'

After I run this golang code:

import (
	"fmt"
	"github.com/magiconair/properties"
        "os"
)

func main() {
        confName := "example.conf"
        filePros, _ := properties.LoadFile(confName, properties.UTF8)
        stat, _ := os.Stat(confName)
        writer, _ := os.OpenFile(confName, os.O_WRONLY|os.O_TRUNC, stat.Mode())
        defer writer.Close()
        filePros.WriteSeparator = "="
        filePros.Write(writer, properties.UTF8)
}

The example.conf content is change to this:

log_line_prefix='1\n\t%p\t%r\t%u\t%d\t%t\t%e\t%T\t%S\t%U\t%E\t\t'

The value's first '\' character lost, I think this is a bug.

Remove assert package from released code

We use viper on our project, and viper has a dependency with magiconair/properties.

When downloading the code for magiconair/properties, we also get magiconair/properties/assert.

This is an issue, because magiconair/properties/assert is easily confused with testify/assert. In fact, under IntelliJ/Goland, magiconair/properties/assert is generally what is imported by defaulted when typing assert. In fact, we regularly need to remove incorrect references to magiconair/properties/assert.

Is there something that can be done? I'm happy to work on a PR that might offer a fix (presumably by moving files under assert to the main folder, plus changing the name of the assert.go file to something ending in _test.go).

Bug(writeComments): Changes comments if \ is included

When reading comments \ are loaded correctly, but when writing they are then replaced by \\. This leads to wrong comments when writing and reading multiple times.

I assume that it is not necessary to encode the comments.

Add LoadMap initializer function

PR #21 added a Map() function to render a Properties object into a string map. LoadMap() should be the counterpart to this to create a Properties object from a string map.

Unable to parse an array of values

Given a properties file consisting of:

Test = foo,bar,baz

A config struct of

type Config struct {
    Test []string `properties:"Test"`
}

And the following decoding

    var cfg Config
    if err := p.Decode(&cfg); err != nil {
        log.Fatal(err)
    }
    fmt.Printf("got test values: %s\n", cfg.Test[1])
```

My program crashes as cfg.Test is of length 1.

Goroutine leak in parser

After parsing some input there is still a goroutine with a lexer around which will never be cleaned up. Parsing a lot of inputs leaks a goroutine per input.

You'll notice it in the goroutine dump as something similar to this:

goroutine 21 [chan send]:
github.com/magiconair/properties.(*lexer).emit(0xc208004240, 0x2)
    /Users/frschroeder/gopath/src/github.com/magiconair/properties/lex.go:99 +0xb9
github.com/magiconair/properties.lexKey(0xc208004240, 0x414148)
    /Users/frschroeder/gopath/src/github.com/magiconair/properties/lex.go:250 +0x132
github.com/magiconair/properties.(*lexer).run(0xc208004240)
    /Users/frschroeder/gopath/src/github.com/magiconair/properties/lex.go:177 +0x4e
created by github.com/magiconair/properties.lex
    /Users/frschroeder/gopath/src/github.com/magiconair/properties/lex.go:170 +0xce

props.Keys() also return the empty key

func ScanProperties(content string) (map[string]string, error) {
    result := map[string]string{}
    props, err := properties.Load([]byte(content), properties.ISO_8859_1)
    if err != nil {
        return result, err
    }
    for _, each := range props.Keys() {
        result[each], _ = props.Get(each)
    }
    return result, nil
}

the result contains an empty string as key.
I did not expect this.

decode into map[string]interface{}

Hello, would be possible to add support to decode into map[string]interface{}. I need to decode a properties file into a generic interface{} but below code is returning an error. Is there a reason to only allow it to be a struct?

	if t.Kind() != reflect.Ptr || v.Elem().Type().Kind() != reflect.Struct {
		return fmt.Errorf("not a pointer to struct: %s", t)
	}

The default value after expansion is an empty String

Problem:

When the property=${foo} set in the file is expanded, the returned value is an empty String.

Is there a particular reason why we have this approach?

Personally, it would be much easier to debug if it was just returned as is when not found.

Expected:
properties.Get("property") = "${foo}"

Actual:
properties.Get("property") = ""

Resolve the properties and exit the programme

I use the code like below , when the programe resolve the properties file , printf the message and exit . I don't want to the program exit , just throw a error to me , I can catch is err and continue reslove the other file .
use code
file_content:=properties.MustLoadFile(path+"/"+name,properties.UTF8) keys := file_content.Keys();
message
2019/10/15 15:37:09 malformed expression

Maintain order of key with FilterXXX

Hi all,

Thanks for sharing your work. I find your package very useful. My properties files must also be edited by administrators. In my program I remove program generated properties and rewrite the properties file. Maintaining the order of the keys and the comments enables a friendlier file for administrators. The comments are transferred using GetComments. I could not find another way to keep the unexpanded property values, except by filtering them.

Currently the order of the keys is not maintained when any of the Filter, FilterRegexp or FilterPrefix is called. A small set of changes in properties.go will enable this.

  // FilterRegexp returns a new properties object which contains all properties
  // for which the key matches the regular expression.
  func (p *Properties) FilterRegexp(re *regexp.Regexp) *Properties {
    pp := NewProperties()
  //for k, v := range p.m {
    for _, k := range p.k {
        if re.MatchString(k) {
  //        pp.Set(k, v)
            pp.Set(k, p.m[k])
        }
    }
    return pp
  }

  // FilterPrefix returns a new properties object which contains all properties
  // for which the key starts with the prefix.
  func (p *Properties) FilterPrefix(prefix string) *Properties {
    pp := NewProperties()
  //for k, v := range p.m {
    for _, k := range p.k {
        if strings.HasPrefix(k, prefix) {
  //        pp.Set(k, v)
            pp.Set(k, p.m[k])
        }
    }
    return pp
  }

Thanks,

Wilbert

Go 1.16 specific test error

On Fedora Rawhide with Go 1.16 beta 1, I get the following test error:

Testing    in: /builddir/build/BUILD/properties-1.8.4/_build/src
         PATH: /builddir/build/BUILD/properties-1.8.4/_build/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
       GOPATH: /builddir/build/BUILD/properties-1.8.4/_build:/usr/share/gocode
  GO111MODULE: off
      command: go test -buildmode pie -compiler gc -ldflags " -X github.com/magiconair/properties/version=1.8.4 -extldflags '-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld  '"
      testing: github.com/magiconair/properties
github.com/magiconair/properties
2021/01/16 17:04:51 properties: doesnotexist.properties not found. skipping
2021/01/16 17:04:51 properties: /tmp/properties477326312foo not found. skipping
2021/01/16 17:04:51 properties: /tmp/properties722766887foo not found. skipping
2021/01/16 17:04:51 properties: http://127.0.0.1:38403/c returned 404. skipping
	assert.go:62: got time: invalid duration "ghi" which does not match time: invalid duration ghi
--- FAIL: TestMustGetParsedDuration (0.00s)
FAIL
exit status 1
FAIL	github.com/magiconair/properties	0.025s

It seems the following code does not handle version over 1.15:

ver := runtime.Version()
switch {
// gotip and go1.15 will return `time: invalid duration "ghi"`
case !strings.HasPrefix(ver, "go") || strings.HasPrefix(ver, "go1.15"):
assert.Panic(t, func() { p.MustGetParsedDuration("key2") }, `time: invalid duration "ghi"`)
default:
assert.Panic(t, func() { p.MustGetParsedDuration("key2") }, `time: invalid duration ghi`)
}

Make expansion of expressions optional per load

As part of our configuration service, we need to store configurations as is in key-value records without substituing the ${..} expressions. Application that retrieve properties from this service will do the expansions.

My suggestion is to provide another Load function that takes the extra boolean parameter expandExpressions

MustGetString documentation is misleading

Hello everyone.

I was testing some code that utilizes the MustGetString function, but was confused when the tests were failing when I was asserting a panic to happen.

I dug into the code and found that it doesn't actually panic, but calls log.Fatal through your ErrorHandler. It would have saved me a lot of time if I knew this from the function's documentation.

I can make the changes myself, but I wanted to see if I am incorrect in my thinking that the documentation is inaccurate first.

Current Documentation:
MustGetString returns the expanded value for the given key if exists or panics otherwise.

Proposed Change:
MustGetString returns the expanded value for the given key if exists or calls log.Fatal otherwise.

Add an ability to do not fail Decode function if no 'default' value specified in the 'properties' tag

Currently the Decode function returns an error if the tag does not contain 'default' and the key does not exist in the properties file.

Sometimes it's preferable to do not fail in this case but use the original field value of the structure passed to the Decode function, e.g by passing some specific key ('predefined') to the properties tag:

type Config struct {
	Field string `properties:"somefield,predefined"`
}

conf := &Config{
    Field: "Default value"
}

p := properties.MustLoadFiles([]string{
    "config.properties",
}, properties.UTF8, true)

err := p.Decode(conf)

if err == nil {
    fmt.Println(conf.Field)
}

WriteComment should ignore empty ones

If no comment was set for a key,value pair then I expect the WriteComment to not write the extra lines.

#
eddy.aq.concurrent.consumers = 10

#
eddy.aq.max.concurrent.consumers = 10

vs

eddy.aq.concurrent.consumers = 10
eddy.aq.max.concurrent.consumers = 10

Test failure on 32-bit platforms: "constant 4294967296 overflows int"

Detected on armhf by a Debian reproducible build server here:

https://reproducible.debian.net/rb-pkg/unstable/armhf/golang-github-magiconair-properties.html

go test -v github.com/magiconair/properties
# github.com/magiconair/properties
src/github.com/magiconair/properties/properties_test.go:811: constant 4294967296 overflows int
FAIL    github.com/magiconair/properties [build failed]

This happens inside TestPanicOn32BitUintOverflow():

https://github.com/magiconair/properties/blob/master/properties_test.go#L811

This test fails on 32-bit platforms such as armhf, but works fine with 64-bit platforms such as amd64.

Thanks in advance!

Anthony

Bugs when properties are specified as metadata of struct

There are some bugs when we specify properties as part of the struct metadata.
e.g

config file

graphite.host=localhost
graphite.metric.name=gtest

cloud.tenant.name=xxx-qa
cloud.keystone.url=https://xxx.zzz.ccc.aaa.xx/v2.0
cloud.swift.username=yyy-bb-ccc
cloud.swift.passwd=changeme

kafka.topic=ttt
kafka.servers=localhost:9092,zzz.dev.yyy.ca:9092
kafka.consumergroup=test
kafka.consumeFromStart=true

bulk.delete.max_wait=10s
log.level=debug
// Configuration holds external configuration for the consumers
type Configuration struct {
	CloudTenant           string              `properties:"cloud.tenant.name"`
	KeystoneURL           string              `properties:"cloud.keystone.url"`
	Username              string              `properties:"cloud.swift.username"`
	Userpwd               string              `properties:"cloud.swift.passwd"`
	Servers               []string			  `properties:"kafka.servers"`
	Cgroupname            string              `properties:"kafka.consumergroup"`
	Topic                 string              `properties:"kafka.topic"`
	ConsumeFromStart      bool                `properties:"kafka.consumeFromStart,default=false"`
        KafkaVersion          sarama.KafkaVersion `properties:"kafka.version,default=0.11.0.2"`
	Port                  int                 `properties:"port,default=8383"`
	Graphite              string              `properties:"graphite.host, default=carbon.service.consul"`
	GraphiteMetricName    string              `properties:"graphite.metric.name"`
	LogLevel              string              `properties:"log.level,default=info"`
	BulkDeleteMaxItems    int                 `properties:"bulk.delete.max_items,default=100"`
	BulkDeleteMaxWaitTime time.Duration       `properties:"bulk.delete.max_wait,default=5s"`
	

}

It fails on the first field/property it has a issue within the struct. All fields after this filed (in struct order) are not processed. e.g if the field KafkaVersion could not be decoded then all fields below it will also not be decoded.

There are issues in following cases
1> The property is missing in property file and no default value is mentioned in the struct
2> if the field type is not supported by decoder eg (sarama.KafkaVersion)
3> if the fields in the struct do not have the property tag and a property with the field name is not present in the property file.

Is there any way we can exclude a field from decode logic?
Regards
Rohan

Equal sign is incorrectly escaped in key

I'm doing the following test:

// Set a key to "a=b" and value "c"
p := properties.NewProperties()
p.Set("a=b", "c")
buf := new(bytes.Buffer)
p.Write(buf, properties.UTF8)
println(buf.String())

I'd espect the serialized version to be:

a\=b = c

Since the \ should be used to escape the first equal sign, but the program prints:

a=b = c

Which seems wrong. If saved in a file and read again, the key becomes "a" (while, when reading from the correct file, the key is correctly interpreted as "a=b").

Ability Decode to Multiple structs

In case the properties file has a logical division of the configurations, I'd like to decode to several dedicated structs.

For example:
Given the following config.properties file

app.port=3000
db.host=127.0.0.1
db.port=8000
db.username=username123
db.password=password123
pglistener.min_reconn=10 * time.Second
pglistener.max_reconn=1 * time.Minute
...
...

I'd like to decode to the following structs:

type Config struct {
	AppPort string `properties:"app.port,default=3000"`
}

type DBConfig struct {
	Host          string  `properties:"db.host,default=127.0.0.1"`
	Port           int       `properties:"db.port,default=5432"`
	Username string `properties:"db.username"`
	Password  string `properties:"db.password"`
}


type PGListener struct {
	MinReconnectInterval time.Duration `properties:"pglistener.min_reconn"`
	MaxReconnectInterval time.Duration `properties:"pglistener.max_reconn"`
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.