chimpler / pyhocon Goto Github PK
View Code? Open in Web Editor NEWHOCON parser for Python
License: Apache License 2.0
pyhocon's People
Stargazers
Watchers
Forkers
txominpelu programmer-util kbakba 2m joehalliwell okdtsk laserson bkuberek liamstewart acx2015 peoplepattern zanhsieh vamega movermeyer catacgc virgil-palanciuc cmenguy pombredanne emilybache liqinshan gilles-duboscq sanzinger vsajip ryban zapster davleopo cnspica iseroccosys yzliao atomerju artwr conkty zhenghao1 israel-lugo jjtk88 hekaldama joelgrus nemanja88 kushwiz mchenmath raghukumarc thejaravi enlightenalpha aalba6675 chunyang-wen kekamui bdmartin roee-allegro renardeinside sambible instagrim-dev relotnek shutianxu elkmonster lune-sta christopheduong magor witzatom seatalk chreekat selingungor yc-feej pbatko themaco mvonthron abhi5658054 gpoulin anujkumar93 bmartinn scottj97 geektemo mrijken jrouly mstrotta vikram-saha lysandergg ussx-hares xinghaixu klamann yuzhujiutian olii jwbargsten jettjones terry-xiaoyu charlietsai ifeitao jimmydevine elbakramer philipluk rch kevinmfong immortalyash maniteja-kurapati precognize python-repository-hub netux afanasev zifeo fohlen atlantic777pyhocon's Issues
config_tree.put() mutates the parameter
Similarly to #95, config_tree.put() also mutates the input parameter.
from pyhocon.config_tree import ConfigTree
#This code is a parameterized version of the test "test_config_list"
value1 = [4, 5]
value2 = [6, 7]
value3 = [8, 9]
config_tree = ConfigTree()
config_tree.put("a.b.c", value1)
assert config_tree.get("a.b.c") == value1
config_tree.put("a.b.c", value2)
assert config_tree.get("a.b.c") == value2
config_tree.put("a.b.c", value3, True)
assert config_tree.get("a.b.c") == (value2 + value3) #Fails!?!
#value2 == [6, 7, 8, 9], since it was mutated.
This is surprising, as I would expect it to not be mutated.
It is possible to escape . ?
i have dictionary with key "hello.world"
def tohocon(args):
return HOCONConverter.convert(ConfigFactory.from_dict(args), 'hocon', 4)
is is possible to escape the . character?
Thank You ๐
HOCONConverter.to_properties escapes equals sign in quoted string and removes quotes
Config:
foo = "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="
Code:
from pyhocon import ConfigFactory
from pyhocon import HOCONConverter
config = ConfigFactory.parse_file("test.conf", resolve=False)
print HOCONConverter.to_properties(config)
Result:
foo = -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath\=
Expected result:
foo = "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="
Support space separated lists (no comma)
Error messages broken by Unicode keys
A number of error messages are broken if the dictionary key is unicode.
For example:
>>> from pyhocon import ConfigFactory
>>> t = ConfigFactory.from_dict({})
>>> t.get_string(u'ฮฆ')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyhocon/config_tree.py", line 202, in get_string
string_value = str(self.get(key, default))
File "pyhocon/config_tree.py", line 190, in get
return self._get(ConfigTree.parse_key(key), 0, default)
File "pyhocon/config_tree.py", line 139, in _get
raise ConfigMissingException("No configuration setting found for key {key}".format(key='.'.join(key_path[:key_index + 1])))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u03a6' in position 0: ordinal not in range(128)Parsing exception for a list with an EOL
Code:
from pyhocon import ConfigFactory
c = ConfigFactory.parse_string('values :\n[\n"null",\n"undefined",\n"0",\n]\n')
This is throwing the following exception :
File "/pyhocon/init.py", line 60, in parse_string
return ConfigParser().parse(content, basedir)
File "/pyhocon/init.py", line 197, in parse
config = config_expr.parseString(content, parseAll=True)[0]
File "/pyparsing.py", line 1125, in parseString
raise exc
pyparsing.ParseException: Expected end of text (at char 9), (line:2, col:1)
Please compare with the working example, when using Egyptian brackets (no \n between ":" and "["):
c = ConfigFactory.parse_string('values : [\n"null",\n"undefined",\n"0",\n]\n')
Nested substitution fails and it's wrongly detected as a cycle
See PR for a test #63.
Any nested substitution that uses an object, that in turn, has some chained substitutions, fails
obj = ${var}
var {
a = "a"
b = $a"b"
c = $b"c"
}
IO Error on include when file does not exist
pyhocon==0.3.2
According to HOCON.md#include-semantics-missing-files, include ... should be silent if the file does not exist.
I have setup my configuration to load files that might exist in production and I get errors when running in environments where those files don't exist:
File "/Users/bkuberek/Development/people-service/.venv/lib/python2.7/site-packages/pyhocon/__init__.py", line 30, in parse_file
with open(filename, 'r') as fd:
IOError: [Errno 2] No such file or directory: '/etc/people-service.conf'
Here is my config:
{
# load the configuration defaults
include file("defaults.conf")
# application overrides
ext {
someaddon {
enabled = true
}
}
# This configuration will override the defults if it exists. It will be ignored if it doesn't.
include file("/etc/people-service.conf")
# This is for user specific configuration.
include file("people-dev.conf")
}
Config with hyphens in keys are not read properly.
For example, the config
a.b.c-d = "value"
is not possible. It is possible in typesafe HOCON.
support for "+=" field separator?
I get an error when parsing a hocon file:
[ec2-user@ip-172-16-2-117 ~]$ pyhocon -i /opt/mapr/drill/drill-1.1.0/conf/drill-override-example.conf
Traceback (most recent call last):
File "/usr/local/bin/pyhocon", line 9, in <module>
load_entry_point('pyhocon==0.3.4', 'console_scripts', 'pyhocon')()
File "/usr/local/lib/python2.7/site-packages/pyhocon/tool.py", line 188, in main
HOCONConverter.convert(args.input, args.output, args.format)
File "/usr/local/lib/python2.7/site-packages/pyhocon/tool.py", line 145, in convert
config = ConfigFactory.parse_file(input_file)
File "/usr/local/lib/python2.7/site-packages/pyhocon/__init__.py", line 40, in parse_file
return ConfigFactory.parse_string(content, os.path.dirname(filename))
File "/usr/local/lib/python2.7/site-packages/pyhocon/__init__.py", line 75, in parse_string
return ConfigParser().parse(content, basedir)
File "/usr/local/lib/python2.7/site-packages/pyhocon/__init__.py", line 216, in parse
config = config_expr.parseString(content, parseAll=True)[0]
File "/usr/local/lib/python2.7/site-packages/pyparsing.py", line 1125, in parseString
raise exc
pyparsing.ParseSyntaxException: Expected "{" (at char 1087), (line:20, col:33)
Line 20 in the above looks like this:
drill.logical.function.packages += "org.apache.drill.exec.expr.fn.impl"
And column 33 implicates the "+=" field separator.
Any plans to add support for this?
https://github.com/typesafehub/config/blob/master/HOCON.md#the--field-separator
Regression: ConfigTree moved
When I upgraded a codebase from 0.3.1 to 0.3.9, this import started to fail:
from pyhocon import ConfigTreelist variable substitution
I'm seeing a strange issue that has to do with lists and variable substitution. I verified that I was using the code from version 0.3.15 but the problem still persists. I'm wondering if this has something to do with config_parser.py checking if a value is a ConfigValues instance, instead of checking whether the value is a ConfigValues instance OR is a list that contains a ConfigValues instance (or a list of lists, etc.).
from pyhocon import ConfigFactory
# OUTPUT USING version 0.3.15
#
# >>> execfile("pyhocon_test.py")
# >>> count_incorrect()
# 23 / 100 entries are bad for config 1, key: group1.list
# bad entries: ["[[ConfigValues: a], 'b', 'c', 'd']"]
# 0 / 100 entries are bad for config 1, key: group2.list
# bad entries: []
# 0 / 100 entries are bad for config 2, key: group1.list
# bad entries: []
# 0 / 100 entries are bad for config 2, key: group2.single
# bad entries: []
# 0 / 100 entries are bad for config 3, key: group1.list
# bad entries: []
# 0 / 100 entries are bad for config 3, key: group2.list
# bad entries: []
# 0 / 100 entries are bad for config 4, key: group1.list
# bad entries: []
# 0 / 100 entries are bad for config 5, key: group1.single2
# bad entries: []
# 0 / 100 entries are bad for config 5, key: group2.single2
# bad entries: []
# there is an intermittent issue parsing this config string
TEST_CONFIG1 = """
{
group1 {
single = a
list = [${group1.single}, "b", "c", "d"] # !!! this can be incorrectly parsed !!!
}
group2 {
list = ${group1.list}
}
}
"""
# similar to TEST_CONFIG1 but group2 refers to group1.single instead of group1.list
TEST_CONFIG2 = """
{
group1 {
single = a
list = [${group1.single}, "b", "c", "d"]
}
group2 {
single = ${group1.single}
}
}
"""
# similar to TEST_CONFIG1 but group1.list does not have a variable substitution
TEST_CONFIG3 = """
{
group1 {
single = a
list = ["a", "b", "c", "d"]
}
group2 {
list = ${group1.list}
}
}
"""
# similar to TEST_CONFIG1 but there is no group2
TEST_CONFIG4 = """
{
group1 {
single = a
list = [${group1.single}, "b", "c", "d"]
}
}
"""
# similar to TEST_CONFIG1 but there there are no list definitions
TEST_CONFIG5 = """
{
group1 {
single = a
single2 = ${group1.single}
}
group2 {
single2 = ${group1.single2}
}
}
"""
TEST_LIST = [
(1, TEST_CONFIG1, 'group1.list', ['a', 'b', 'c', 'd']),
(1, TEST_CONFIG1, 'group2.list', ['a', 'b', 'c', 'd']),
(2, TEST_CONFIG2, 'group1.list', ['a', 'b', 'c', 'd']),
(2, TEST_CONFIG2, 'group2.single', 'a'),
(3, TEST_CONFIG3, 'group1.list', ['a', 'b', 'c', 'd']),
(3, TEST_CONFIG3, 'group2.list', ['a', 'b', 'c', 'd']),
(4, TEST_CONFIG4, 'group1.list', ['a', 'b', 'c', 'd']),
(5, TEST_CONFIG5, 'group1.single2', 'a'),
(5, TEST_CONFIG5, 'group2.single2', 'a'),
]
def count_incorrect():
for (config_id, config_str, key, correct_entry) in TEST_LIST:
bad_count = 0
bad_entries = set()
total_count = 100
for i in range(total_count):
config = ConfigFactory.parse_string(config_str)
entry = config.get(key)
if entry != correct_entry:
bad_count += 1
bad_entries |= set([repr(entry)])
print "%4d / %4d entries are bad for config %d, key: %s" % (bad_count, total_count, config_id, key)
print " bad entries: %s" % (list(bad_entries),)How to dump HOCON file?
Hi,
Just wonder how to do this with pyhocon?
https://marcinkubala.wordpress.com/2013/10/09/typesafe-config-hocon/
Thanks.
isinstance(config, str) will fail on 2.x
I've tried to parse a config like this:
config = ConfigFactory.parse_file('path1').with_fallback('path2')
with_fallback takes either a ConfigTree or a string. It is testing this with isinstance(config, str), but this only works correctly in Python 3.x. In my case (2.7) it interpreted 'path2' as a ConfigTree which crashes with a parse-error obviously.
The best way of supporting this in 2.x and 3.x simultaneously is probably the library six which you can see in this answer: http://stackoverflow.com/questions/11301138/how-to-check-if-variable-is-string-with-python-2-and-3-compatibility
Of course you could also just reverse the if-clause by checking for a ConfigTree object instead.
You should also consider checking if there are other locations in your code where checking for a string occurs.
I can fix it myself and submit a pullrequest if you tell me what's your decision, of course. :)
values with quotes are not escaped properly
I had a configuration file where a property contained a piece of string that had a double quote in it, but pyhocon was not able to parse it correctly. Here is a breakdown of the issue:
if I have a configuration file like so:
// test.conf
{
test {
foo = abc"def
bar = "abc\"def"
foobar = "abc"def"
}
}
parse it into Pyhocon and print out the values:
config = ConfigFactory.parse_file('test.conf')
print(config['test']['foo'])
print(config['test']['bar'])
print(config['test']['foobar'])
I get as output:
'abc"def'
'abc\\def"'
'abcdef"'
The first output is correct, but the second and third is not. This was done in Python 3.4.3 btw.
Parser fails if the last line is a comment
The grammar fails to parse files where the last line is a comment.
Example:
foo: "1"
bar: "2"
#DO NOT CHANGE ANY OF THE ABOVE SETTINGS!
Gives the exception:
Traceback (most recent call last):
File "test.py", line 6, in <module>
config = ConfigFactory.parse_string(input)
File "/<redacted>/pyhocon/config_parser.py", line 90, in parse_string
return ConfigParser().parse(content, basedir, resolve)
File "/<redacted>/pyhocon/config_parser.py", line 273, in parse
config = config_expr.parseString(content, parseAll=True)[0]
File "/<redacted>/env/lib/python2.7/site-packages/pyparsing.py", line 1617, in parseString
raise exc
pyparsing.ParseSyntaxException: No match found for Suppress:(W:(
)) (at char 1), (line:1, col:2)
The HOCON spec does not say this isn't allowed, so it should parse correctly as a comment.
Inconsistent behavior python 2.7.9 vs 2.7.6
pyhocon==0.3.3
I've noticed that the following is happening for me when deploying my app to an ubuntu server. On my mac running python 2.7.9 the output is fine, but when running in my ubuntu server on python 2.7.6 I get wrong output. I'm still trying to debug this.
Configuration
# config/people.conf
{
database {
host = "localhost"
port = "5432"
user = "people"
name = "peopledb"
pass = "dev"
url = "postgresql://"${database.user}":"${database.pass}"@"${database.host}":"${database.port}"/"${database.name}
host = ${?DB_HOST}
}
include file('user.conf')
}
# config/user.conf
{
database {
host = "192.168.33.10"
}
}
Python 2.7.9 (OS X and homebrew)
pyhocon -i config/people.conf
{
"database": {
"host": "192.168.33.10",
"port": "5432",
"user": "people",
"name": "peopledb",
"pass": "dev",
"url": "postgresql://people:[email protected]:5432/peopledb"
}
}
Python 2.7.6 (Ubuntu trusty)
pyhocon -i config/people.conf
{
"database": {
"host": "192.168.33.10",
"port": "5432",
"user": "people",
"name": "peopledb",
"pass": "dev",
"url": "postgresql://people:dev@localhost:5432/peopledb"
}
}
Notice how the hostname part of the url differs.
Substitution with merged configs is incorrect
Hi dear maintainers,
with pyhocon 0.3.35 I get unexpected substitution results with merged configs.
I have following configurations:
base.conf
base {
baseattr1: 1
baseattr2: 1
}
derived = ${base} {
derivedattr: 1
lost: 1
}
test.conf
include "base.conf"
base {
baseattr1: 2
baseattr3: 1
}
derived = ${base} {
derivedattr1: 2
derivedattr2: 1
}
test.py:
#!/usr/bin/env python
from pyhocon import ConfigFactory
from pyhocon.tool import HOCONConverter
conf = ConfigFactory.parse_file('test.conf')
print(HOCONConverter.to_json(conf))
I executed the test.py with the configs being in the same directory.
Expected result:
{
"base": {
"baseattr1": 2,
"baseattr2": 1,
"baseattr3": 1
},
"derived": {
"baseattr1": 2,
"baseattr2": 1,
"baseattr3": 1,
"derivedattr1": 2,
"lost": 1,
"derivedattr2": 1
}
}
Actual result:
{
"base": {
"baseattr1": 2,
"baseattr2": 1,
"baseattr3": 1
},
"derived": {
"baseattr1": 2,
"baseattr2": 1,
"baseattr3": 1,
"derivedattr1": 2,
"derivedattr2": 1
}
}
Notice that the key 'lost' is missing.
In fact, if I adapt the test.conf like this; the output is like expected.
include "base.conf"
base {
baseattr1: 2
baseattr3: 1
}
derived {
derivedattr1: 1
}
Notice that the only change was to remove the object merge for key 'derived'.
I did not dig into the code yet, but I think it is related to merging and not the substitution. If I set 'resolve=False' in test.py, the output looks like this:
{
"base": {
"baseattr1": 2,
"baseattr2": 1,
"baseattr3": 1
},
"derived": [ConfigValues: [ConfigSubstitution: base],ConfigTree([(u'derivedattr1', 2), (u'derivedattr2', 1)])]
}
The key 'lost' is already missing.
I tested this stuff with the JVM implementation com.typesafe.config v1.3.0 which yields the expected results.
Do you have any insights on that?
Many thanks in advance
Greets,
Alexander
Merging objects with array fields behaves incorrectly
Given the following input:
base : {
foo: ["a"]
bar: ${base.foo} ["b"]
}
sub : ${base} {
baz: ${base.bar} ["c"]
}
running pyhocon -f json -i test.hocon gives:
{
"base": {
"foo": [
"a"
],
"bar": [
"a",
"b"
]
},
"sub": {
"foo": [
"a"
],
"bar": [
"a",
"b"
],
"baz": [
"a",
"b",
"c"
]
}
}
That behaves as expected. However, if we now add an additional object sub2 at the end like this:
base : {
foo: ["a"]
bar: ${base.foo} ["b"]
}
sub : ${base} {
baz: ${base.bar} ["c"]
}
sub2: ${sub}
then running pyhocon -f json -i test.hocon gives:
{
"base": {
"foo": [
"a"
],
"bar": [
"a",
"b"
]
},
"sub": {
"foo": [
"a"
],
"bar": [
"a",
"b"
],
"baz": [
[
"a",
"b",
"c"
]
]
},
"sub2": {
"foo": [
"a"
],
"bar": [
"a",
"b"
],
"baz": [
"a",
"b",
"c"
]
}
}
Note that both sub.baz now seems incorrect - an additional array nesting level seems to have appeared out of nowhere.
Substitutions don't work correctly.
The following example, taken from the typesafe config README will not result in the expected config.
data-center-generic = { cluster-size = 6 }
data-center-east = ${data-center-generic}
data-center-east = { name = "east" }
data-center-west = ${data-center-generic}
data-center-west = { name = "west", cluster-size = 8 }
Instead of data-center-east being extended to have the a name key in addition to the cluster-size key, the old keys are removed and only the name key is remaining.
This can be verified by using the included tool.
Objects merging doesn't work
Using simple example from HOCON readme:
data-center-generic = { cluster-size = 6 }
data-center-east = ${data-center-generic} { name = "east" }
(saved as test.conf)
In [13]: conf = ConfigFactory.parse_file('test.conf')
---------------------------------------------------------------------------
ParseException Traceback (most recent call last)
<ipython-input-13-d3bb51afbb00> in <module>()
----> 1 conf = ConfigFactory.parse_file('test.conf')
/usr/local/lib/python2.7/site-packages/pyhocon/__init__.pyc in parse_file(filename)
27 with open(filename, 'r') as fd:
28 content = fd.read()
---> 29 return ConfigFactory.parse_string(content, os.path.dirname(filename))
30
31 @staticmethod
/usr/local/lib/python2.7/site-packages/pyhocon/__init__.pyc in parse_string(content, basedir)
55 :type return: Config
56 """
---> 57 return ConfigParser().parse(content, basedir)
58
59
/usr/local/lib/python2.7/site-packages/pyhocon/__init__.pyc in parse(content, basedir)
190 + (list_expr | dict_expr | inside_dict_expr) \
191 + ZeroOrMore(comment | eol_comma)
--> 192 config = config_expr.parseString(content, parseAll=True)[0]
193 ConfigParser._resolve_substitutions(config, substitutions)
194 return config
/usr/local/lib/python2.7/site-packages/pyparsing.pyc in parseString(self, instring, parseAll)
1123 else:
1124 # catch and re-raise exception from here, clears out pyparsing internal stack trace
-> 1125 raise exc
1126 else:
1127 return tokens
ParseException: Expected end of text (at char 85), (line:2, col:43)
Inheritance is a such a nice feature in HOCON, hopefully this can be fixed.
pyhocon is missing the argparse dependency when installed for Python 2.6
When you install pyhocon for Python 2.6:
pip install pyhocon
It does not download the 'argparse' dependency automatically. Note that this library is included in the standard library for Python >= 2.7 and >= 3.2
If you try to use the pyhocon library without it, you get:
>>> import pyhocon
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.6/site-packages/pyhocon/__init__.py", line 5, in <module>
from .tool import HOCONConverter # noqa
File "/usr/lib/python2.6/site-packages/pyhocon/tool.py", line 1, in <module>
import argparse
ImportError: No module named argparse
It seems that pyhocon supports Python 2.6, given that the package is in the Python 2.6 pypi category.
JSON output redundant key
Given the configuration:
{
database {
host = "localhost"
port = 8000
url = ${database.host}":"${database.port}
}
database.host = "other.host.net"
}
The result:
pyhocon -i db.conf
{
"database": {
"host": "other.host.net",
"port": 433,
"url": "other.host.net:433"
},
"database.host": "other.host.net"
}
So the json output has a database.host key.
Issue with environment variables not expanding correctly
pyhocon==0.3.1
Hi,
I am having trouble getting environment variables to work properly. I have tried both ${VAR} and ${?VAR}. See config below:
# defaults.conf
{
database {
host = localhost
port = 5432
user = people
name = peopledb
pass = D3V_0NLY
}
}
# environment.conf
{
include file("defaults.conf")
database {
host = ${DB_HOST}
pass = ${?DB_PASS}
}
}
If the environment variable exists then I get this results, which are invalid and fails when parsing:
pyhocon -i config/people.conf -f json
{
"database": {
"host": [ConfigValues: 192.168.1.10],
"port": 5432,
"user": "people",
"name": "peopledb",
"pass": [ConfigValues: 12345]
}
}
If the environment variable doesn't exist I get an exception:
pyhocon -i config/people.conf -f json
Traceback (most recent call last):
File "/Users/bkuberek/projects/service/.venv/bin/pyhocon", line 9, in <module>
load_entry_point('pyhocon==0.3.1', 'console_scripts', 'pyhocon')()
File "/Users/bkuberek/projects/service/.venv/lib/python2.7/site-packages/pyhocon/tool.py", line 167, in main
HOCONConverter.convert(args.input, args.output, args.format)
File "/Users/bkuberek/projects/service/.venv/lib/python2.7/site-packages/pyhocon/tool.py", line 141, in convert
config = ConfigFactory.parse_file(input_file)
File "/Users/bkuberek/projects/service/.venv/lib/python2.7/site-packages/pyhocon/__init__.py", line 32, in parse_file
return ConfigFactory.parse_string(content, os.path.dirname(filename))
File "/Users/bkuberek/projects/service/.venv/lib/python2.7/site-packages/pyhocon/__init__.py", line 60, in parse_string
return ConfigParser().parse(content, basedir)
File "/Users/bkuberek/projects/service/.venv/lib/python2.7/site-packages/pyhocon/__init__.py", line 195, in parse
ConfigParser._resolve_substitutions(config, substitutions)
File "/Users/bkuberek/projects/service/.venv/lib/python2.7/site-packages/pyhocon/__init__.py", line 229, in _resolve_substitutions
resolved_value = ConfigParser._resolve_variable(config, substitution)
File "/Users/bkuberek/projects/service/.venv/lib/python2.7/site-packages/pyhocon/__init__.py", line 211, in _resolve_variable
col=col(substitution.loc, substitution.instring)))
pyhocon.exceptions.ConfigSubstitutionException: Cannot resolve variable ${?DB_HOST} (line: 5, col: 12)
String substitution does not work in merged lists/dicts
Config:
application.foo = 128m
application.default-jvm-opts = [-XX:+UseParNewGC]
application.large-jvm-opts = ${application.default-jvm-opts} [-Xm16g, ${application.foo}]
Code:
conf = ConfigFactory.parse_file('some.conf')
print(conf["application.large-jvm-opts"])
Result:
['-XX:+UseParNewGC', '-Xm16g', [ConfigValues: [ConfigSubstitution: application.foo]]]
Expected result:
['-XX:+UseParNewGC', '-Xm16g', '128m']
String substitution adds extra space
Config file test.conf:
app.heap_size = 128
app.java_opts = [
-Xms${app.heap_size}m
-Xmx${app.heap_size}m
]
Code:
from pyhocon import ConfigFactory
config = ConfigFactory.parse_file("test.conf")
print(config['app']['java_opts'])
Result: ['-Xms128 m', '-Xmx128 m']
Expected result: ['-Xms128m', '-Xmx128m']
ConfigTree._merge_config_tree() to @staticmethod
Usecase:
from pyhocon.config_tree import ConfigTree
from pyhocon import ConfigFactory
from pyhocon import ConfigParser
env = ConfigFactory.parse_file("env.conf", resolve=True)
# app.conf has non-existent properties which will be generated later so I'm not resolving it
app = ConfigFactory.parse_file("app.conf", resolve=False)
'''
Some code to generate additional properties for app.conf based on env.conf and arguments passed to script
'''
ConfigTree.put(app, "app.new_properties", ["foo", "bar"])
ConfigTree.put(app, "app.some_other_property", "baz")
conf = ConfigTree._merge_config_tree(env,app)
ConfigParser.resolve_substitutions(conf)
_merge_config_tree is useful method and imho should be static or maybe added as static to ConfigParser as e.g. merge_configs
Self-referential substitutions fail if key is a path
As reported in #35 these 2 examples fail:
a.b.c += "foo"a.b.c = ${?a.b.c} ["foo"](to see the actual exceptions, this first patch is necessary:
diff --git a/pyhocon/config_parser.py b/pyhocon/config_parser.py
index 9e20236..f29869c 100644
--- a/pyhocon/config_parser.py
+++ b/pyhocon/config_parser.py
@@ -510,10 +510,10 @@ class ConfigTreeParser(TokenConverter):
else:
value = values[0]
if isinstance(value, list) and operator == "+=":
- value = ConfigValues([ConfigSubstitution(key, True, '', False, loc), value], False, loc)
+ value = ConfigValues([ConfigSubstitution(key, True, '', instring, loc), value], False, loc)
config_tree.put(key, value, False)
elif isinstance(value, str) and operator == "+=":
- value = ConfigValues([ConfigSubstitution(key, True, '', True, loc), ' ' + value], True, loc)
+ value = ConfigValues([ConfigSubstitution(key, True, '', instring, loc), ' ' + value], True, loc)
config_tree.put(key, value, False)
elif isinstance(value, list):
config_tree.put(key, value, False))
${?foo} unsets previous value if new is undefined
pyhocon==0.3.2
Apparently ${?VAR_NAME} no longer complains about undefined variables but is seems to be unsetting the key. My expectation is that ${?foo} would leave the key referencing the last value unless it had something to replace it with.
...
File "/Users/bkuberek/Development/people-service/.venv/lib/python2.7/site-packages/pyhocon/__init__.py", line 203, in parse
ConfigParser._resolve_substitutions(config, substitutions)
File "/Users/bkuberek/Development/people-service/.venv/lib/python2.7/site-packages/pyhocon/__init__.py", line 246, in _resolve_substitutions
is_optional_resolved, resolved_value = ConfigParser._resolve_variable(config, substitution)
File "/Users/bkuberek/Development/people-service/.venv/lib/python2.7/site-packages/pyhocon/__init__.py", line 228, in _resolve_variable
col=col(substitution.loc, substitution.instring)))
pyhocon.exceptions.ConfigSubstitutionException: Cannot resolve variable ${database.name} (line: 20, col: 102)
Here is my configuration:
{
database {
host = "localhost"
port = "5432"
user = "people"
name = "peopledb"
pass = "dev"
url = "postgresql://"${database.user}":"${database.pass}"@"${database.host}":"${database.port}"/"${database.name}
# Override from environment variables if they exist
host = ${?DB_HOST}
port = ${?DB_PORT}
user = ${?DB_USER}
name = ${?DB_NAME}
pass = ${?DB_PASS}
}
}
NoneValue object in parse result of "a=null" in PyPy
In PyPy, parse result of "a=null" has a NoneValue object which is wrong:
Python 2.7.12 (aff251e543859ce4508159dd9f1a82a2f553de00, Nov 25 2016, 00:02:08)
[PyPy 5.6.0 with GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>> from pyhocon import ConfigFactory
>>>> ConfigFactory.parse_string('a=null')
ConfigTree([('a', <pyhocon.config_tree.NoneValue object at 0x00000001116cbe18>)])
In CPython, it's ok:
Python 2.7.13 (default, Dec 18 2016, 07:03:39)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyhocon import ConfigFactory
>>> ConfigFactory.parse_string('a=null')
ConfigTree([('a', None)])
Versions:
- pyhocon==0.3.35
- pyparsing==2.1.10
- PyPy and CPython versions are above.
Fixup substitutions in included files
According to the docs substitutions should be fixed up for included files:
Say for example that the root configuration is this:
{ a : { include "foo.conf" } }And "foo.conf" might look like this:
{ x : 10, y : ${x} }If you parsed "foo.conf" in isolation, then
${x}would evaluate
to 10, the value at the pathx. If you include "foo.conf" in an
object at keya, however, then it must be fixed up to be
${a.x}rather than${x}.
And:
Say that the root configuration redefines
a.x, like this:{ a : { include "foo.conf" } a : { x : 42 } }Then the
${x}in "foo.conf", which has been fixed up to
${a.x}, would evaluate to42rather than to10.
Substitution happens after parsing the whole configuration.
Currently, this is not supported in pyhocon. I've turned the examples from the docs into tests: zapster@466e298 (feel free to use them)
pyparsing ==2.0.3
Is there any chance this can be updated to 2.0.6? Or not pinned to a specific version?
Appending to a nested array causes an exception in pyparsing due to wrong type
pyparsing.lineno expects the second parameter to be a string.
However, in 2 locations within config_parser.py (here, and here), you set the instring parameter for a ConfigSubstitution to a boolean instead of a string, causing pyhocon to crash (instead of giving the expected error message) in cases where the key cannot be resolved.
Example:
foo: {bar: []}
foo.bar += []
And code:
import sys
from pyhocon import ConfigFactory
input = sys.stdin.read()
config = ConfigFactory.parse_string(input)
Causes:
Traceback (most recent call last):
File "test.py", line 6, in <module>
config = ConfigFactory.parse_string(input)
File "<redacted>/pyhocon/config_parser.py", line 90, in parse_string
return ConfigParser().parse(content, basedir, resolve)
File "<redacted>/pyhocon/config_parser.py", line 275, in parse
ConfigParser.resolve_substitutions(config)
File "<redacted>/pyhocon/config_parser.py", line 446, in resolve_substitutions
col=col(substitution.loc, substitution.instring)) for substitution in substitutions)))
File "<redacted>/pyhocon/config_parser.py", line 446, in <genexpr>
col=col(substitution.loc, substitution.instring)) for substitution in substitutions)))
File "<redacted>/env/lib/python2.7/site-packages/pyparsing.py", line 968, in lineno
return strg.count("\n",0,loc) + 1
AttributeError: 'bool' object has no attribute 'count'
_Aside_
Further, I'm not sure that this should be raising an exception anyway.
For example:
a = []
a += []
is allowed in the HOCON spec.
It seems to me that pyhocon should be able to resolve foo.bar. However that is another issue (possibly #97), and fixing this issue will at least make the code consistent with this case (which the HOCON spec says is equivalent):
foo: {bar: []}
foo.bar = ${?foo.bar} []
A shout out to American Fuzzy Lop which helped me find these issues.
ConfigFactory can't parse single quoted string
Config:
foo = '-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath='
Code:
from pyhocon import ConfigFactory
config = ConfigFactory.parse_file("test.conf", resolve=False)
Result:
Traceback (most recent call last):
File "converter.py", line 5, in <module>
config = ConfigFactory.parse_file("test.conf", resolve=False)
File "C:\Python27\lib\site-packages\pyhocon\config_parser.py", line 48, in parse_file
return ConfigFactory.parse_string(content, os.path.dirname(filename), resolve)
File "C:\Python27\lib\site-packages\pyhocon\config_parser.py", line 87, in parse_string
return ConfigParser().parse(content, basedir, resolve)
File "C:\Python27\lib\site-packages\pyhocon\config_parser.py", line 255, in parse
config = config_expr.parseString(content, parseAll=True)[0]
File "C:\Python27\lib\site-packages\pyparsing.py", line 1172, in parseString
raise exc
pyparsing.ParseException: Expected end of text (at char 6), (line:1, col:7)
substitution is not working if variable is set to override using environment variable
below is the example given on readme page
{
databases {
"mysql" = {
host = "abc.com" # change it
port = 3306 # default
username: scott // can use : or =
password = tiger, // can optionally use a comma
// number of retries
retries = 3
}
}
# you can use substitution with unquoted strings
retries_msg = You have ${databases.mysql.retries} retries
# retries message will be overriden if environment variable CUSTOM_MSG is set
retries_msg = ${?CUSTOM_MSG}
}
Variable retries_msg is used substitution and same is also set to be overridden using environment variable. With this config file and environment variable CUSTOM_MSG is NOT set, parse file and try to get retries_msg
>>> from pyhocon import ConfigFactory
>>> conf = ConfigFactory.parse_file('database.conf')
>>> conf.get('retries_msg')
[ConfigValues: You have ,[ConfigSubstitution: databases.mysql.retries],retries]
As shown above it fails to do the substitution. in case we do not use overriding using environment variable, it works as expected.
get_string() tries to cast unicode string to str in Python 2.x
Given the following configuration file,
a = ร
the following code fails:
from pyhocon import ConfigFactory
conf = ConfigFactory.parse_file("my_conf_file.conf")
conf.get_string('a')
with the exception:
Traceback (most recent call last):
File "test.py", line 2, in <module>
conf = ConfigFactory.parse_file("my_test_conf.conf")
File "F:\Repos\pyhocon\pyhocon\config_parser.py", line 51, in parse_file
return ConfigFactory.parse_string(content, os.path.dirname(filename), resolve)
File "F:\Repos\pyhocon\pyhocon\config_parser.py", line 90, in parse_string
return ConfigParser().parse(content, basedir, resolve)
File "F:\Repos\pyhocon\pyhocon\config_parser.py", line 272, in parse
config = config_expr.parseString(content, parseAll=True)[0]
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1607, in parseString
loc, tokens = self._parse( instring, 0 )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 3376, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 3511, in parseImpl
ret = e._parse( instring, loc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 3698, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 3900, in parseImpl
return super(ZeroOrMore, self).parseImpl(instring, loc, doActions)
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 3829, in parseImpl
loc, tokens = self_expr_parse( instring, loc, doActions, callPreParse=False )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 3511, in parseImpl
ret = e._parse( instring, loc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 3698, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 3698, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 3367, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 3511, in parseImpl
ret = e._parse( instring, loc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 3367, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 3698, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 3900, in parseImpl
return super(ZeroOrMore, self).parseImpl(instring, loc, doActions)
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 3829, in parseImpl
loc, tokens = self_expr_parse( instring, loc, doActions, callPreParse=False )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 3511, in parseImpl
ret = e._parse( instring, loc, doActions )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1405, in _parseNoCache
tokens = fn( instring, tokensStart, retTokens )
File "F:\Repos\pyhocon\env\lib\site-packages\pyparsing.py", line 1049, in wrapper
ret = func(*args[limit[0]:])
File "F:\Repos\pyhocon\pyhocon\config_parser.py", line 148, in unescape_string
return ConfigUnquotedString(norm_string(tokens[0]))
File "F:\Repos\pyhocon\pyhocon\config_tree.py", line 463, in __new__
return super(ConfigUnquotedString, cls).__new__(cls, value)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xc5' in position 0: ordinal not in range(128)
This is because you cannot coerce this unicode string to ASCII (which is done here).
I expected this call to get_string to return the unicode string, especially since the comments at the top of the function state that get_string takes a basestring as input.
parse_file() expects some extra curly brackets if next value is not a list.
Config file test.conf:
app.heap_size = 128
app.java_opts = [
-Xms${app.heap_size}m
-Xmx${app.heap_size}m
]
foo = bar
Code:
from pyhocon import ConfigFactory
config = ConfigFactory.parse_file("test.conf")
print(config['foo'])
Result:
Traceback (most recent call last):
File "<Project_path>/hocon.py", line 2, in <module>
config = ConfigFactory.parse_file("test.conf")
File "C:\Python27\lib\site-packages\pyhocon\config_parser.py", line 49, in parse_file
return ConfigFactory.parse_string(content, os.path.dirname(filename), resolve)
File "C:\Python27\lib\site-packages\pyhocon\config_parser.py", line 88, in parse_string
return ConfigParser().parse(content, basedir, resolve)
File "C:\Python27\lib\site-packages\pyhocon\config_parser.py", line 241, in parse
config = config_expr.parseString(content, parseAll=True)[0]
File "C:\Python27\lib\site-packages\pyparsing.py", line 1125, in parseString
raise exc
pyparsing.ParseSyntaxException: Expected "{" (at char 105), (line:6, col:10)
Expected result: bar
How to use pyhocon to validate hocon in cli?
Hi,
The user could validate a json file or anything could pipe in cli as:
$ echo '{"foo":1, "bar":2}' | python -m json.tool
{
"bar": 2,
"foo": 1
}
Just curious how to do it with pyhocon. Thanks.
Checking if item is in configuration
It appears that contains is not overridden in ConfigTree, so checking for the existence of a key does not work if it is dot-separated:
>>> from pyhocon import ConfigFactory
>>> t = ConfigFactory.from_dict({"test": {"test": 1}})
>>> "test.test" in t
False
>>> t.get_int("test.test")
1ConfigFactory cannot parse configuration key containing a double quote
If I use a key which contains a quoted component, the parser throws an exception.
Minimal reproductive case:
import pyhocon
pyhocon.ConfigFactory.parse_string("""
key1."key2.key3" = "result"
""")
Versions:
$ pip search pyhocon
pyhocon (0.3.19) - HOCON parser for Python
INSTALLED: 0.3.25
LATEST: 0.3.19
$ python --version
Python 2.7.11 :: Continuum Analytics, Inc.
OS: Mac OSX 10.11.1
I'm using python from a conda environment.
py26 logging has no attribute 'NullHandler'
Traceback (most recent call last):
File ".tox/py26/bin/pyhocon", line 9, in <module>
load_entry_point('pyhocon==0.3.3', 'console_scripts', 'pyhocon')()
File "/Users/bkuberek/Development/Projects/vendor/pyhocon/.tox/py26/lib/python2.6/site-packages/pyhocon/tool.py", line 175, in main
log_handler = logging.StreamHandler() if args.verbosity > 0 else logging.NullHandler()
AttributeError: 'module' object has no attribute 'NullHandler'
Config can't have keys with dots in them.
Although doesn't really affect too much, a single key can't contain dots(.). In typesafe, such cases are possible with quotes enclosed keys.
Syntax error while parsing variables of form include-blah-blah
Trying to parse the following config:
{
include-other-stuff = true
}
with conf = pyhocon.ConfigFactory.parse_file('/tmp/foo.conf') results in the following stacktrace:
File "/home/blah/blahblah.py", line 53, in <module>
conf = pyhocon.ConfigFactory.parse_file('/tmp/foo.conf')
File "/opt/anaconda3/lib/python3.4/site-packages/pyhocon/config_parser.py", line 48, in parse_file
return ConfigFactory.parse_string(content, os.path.dirname(filename), resolve)
File "/opt/anaconda3/lib/python3.4/site-packages/pyhocon/config_parser.py", line 87, in parse_string
return ConfigParser().parse(content, basedir, resolve)
File "/opt/anaconda3/lib/python3.4/site-packages/pyhocon/config_parser.py", line 269, in parse
config = config_expr.parseString(content, parseAll=True)[0]
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 1125, in parseString
raise exc
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 1115, in parseString
loc, tokens = self._parse( instring, 0 )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 989, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 2378, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 989, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 2483, in parseImpl
ret = e._parse( instring, loc, doActions )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 989, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 2624, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 989, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 2361, in parseImpl
loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 989, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 2369, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 989, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 2624, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 989, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 2739, in parseImpl
loc, tmptokens = self.expr._parse( instring, preloc, doActions )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 989, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 2483, in parseImpl
ret = e._parse( instring, loc, doActions )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 989, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/opt/anaconda3/lib/python3.4/site-packages/pyparsing.py", line 2374, in parseImpl
raise ParseSyntaxException(pe)
pyparsing.ParseSyntaxException: Expected Re:('".*?"[ \t]*') (at char 13), (line:2, col:12)
Is it possible to support python 2.6?
I would love to use HOCON for my project, but I need to work in a RHEL/CentOS environment that only supplies python 2.6. I'm not familiar with the codebase so I don't know if that's a huge effort or whether it's even desired.
Key-value inside an array throws pyparsing exception if we use = instead of :
This example:
foo = [
bar = { baz=1 }
]
With this code:
import pyhocon
conf = pyhocon.ConfigFactory.parse_file("test.conf")
Causes:
pyparsing.ParseException: Expected "]" (at char 14), (line:2, col:7)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/capi/.local/lib64/python3.4/site-packages/pyhocon/config_parser.py", line 51, in parse_file
return ConfigFactory.parse_string(content, os.path.dirname(filename), resolve)
File "/home/capi/.local/lib64/python3.4/site-packages/pyhocon/config_parser.py", line 90, in parse_string
return ConfigParser().parse(content, basedir, resolve)
File "/home/capi/.local/lib64/python3.4/site-packages/pyhocon/config_parser.py", line 272, in parse
config = config_expr.parseString(content, parseAll=True)[0]
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1613, in parseString
raise exc
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1603, in parseString
loc, tokens = self._parse( instring, 0 )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3283, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3418, in parseImpl
ret = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3605, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3811, in parseImpl
return super(ZeroOrMore, self).parseImpl(instring, loc, doActions)
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3735, in parseImpl
loc, tokens = self_expr_parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3418, in parseImpl
ret = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3605, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3605, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3274, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3418, in parseImpl
ret = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3274, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3605, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3811, in parseImpl
return super(ZeroOrMore, self).parseImpl(instring, loc, doActions)
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3735, in parseImpl
loc, tokens = self_expr_parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3418, in parseImpl
ret = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3605, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3279, in parseImpl
raise ParseSyntaxException._from_exception(pe)
pyparsing.ParseSyntaxException: Expected "]" (at char 14), (line:2, col:7)
Whereas if we use : instead:
foo = [
bar: { baz=1 }
]
With the same code, we get a different outcome:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/capi/.local/lib64/python3.4/site-packages/pyhocon/config_parser.py", line 51, in parse_file
return ConfigFactory.parse_string(content, os.path.dirname(filename), resolve)
File "/home/capi/.local/lib64/python3.4/site-packages/pyhocon/config_parser.py", line 90, in parse_string
return ConfigParser().parse(content, basedir, resolve)
File "/home/capi/.local/lib64/python3.4/site-packages/pyhocon/config_parser.py", line 272, in parse
config = config_expr.parseString(content, parseAll=True)[0]
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1603, in parseString
loc, tokens = self._parse( instring, 0 )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3283, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3418, in parseImpl
ret = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3605, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3811, in parseImpl
return super(ZeroOrMore, self).parseImpl(instring, loc, doActions)
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3735, in parseImpl
loc, tokens = self_expr_parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3418, in parseImpl
ret = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3605, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3605, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3274, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3418, in parseImpl
ret = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3274, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3605, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3811, in parseImpl
return super(ZeroOrMore, self).parseImpl(instring, loc, doActions)
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3735, in parseImpl
loc, tokens = self_expr_parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3418, in parseImpl
ret = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3605, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3266, in parseImpl
loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3274, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3605, in parseImpl
return self.expr._parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3274, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3811, in parseImpl
return super(ZeroOrMore, self).parseImpl(instring, loc, doActions)
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3735, in parseImpl
loc, tokens = self_expr_parse( instring, loc, doActions, callPreParse=False )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1375, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 3274, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "/usr/lib64/python3.4/site-packages/pyparsing.py", line 1381, in _parseNoCache
tokens = self.postParse( instring, loc, tokens )
File "/home/capi/.local/lib64/python3.4/site-packages/pyhocon/config_parser.py", line 474, in postParse
return [config_values.transform()]
File "/home/capi/.local/lib64/python3.4/site-packages/pyhocon/config_tree.py", line 411, in transform
col=col(self._loc, self._instring)))
pyhocon.exceptions.ConfigWrongTypeException: Token 'ConfigTree([('baz', 1)])' of type ConfigTree (index 1) must be of type str (line: 2, col: 1)
Granted, both are errors. I am still learning HOCON, so I'm assuming you just can't place a key-value pair inside an array, which actually makes sense. However, the pyparsing exception leaking out makes it harder for someone that uses pyhocon, since we have to catch not only pyhocon.exceptions.ConfigException, but also pyparsing stuff (which means we have to import pyparsing in our own code).
Surprisingly, with_fallback() mutates parameter
It's not in the docs proper, but with_fallback() is described in the readme:
Usage:
config3 = config1.with_fallback(config2)orconfig3 = config1.with_fallback('samples/aws.conf')
I read this to mean that config3 is a new object, independent of config1 or config2โa copy.
That's not what happens, though. Actually, config3 is config2!
For example:
In [1]: from pyhocon import ConfigTree
In [2]: a = ConfigTree()
In [3]: a.put('foo.bar', ['a'])
In [4]: b = ConfigTree()
In [5]: b.put('foo.bar', ['b'])
In [6]: a
Out[6]: ConfigTree([('foo', ConfigTree([('bar', ['a'])]))])
In [7]: b
Out[7]: ConfigTree([('foo', ConfigTree([('bar', ['b'])]))])
In [8]: c = a.with_fallback(b)
In [9]: a
Out[9]: ConfigTree([('foo', ConfigTree([('bar', ['a'])]))])
In [10]: b
Out[10]: ConfigTree([('foo', ConfigTree([('bar', ['a'])]))])
In [11]: c
Out[11]: ConfigTree([('foo', ConfigTree([('bar', ['a'])]))])
In [12]: c is a
Out[12]: False
In [13]: c is b
Out[13]: True
I would expect a method on a to either return a copy of a, or mutate a (and return None). Mutating and returning b is quite surprising and at the least should be called out explicitly in the docs! Returning a new ConfigTree instance would be best, though.
String substitution broken?
Could it be that the fix for object merging (issue #10) broken string substitution?
Expanding on the simple unit test from pyhocon codebase:
In [54]: config = ConfigFactory.parse_string(
....: """
....: {
....: a: {
....: b: {
....: c = buzz
....: }
....: }
....: d = test ${a.b.c} me
....: e = ${a.b.c} me
....: }
....: """
....: )
In [55]: config['d']
Out[55]: 'test buzzme'
In [56]: config['e']
Out[56]: 'buzz'
Whereas I expected to get test buzz me (with the space between subst expr and the last string) and buzz me (here the appended string got lost entirely)
Let me know if I'm misunderstanding how string subst is supposed to work.
Single quotes are being read as part of a string.
test.conf:
foo = 'bar'
Code:
from pyhocon import ConfigFactory
config = ConfigFactory.parse_file("test.conf")
print config['foo']
Result: 'bar'
Is it a bug or a feature?
A way to update configuration using pyhocon
Let's say I've got some property-like configuration:
server.property1 = ${foo.bar}
server.property2 = ${baz}
And I want to add another two properties without resolving existing variables
server.property2 = replacement
server.property3 = 100500
overwriting existing ones values.
So I wrote following code:
from pyhocon import ConfigFactory
from pyhocon import ConfigTree
from pyhocon import HOCONConverter
config = ConfigFactory.parse_file("server.properties", resolve=False)
ConfigTree.put(config,"server.property3","100500")
ConfigTree.put(config,"server.property2","replacement")
print HOCONConverter.to_properties(config)
And the result is almost good, but it prints out object names:
server.property1 = [ConfigValues: [ConfigSubstitution: foo.bar]]
server.property2 = replacement
server.property3 = 100500
Is there any way I can get my ${foo.bar} back?
Wrong concat of quoted strings
Pyhocon ignores white spaces between quoted strings during string concatenation.
Given the following config:
a = "foo" "bar"The Java implementation gives an object that, in JSON looks like:
{
"a" : "foo bar"
}while pyhocon gives:
{
"a" : "foobar"
}Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.