gpoulin/pyhocon

HOCON parser for Python

pyhocon

HOCON parser for Python

Specs

https://github.com/typesafehub/config/blob/master/HOCON.md

Installation

It is available on pypi so you can install it as follows:

$ pip install pyhocon

Usage

The parsed config can be seen as a nested dictionary (with types automatically inferred) where values can be accessed using normal dictionary getter (e.g., conf['a']['b'] or using paths like conf['a.b']) or via the methods get, get_int (throws an exception if it is not an int), get_string, get_list, get_float, get_bool, get_config.

from pyhocon import ConfigFactory

conf = ConfigFactory.parse_file('samples/database.conf')
host = conf.get_string('databases.mysql.host')
same_host = conf.get('databases.mysql.host')
same_host = conf['databases.mysql.host']
same_host = conf['databases']['mysql.host']
port = conf['databases.mysql.port']
username = conf['databases']['mysql']['username']
password = conf.get_config('databases')['mysql.password']
password = conf.get('databases.mysql.password', 'default_password') #  use default value if key not found

Example of HOCON file

//
// You can use # or // for comments
//
{
  databases {
    # MySQL
    active = true
    enable_logging = false
    resolver = null
    # you can use substitution with unquoted strings. If it it not found in the document, it defaults to environment variables
    home_dir = ${HOME} # you can substitute with environment variables
    "mysql" = {
      host = "abc.com" # change it
      port = 3306 # default
      username: scott // can use : or =
      password = tiger, // can optionally use a comma
      // number of retries
      retries = 3
    }
  }

  // multi line support
  motd = """
            Hello "man"!
            How is it going?
         """
  // this will be appended to the databases dictionary above
  databases.ips = [
    192.168.0.1 // use white space or comma as separator
    "192.168.0.2" // optional quotes
    192.168.0.3, # can have a trailing , which is ok
  ]

  # you can use substitution with unquoted strings
  retries_msg = You have ${databases.mysql.retries} retries

  # retries message will be overriden if environment variable CUSTOM_MSG is set
  retries_msg = ${?CUSTOM_MSG}
}

// dict merge
data-center-generic = { cluster-size = 6 }
data-center-east = ${data-center-generic} { name = "east" }

// list merge
default-jvm-opts = [-XX:+UseParNewGC]
large-jvm-opts = ${default-jvm-opts} [-Xm16g]

Conversion tool

We provide a conversion tool to convert from HOCON to the JSON, .properties and YAML formats.

usage: tool.py [-h] [-i INPUT] [-o OUTPUT] [-f FORMAT] [-n INDENT] [-v]

pyhocon tool

optional arguments:
  -h, --help                 show this help message and exit
  -i INPUT, --input INPUT    input file
  -o OUTPUT, --output OUTPUT output file
  -c, --compact              compact format
  -f FORMAT, --format FORMAT output format: json, properties, yaml or hocon
  -n INDENT, --indent INDENT indentation step (default is 2)
  -v, --verbosity            increase output verbosity

If -i is omitted, the tool will read from the standard input. If -o is omitted, the result will be written to the standard output. If -c is used, HOCON will use a compact representation for nested dictionaries of one element (e.g., a.b.c = 1)

JSON

$ cat samples/database.conf | pyhocon -f json

{
  "databases": {
    "active": true,
    "enable_logging": false,
    "resolver": null,
    "home_dir": "/Users/darthbear",
    "mysql": {
      "host": "abc.com",
      "port": 3306,
      "username": "scott",
      "password": "tiger",
      "retries": 3
    },
    "ips": [
      "192.168.0.1",
      "192.168.0.2",
      "192.168.0.3"
    ]
  },
  "motd": "\n            Hello \"man\"!\n            How is it going?\n         ",
  "retries_msg": "You have 3 retries"
}

.properties

$ cat samples/database.conf | pyhocon -f properties

databases.active = true
databases.enable_logging = false
databases.home_dir = /Users/darthbear
databases.mysql.host = abc.com
databases.mysql.port = 3306
databases.mysql.username = scott
databases.mysql.password = tiger
databases.mysql.retries = 3
databases.ips.0 = 192.168.0.1
databases.ips.1 = 192.168.0.2
databases.ips.2 = 192.168.0.3
motd = \
            Hello "man"\!\
            How is it going?\

retries_msg = You have 3 retries

YAML

$ cat samples/database.conf | pyhocon -f yaml

databases:
  active: true
  enable_logging: false
  resolver: None
  home_dir: /Users/darthbear
  mysql:
    host: abc.com
    port: 3306
    username: scott
    password: tiger
    retries: 3
  ips:
    - 192.168.0.1
    - 192.168.0.2
    - 192.168.0.3
motd: |

            Hello "man"!
            How is it going?

retries_msg: You have 3 retries

Includes

We support the include semantics using one of the followings:

include "test.conf"
include "http://abc.com/test.conf"
include "https://abc.com/test.conf"
include "file://abc.com/test.conf"
include file("test.conf")
include required(file("test.conf"))
include url("http://abc.com/test.conf")
include url("https://abc.com/test.conf")
include url("file://abc.com/test.conf")

When one uses a relative path (e.g., test.conf), we use the same directory as the file that includes the new file as a base directory. If the standard input is used, we use the current directory as a base directory.

For example if we have the following files:

cat.conf:

{
  garfield: {
    say: meow
  }
}

dog.conf:

{
  mutt: {
    say: woof
    hates: {
      garfield: {
        notes: I don't like him
        say: meeeeeeeooooowww
      }
      include "cat.conf"
    }
  }
}

animals.conf:

{
  cat : {
    include "cat.conf"
  }

  dog: {
    include "dog.conf"
  }
}

Then evaluating animals.conf will result in the followings:

$ pyhocon -i samples/animals.conf
{
  "cat": {
    "garfield": {
      "say": "meow"
    }
  },
  "dog": {
    "mutt": {
      "say": "woof",
      "hates": {
        "garfield": {
          "notes": "I don't like him",
          "say": "meow"
        }
      }
    }
  }
}

As you can see, the attributes in cat.conf were merged to the ones in dog.conf. Note that the attribute "say" in dog.conf got overwritten by the one in cat.conf.

Duration/Period support

Difference from HOCON spec

• nanoseconds supported only in the sense that it is converted to microseconds with lowered accuracy (divided by 1000 and rounded to int).
• m suffix only applies to minutes. Spec specifies that m can also be applied to months, but that would cause a conflict in syntax.
• months and years only available if dateutils is installed (relativedelta is used instead of timedelta).

Misc

with_fallback

• with_fallback: Usage: config3 = config1.with_fallback(config2) or config3 = config1.with_fallback('samples/aws.conf')

from_dict

d = OrderedDict()
d['banana'] = 3
d['apple'] = 4
d['pear'] = 1
d['orange'] = 2
config = ConfigFactory.from_dict(d)
assert config == d

TODO

Items	Status
Comments	✅
Omit root braces	✅
Key-value separator	✅
Commas	✅
Whitespace	✅
Duplicate keys and object merging	✅
Unquoted strings	✅
Multi-line strings	✅
String value concatenation	✅
Array concatenation	✅
Object concatenation	✅
Arrays without commas	✅
Path expressions	✅
Paths as keys	✅
Substitutions	✅
Self-referential substitutions	✅
The += separator	✅
Includes	✅
Include semantics: merging	✅
Include semantics: substitution	✅
Include semantics: missing files	❌
Include semantics: file formats and extensions	❌
Include semantics: locating resources	❌
Include semantics: preventing cycles	❌
Conversion of numerically-index objects to arrays	✅

API Recommendations	Status
Conversion of numerically-index objects to arrays	❌
Automatic type conversions	❌
Units format	❌
Duration format	❌
Size in bytes format	❌
Config object merging and file merging	❌
Java properties mapping	❌

Contributors

• Aleksey Ostapenko (@kbabka)
• Martynas Mickevičius (@2m)
• Joe Halliwell (@joehalliwell)
• Tasuku Okuda (@okdtsk)
• Uri Laserson (@laserson)
• Bastian Kuberek (@bkuberek)
• Varun Madiath (@vamega)
• Andrey Proskurnev (@ariloulaleelay)
• Michael Overmeyer (@movermeyer)
• Virgil Palanciuc (@virgil-palanciuc)
• Douglas Simon (@dougxc)
• Gilles Duboscq (@gilles-duboscq)
• Stefan Anzinger (@sanzinger)
• Ryan Van Gilder (@ryban)
• Martin Kristiansen (@lillekemiker)
• Yizheng Liao (@yzliao)
• atomerju (@atomerju)
• Nick Gerow (@NickG123)
• jjtk88 (@jjtk88)
• Aki Ariga (@chezou)
• Joel Grus (@joelgrus)
• Anthony Alba @aalba6675
• hugovk @hugovk
• chunyang-wen @chunyang-wen
• afanasev @afanasev
• derkcrezee @derkcrezee
• Roee Nizan @roee-allegro
• Samuel Bible @sambible
• Christophe Duong @ChristopheDuong
• lune* @lune-sta
• Sascha @ElkMonster
• Tomas Witzany @Tommassino
• Gabriel Shaar @gabis-precog
• Brandon Martin @bdmartin
• Bryan Richter @chreekat
• dtarakanov1 @dtarakanov

Thanks

• Agnibha (@agnibha)
• Ernest Mishkin (@eric239)
• Alexey Terentiev (@alexey-terentiev)
• Prashant Shewale (@pvshewale)
• mh312 (@mh321)
• François Farquet (@farquet)
• Gavin Bisesi (@Daenyth)
• Cosmin Basca (@cosminbasca)
• cryptofred (@cryptofred)
• Dominik1123 (@Dominik1123)
• Richard Taylor (@richard534)
• Sergii Lutsanych (@sergii1989)