tolitius/cprop

likes properties, environments, configs, profiles..

cprop's Issues

`load-config :merge` option not merging in specified order

I'm trying to get my config.edn values to override the environment variables but this seems impossible? It doesn't seem to matter what order I specify the arguments to merge.

(def bad (load-config
           :merge
           [(mount/args)
            (source/from-system-props)
            (source/from-env)
            (source/from-file "config.edn")]))

I even tried to merge things myself and it still overrides w/ the environment variables:

(def still-bad (load-config
                             :merge
                             [(apply merge [(mount/args)
                                            (source/from-system-props)
                                            (source/from-env)
                                            (source/from-file "config.edn")])]))

Or specified the file:

(def nope (load-config
                             :file "config.edn"
                             :merge
                             [(mount/args)
                              (source/from-system-props)
                              (source/from-env)]))

Is this a bug as it obviously seems to be or just an impossibility with the current system? The docs says it's composable, but clearly, that does not refer to respecting the order property sources or not overwriting them w/ env variables. Why even have a :merge option if it doesn't do anything different than calling (load-config) with no args?

How to update the java env settings?

This isn't a cprop problem, but I think this might be the best place to get a simple answer. I want to use env vars in my app, hence cprop; however, once I add something to my bashrc and source it, I still don't seem to see it in any clojure app until I've restarted my entire system; I've tried restarting the REPL, restarting the shell, closing down all java apps, etc, and none of these pulls in the vars. Surely I don't need to restart every time to get java to see my new env vars! What do I need to do?

Thanks!

Support For Boolean Types?

Hi!

I am using cprop for a project and I was wondering if it could have support for coercing ENV/System Property values to boolean types.

For example:

DEBUG=true lein test

Should set DEBUG to true (right now it sets it to "true")

Feature Request: EDN to Properties class

As a developer who embeds Java frameworks into Clojure enclosure (pun intended) I would like to be able to convert EDN into java.util.Properties in memory.

So in addition to the existing route via file:

(t/map->props-file config)
"/tmp/cprops-1475854845508-538388633502378948.tmp"

I would appreciate having a separate function to get the Object:

(t/map->props config)

same file overrides

Just like Nomad: https://github.com/jarohen/nomad#grouping-hosts-together---environments-041

I like this feature a lot because it means I don't have to spread my config around unnecessarily. At the moment I work around this by doing:

(cprop/load-config :resource "defaults.edn"
                   :merge [(get-in (from-resource "defaults.edn") [:environments "prod"] {})])

But of course defaults.edn gets read and parsed twice. Not the end of the world since load-config is only called once anyway, but would be nice to have this feature supported if possible.

Cheers!

Support for ~ shorthand in load-config :file

For example:

(is (= (load-config :file (str (System/getenv "HOME") "/.secrets.edn"))
       (load-config :file "~/.secrets.edn")))

aero supports this, but I prefer cprop's cursors over aero's reader tags.

Composable Cursors

Please consider making 'cursor' composable. Currently it's not possible to do the following:

(def foo (cursor :foo))
(def bar (cursor foo :bar))

It would really simplify my workflow if that was the case.

`cprop.tools/contains-in?` doesn't work with vectors

Given the following config:

{:zookeeper {:hosts ["localhost:2181"]}}

If you attempt to override the localhost string with ZOOKEEPER__HOSTS__0 (by using the cprop.tools/parse-num-keys key parse fn), nothing will happen.

This is because cprop.tools/contains-in? assumes it is only working with maps.

Pluggable ":format"

Currently the default configuration format is EDN, but cprop also knows how to work with .properies, so it make sense to add a reader for that.

Also nothing stops cprop to later know YAML or JSON or somethin more exotic, etc..

Needs more thinking and would probably look something like:

(load-config :format :lua)

(load-config :format :yaml)

with :edn be a default format.

(needs to be pluggable, so it is easy for others to contribute more formats)

Substitution with falsey values

Let's say I have a config.edn like so:

{:a "a"
 :b true
 :c false
 :d nil}

And I'm doing a plain (load-config) with no special options set.

By setting environment variables, I can only substitute :a and :b. Both :c and :d are ignored, and I can't seem to substitute them through environment variables no matter what I do. I'm assuming this is happening because both :c and :d are falsey, but this is not intended behavior is it? I can't imagine that it is, but I just wanted to check.

Optional :strings-only restriction

There is a valid security concern to read configuration as EDN:

what if someone injects bad EDN that would steal all the money?

While in most cases it won't be a problem, in some cases, for example where apps eval pieces of configuration, it could be risky to "leave the door open".

Add an optional :strings-only restriction, which would read everything as strings, while still enjoying the hierarchy of EDN.

cprop cannot handle setting nils

So it turns out cprop cannot handle niling a setting. After some looking around and some testing I think the problem is just one of unnecessary complexity:

The str->value function tries to be smart and does some long/boolean/string regex matching before sending the value to edn/read-string. This processing is both unnecessary and can actually limit functionality as mentioned in the title:

cprop.source> (let [or-nil #(or % "nil")]
                (doseq [v ["hello" -1 100 100000000000000000000 -100000000000000000000 nil true false]
                        :let [cprop (str->value (str v) {})
                              edn (edn/read-string (str v))]]
                  (clojure.pprint/print-table [:parser :value :type]
                                              [{:parser "" :value (or-nil (str v)) :type (or-nil (type v))}
                                               {:parser "cprop" :value (or-nil cprop) :type (or-nil (type cprop))}
                                               {:parser "edn" :value (or-nil edn) :type (or-nil (type edn))}])))

| :parser | :value |                     :type |
|---------+--------+---------------------------|
|         |  hello |    class java.lang.String |
|   cprop |  hello |    class java.lang.String |
|     edn |  hello | class clojure.lang.Symbol |

| :parser | :value |                :type |
|---------+--------+----------------------|
|         |     -1 | class java.lang.Long |
|   cprop |     -1 | class java.lang.Long |
|     edn |     -1 | class java.lang.Long |

| :parser | :value |                :type |
|---------+--------+----------------------|
|         |    100 | class java.lang.Long |
|   cprop |    100 | class java.lang.Long |
|     edn |    100 | class java.lang.Long |

| :parser |                :value |                     :type |
|---------+-----------------------+---------------------------|
|         | 100000000000000000000 | class clojure.lang.BigInt |
|   cprop | 100000000000000000000 | class clojure.lang.BigInt |
|     edn | 100000000000000000000 | class clojure.lang.BigInt |

| :parser |                 :value |                     :type |
|---------+------------------------+---------------------------|
|         | -100000000000000000000 | class clojure.lang.BigInt |
|   cprop | -100000000000000000000 | class clojure.lang.BigInt |
|     edn | -100000000000000000000 | class clojure.lang.BigInt |

| :parser | :value |                  :type |
|---------+--------+------------------------|
|         |    nil |                    nil |
|   cprop |        | class java.lang.String |
|     edn |    nil |                    nil |

| :parser | :value |                   :type |
|---------+--------+-------------------------|
|         |   true | class java.lang.Boolean |
|   cprop |   true | class java.lang.Boolean |
|     edn |   true | class java.lang.Boolean |

| :parser | :value |                   :type |
|---------+--------+-------------------------|
|         |  false | class java.lang.Boolean |
|   cprop |    nil | class java.lang.Boolean |
|     edn |    nil | class java.lang.Boolean |
nil

I'm fairly certain simply removing the regex matching from str->value would only add functionality to the parsing. Anecdotally: before babashka supported cprop as a library, I mimic'd the behavior using just edn/read-string and it worked like a charm.

load-config doesn't work with only :merge (or no key)

Simply reproduce with:

$ lein try cprop
=> (use 'cprop.core)
=> (load-config :merge [{:foo "bar"}])
RuntimeException could not find a configuration file to load. looked in the classpath (as a "resource") and on a file system via "conf" system property  cprop.core/load-config (core.clj:24)

I guess this is due to ignore-missing-default returning an empty map when no :file or :resource are provided.
However the doc has examples like (load-config :merge [...]).
Moreover it is also stated that simply (load-config) should work.

Confusing readme.

here
I'm told that underscores are used for nesting

a system property http_pool_socket.timeout would point to a {:http {:pool {:socket-timeout value}}}

Then I'm told than the underscore is converted to a dash.

The only way to mimic the structure is via use of an underscore character. The _ is converted to - by cprop, so instead, to identify nesting, two underscores can be used.

Low priority bug : `cprop.tool/map->env-file` doesn't support dotted namespaced keys

(cprop.tools/map->env-file {:composite.key/namespaced 43})
=> "/tmp/cprops-1629906434989-8080623925560935467.tmp"
(println (slurp *1))
export NAMESPACED=43
=> nil

The content of the file should be :

export COMPOSITE.KEY___NAMESPACED=43

Empty files cause issues while loading

When using cprop as part of the luminus framework, there is a config.edn for each environment: dev, test, and prod. The files generated by luminus are mostly empty, and the developer can add settings if needed.

When pushing to prod on Heroku, however, these empty files cause a problem. We get an error saying a non-empty config file was not found. The file is present, but has an empty map {}. The workaround is to put a redundant or meaningless value in the map, but it seems the system should handle an empty map.

:env key?

With Leiningen, I was able to embed variables into my project.clj by using a top-level project.clj :env key with a map of variables as it's value. I believe the same key works in profiles. But I couldn't find this documented anywhere except in environ docs: https://github.com/weavejester/environ. Is this a Leiningen core functionality? Should we add this to the README for cprop?

Add CLI as a source of configuration data

It is useful to be able to supply and/or override configuration information from the command line. This small gist is the snippet I use to merge options taken from CLI into configuration loaded by cprop. It interprets CLI options containing double hyphens as indicating nesting, analogous to how cprop treats environment variables. I would like to see a (from-cli) source added, and will submit a PR if that helps.

Empty EDN causes misleading exception

When my conf file was completely empty (no {} even), I got the very perplexing error:
Caused by: java.lang.RuntimeException: could not find a configuration file to load. looked in the classpath (as a "resource") and on a file system via "conf" system property

I believe this is because the deep merging fails when either the resource or default file is empty.

Not allowing an empty EDN file seems fine to me, but perhaps a specific error message is warranted for this case.

Non-numeric values are parsed as numbers

Given an environment variable with the value 7 Nov 22:44:53 2015, then cprop.source/read-system-env accidentally parses it as the number 7 instead of keeping it as a string.

parse-runtime-args doesn't exist in the codebase

Readme uses this function, so i assumed it existed

i couldn't find it, but i found these

       (read-system-props opts)
        (read-system-env opts)

Question marks in config values - from ENV -> cprop ?

I've built an app that has my dev configuration with something like the following:

{ :web {:port 8080
       :basic-auth? false}}

Upon time-to-deploy I've learned I can't have an environment variable with a ? in it...

I know I can "nest maps" with the double underscore trick (and that works great) but is there anyway to convey a config with a question mark from the ENV into my config?

If not, it seems I should refactor my app so that any config variables with question marks aren't present?

I was hoping to do something like this, but now i'm realizing i might have some config surgery to do:

WEB__BASIC_AUTH\?=true java -jar my-uberjar.jar

Maybe i shouldn't have used question marks for "flag" type configuration values...

any tips appreciated, thanks in advance.

why env has a higher priority than command line arguments?

For example here:
https://github.com/tolitius/cprop#merging-with-all-system-and-env

             :file "/path/to/some.edn"
             :merge [{:datomic {:url "foo.bar"}}
                     (from-file "/path/to/another.edn")
                     (from-resource "path/within/classpath/to-another.edn")
                     (parse-runtime-args ...)
                     (from-props-file "/path/to/some.properties")
                     (from-system-props)
                     (from-env)])```

How to make cli have the highest precedence?

Add deps.edn - for git consumption ?

Hi,

Would you be open to adding a deps.edn for this library so that it can be consumed from git ?
I found that deps.edn makes it easy to experiment / use other branches.

CLJC files are incompatible with ClojureScript

I'm seeing use of java-only features, so I'm confused as to why the sources are in .cljc. Is clojurescript support planned?

handle non-existent (or empty) configurations

This is actually not an issue but rather an ask for advice. I'd like to implement scenario where application could be configured by hierarchical (merged at the end) resources, eg:

app.edn
app-dev.edn
app-prod.edn

In this case app.config would contain all properties common for -dev and -prod environments.

The problem is that for each environment I would like to have these files optional, eg. to have no app-dev.config for development environment (so only properties from app.edn would get loaded) or to not have app.edn at all (so only environment-specific resources would be loaded).

What I tried was something like this:

(cprop/load-config :resource "app.edn"
                   :merge [(from-resource "app-dev.edn"))])

but it fails when any of resources does not exist or is empty. Is there any way to get my scenario working?

Thanks.

Converting to properties (t/map->properties) fails with Mac environmental variables

(def config (load-config :resource "config.edn"
                         :merge [(from-env)]))

(defn config-properties []
        (t/map->properties config))

Results in:

Exception in thread "main" java.lang.IllegalArgumentException: Malformed \uxxxx encoding., compiling:(/private/var/folders/hl/dz69nmk50kl_cn4fgd904dwh0000gn/T/form-init5308382940781279132.clj:1:125)
	at clojure.lang.Compiler.load(Compiler.java:7391)
	at clojure.lang.Compiler.loadFile(Compiler.java:7317)
	at clojure.main$load_script.invokeStatic(main.clj:275)
	at clojure.main$init_opt.invokeStatic(main.clj:277)
	at clojure.main$init_opt.invoke(main.clj:277)
	at clojure.main$initialize.invokeStatic(main.clj:308)
	at clojure.main$null_opt.invokeStatic(main.clj:342)
	at clojure.main$null_opt.invoke(main.clj:339)
	at clojure.main$main.invokeStatic(main.clj:421)
	at clojure.main$main.doInvoke(main.clj:384)
	at clojure.lang.RestFn.invoke(RestFn.java:421)
	at clojure.lang.Var.invoke(Var.java:383)
	at clojure.lang.AFn.applyToHelper(AFn.java:156)
	at clojure.lang.Var.applyTo(Var.java:700)
	at clojure.main.main(main.java:37)
Caused by: java.lang.IllegalArgumentException: Malformed \uxxxx encoding.
	at java.util.Properties.loadConvert(Properties.java:574)
	at java.util.Properties.load0(Properties.java:391)
	at java.util.Properties.load(Properties.java:317)
	at cprop.tools$map__GT_properties.invokeStatic(tools.cljc:76)
	at cprop.tools$map__GT_properties.invoke(tools.cljc:71)

If "(from-env)" is removed, everything works. I suspect that parsing of one of the env variables might be causing this. I'm on Mac.

I am attaching a code sample to reproduce.

The output included as result.txt is from the first call that works: semi - custom logic, based on other private methods in the lib to convert config to Properties.

issue-demo.zip

Different behavior of System/setProperty vs importing from env

When in my repl use,

(System/setProperty "FTTV_DB_classname" "value") I get what I want from cprop:
{:fttv {:db {:classname "value"}}}

However, with the following in my bashrc, I have a problem:
export FTTV_DB_CLASSNAME='org.postgresql.Driver'

Results in {:fttv-db-classname "org.postgresql.Driver"}

What's going on?

enhancement to `:as-is?` for specific value conversions

:as-is? was introduced in a previous issue as a solution to handle situations where number conversion is attempted if a value is a date or an ip address. #27
But I think it is still useful to preserve the normal parsing behavior of cprop for all other values in my env or property edn files.
I proposed a way to allow this using :as-is-paths which is a set of paths that can be provided for which if the property keys match, then no value conversion is done. All other values are parsed in the normal manner.
See PR #40

Merging a nil value nils the entire config

If you pass a nil value when using the :merge keyword, the entire returned config is nil.

user> (require '[cprop.core :as cprop])
nil
user> (cprop/load-config)
{:hello :world, :foo "bar"}
user> (cprop/load-config :merge [nil])
nil

Given Clojure's handling of nil, the above result feels wrong. Below is a less contrived example:

user> (require '[cprop.source :as source]
               '[clojure.java.io :as io])
nil
user> (defn optional-resource [path]
        (when (io/reader path)
          (source/from-resource path)))
nil
user> (cprop/load-config :merge [(optional-resource "maybe/a/missing/file.edn")])
nil

Possible solutions:

1. (remove nil? merge) here.
   This is a super simple change, however it doesn't alert the user that one of their resources isn't getting merged. This can lead to confusion if the config was expected to exist and be merged.
2. Allow ignore-missing-default to return an empty map when providing a source.
   This could be a new arity or just a removal of the (if source ...) conditional.

(def ignore-missing-default
  ([f source]
    (ignore-missing-default f source true))
  ([f source ignore-missing-source?]
    (try
     (f source)
     (catch MissingResourceException mre
       (when-not ignore-missing-source?
         (throw mre)
       (log/infof "Ignoring missing config: %s" source)
       {})

better match the behavior of `assoc-in`

Currently cprop assumes that all parts of a key path are keywords. Because of this assumption, only nested maps may be overridden by ENV vars, System properties, et al.

However, cprop uses assoc-in to make these updates which can support indices if the current root level is a non-map:

user=> (def servers {:redis [{:url "hi"}]
  #_=>               :es [{:url "world"}
  #_=>                    {:url "!"}]})
#'user/servers
user=> (assoc-in servers [:redis 0 :url] "hello")
{:redis [{:url "hello"}], :es [{:url "world"} {:url "!"}]}

I propose one of two options:

1. If a part of a path matches #"^\d+$" then it should be treated as a long instead of a keyword to better align with assoc-in's behavior.
2. Provide a way to inject a parsing function into the k->path function so users can parse paths in ways they would expect.

Don't use println for status reporting

In a couple of places cprop uses (println) for status logging, but it would be better if it used a logging framework (e.g. tools.logging).

Amongst other concerns of unconfigurable output to stdout, this messes with logging within Docker containers.