juxt/aero

A small library for explicit, intentful configuration.

aero's Issues

#or issue with ^:ref

Piggybacking off #17 , I think it would make sense for the following to work:

config.edn

{:test-val nil
 :actual-val #or [^:ref [:test-val] "abc123"]}

Result:

(aero/read-config (io/resource "config.edn") {:profile  (keyword (env/env :mode))
                                                                       :resolver aero/resource-resolver})
=> {:test-val nil, :actual-val nil}

cannot use ^:ref inside #join

I am trying to share a port value to avoid inadvertently using updating it in one place and forgetting others.

It seems that the following does not work, is it in the scope of what aero covers?

{
  :port 123
  :host-string #join ["127:0.0.1:" ^:ref [:port]]
}

In this case I end up with this config:

{:host-string 127:0.0.1:[:port]}

EDIT:
Looks like I need to update aero #23

Friendly error message on missing include

If I #include a non-existing file I get the following stack trace:

Exception in thread "main" java.lang.IllegalArgumentException: Cannot open <nil> as a Reader.
	at clojure.java.io$fn__10992.invokeStatic(io.clj:288)
	at clojure.java.io$fn__10992.invoke(io.clj:288)

My use-case is a project template with database credentials stored in a private file that needs to be created by the user.
What I would like is a friendly message saying something like "File secrets.edn does not exist - please make a copy of secrets.edn.template and enter your password"...or something similar.

Pass values into config?

I'm using aero to construct Nomad job specs. Those maps can get fairly large with lists and nesting making it cumbersome to inject data after read-config has returned a value.

In this situation I found it easier to spread little "markers" in the .edn source which I'm currently doing with the custom reader shown below.

I was wondering if there might be an easier way to achieve this with Aero. Maybe it would also be an option to include similar functionality in Aero itself if there isn't.

There are some similarities to the existing include resolver. option, maybe this could end up being a more generalised form of the same.

(def ^:dynamic *deploy-opts*)

(defmethod aero/reader `opt
  [_ tag value]
  (or (get *deploy-opts* value)
      (throw (ex-info (str "could not find deploy opt for " value)
                      {:deploy-opts *deploy-opts*}))))

(binding [*deploy-opts* {:docker-tag "latest"}]
  (aero/read-config jobspec))

{:Job
   {:Datacenters ["dc1"],
    :ID "cljdoc",
    :Name "cljdoc",
    :TaskGroups
      [{:Count 1,
        :Name "cljdoc",
        :RestartPolicy {:Attempts 2,
                        :Delay 15000000000,
                        :Interval 1800000000000,
                        :Mode "fail"},
        :Tasks
          [{:Config {:image #join ["cljdoc/cljdoc:" #cljdoc.deploy/opt :docker-tag],
                     :port_map [{:http 8000}]}}]}]}}

v1.1.4 regression on parsing #ref inside of #or

On v1.1.4 a config file containing:

{:a "a"
 :b #or [#ref [:a] "I don't know what :a is"]
 :c #or [#ref [:d] "I don't know what :d is"]
 :d "d"
 :e #or [#ref [:z] "I don't know what :z is"]}

yields when parsed using aero/read-config

{:a "a", :b "a", :c nil, :d "d", :e nil}

Regressions:

• :c is not parsed correctly despite :d being in the map (although after :c)
• the default value in the #or literal for :e is not returned

On v1.1.3 the same config file yields

{:a "a", :b "a", :c "d", :d "d", :e "I don't know what :z is"}

I view this as the expected result.

What is the recommended way to override (e.g., from a CLI) config values that are used by #ref?

Let's say I have a config.edn such as

{:port #or [#env PORT 3000]
 :server {:port #ref [:port]}
 :client {:port #ref [:port]}
}

When I run the app, I give it the port on the command line like app --port 5000.
Inside the app, I have the CLI options as a map {:port 5000}.
I could do something like:

(def opts {:port 5000})
(def raw-config (aero/read-config "config.edn"))
(def config (-> raw-config 
                (merge opts)
                (update-in [:server] merge opts)
                (update-in [:client] merge opts)))

Is there a better way?

make #ref from parent work inside #include file

At the moment #include works by reading the config on the file:

aero/src/aero/core.cljc

Line 86 in 1749e71

(read-config

This breaks #ref that are in another file "higher" up in the include hierarchy, ex:

(aero.core/read-config (clojure.java.io/resource "config_test.edn"))
;; =>
{:a "AAA"
 :b "AAA"
 :c {:d nil} ;; nil here - should be "AAA"
 }

;; config_test.edn
{:a "AAA"
 :b #ref [:a] ;; OK
 :c #include "config_include.edn"
 }

;;config_include.edn 
{:d #ref [:a]} ;; this is not resolved

Without knowing too much about aero's internals, it seems a possible solution would be to include .edn files without resolving the tags at the 'include level.

#merge #ref fails under some circumstances.

My convoluted configuration looks something like

{ ...
:bar #profile {:some-profile {:value #or [#ref [:some-path #keyword #env MY_ENV]
                                          "my-backup-value"]}}
:foo #merge [{:w :x :y :z}
             #ref [:bar]]
...}

When I apply aero's reader on this .edn I'll get something like

{ ...
:bar {:value "my-backup-value"}
:foo {:w :x :y :z}
...}

The merge bit fails. However if I alter some other piece of the map such as duplicating a key-value pair such as

{ ...
:bar #profile {:some-profile {:value #or [#ref [:some-path #keyword #env MY_ENV]
                                          "my-backup-value"]}}
:foo #merge [{:w :x :y :z}
             #ref [:bar]]
:foo-again #merge [{:w :x :y :z}
                    #ref [:bar]]
...}

Then it will properly evaluate to

{ ...
:bar {:value "my-backup-value"}
:foo {:w :x :y :z :value "my-backup-value"}
:foo-again {:w :x :y :z :value "my-backup-value"}
...}

This makes me suspect that some evaluation-order issue is responsible. Looking at the implementation of the #merge reader I don't see any piece for checking for incomplete contents so I'm surprised it seems to work so often.

Unfortunately I could not recreate a small example, and even anonymizing the keys and values in my personal config destroy the behavior so I cannot share that either.

Suggestion: #merge tag

Sometimes it might be useful to include a configuration, but slightly alter it.

For example, you might start with a config like:

{:ring-defaults #include "site-defaults.edn"}

Then decide to alter a small part. Perhaps a #merge tag can merge configurations, in the same way that Leiningen merges profiles:

{:ring-defaults
 #merge [#include "site-defaults.edn" {:static {:resources "foo/public"}}]}

I've written a meta-merge library that might be useful. Feel free to close this issue if you think this is too heavy or not useful for Aero's specific goals.

Integrate with clojure.spec

See this paper for why: https://blog.acolyer.org/2016/11/29/early-detection-of-configuration-errors-to-reduce-failure-damage/

Project abandoned?

I'm wondering if aero has been abandoned. I certainly hope not and that everything is okay at JUXT, but there are three PRs that have been sitting around without comment for a while and no real activity. They're all failing but that seems to be entirely due to an issue with rake in the build process. An status update would be appreciated, as I don't think it's an overstatement to say that this is probably the top config library for Clojure right now.

With lumo 1.9+, fs dependency is not found

Summary

I'm seeing an issue when I read an EDN file via aero.core/read-config.
It works fine when used on Lumo up to version 1.8.0, regardless if it's aero 1.1.2 or 1.1.3. But from Lumo 1.9.0 and up, an error is thrown. See below for details.

Detailed description

If you create an EDN file:

$ echo "{:foo :bar}" > resources/aha.edn

And from Lumo 1.8, at the REPL, which is started via:

lumo -D aero:1.1.3 -c src:test -d

(require '[aero.core :refer [read-config]])
(read-config "resources/aha.edn")

the EDN file is read as expected.

But from Lumo 1.9+, if you do the same as above, you get this exception:

fs is not defined
         aero$core$read_config_into_tag_wrapper (evalmachine.<anonymous>:829:386)
         Function.aero.core.read_config.cljs$core$IFn$_invoke$arity$2 (evalmachine.<anonymous>:860:61)
         aero$core$read_config (evalmachine.<anonymous>:844:30)
         Function.aero.core.read_config.cljs$core$IFn$_invoke$arity$1 (evalmachine.<anonymous>:865:30)
         aero$core$read_config (evalmachine.<anonymous>:848:30)
         (evalmachine.<anonymous>:1:23)
         Script.runInThisContext (vm.cljs:90:20)
         Object.runInThisContext (vm.cljs:297:38)
         (Object.It)
         (Object.lumo.repl.caching_node_eval)

At first I thought that it might be the change that was introduced in this PR https://github.com/juxt/aero/pull/59/files, but trying fs, requiring it the same way, in Lumo 1.9.0 & Lumo 1.10.1, it works fine for reading an EDN file:

cljs.user=> (require '["fs" :as fs])
nil
cljs.user=> (fs/readFileSync "resources/aha.edn" "utf-8")
"{:foo :bar}\n"

Any hints as to what could be causing this?

Thanks

Merge + Ref combination doesn't work

Given this edn:

#merge [{:b 1} {:c #ref [:b]}]

You'll get:

WARNING: Unable to resolve "#ref [:b]" at [:form 1 :c]
=> {:b 1, :c nil}

This is because the ref tag seems to operate on some different representation. The following edn renders correctly though:

#merge [{:b 1} {:c #ref [:form 0 :b]}]

results in:

=> {:b 1, :c 1}

Having to name the index of reffed item in the merges decreases the value of the ref a lot and it will introduce breakage when people move items around.

I think this is the same problem as experienced in #85

^:ref's don't compose with edn sets

I suspect this is related to #23 and #17 but it looks like you can't nest ^:refs inside edn set literals

For example parsing the following aero file:

{
   :foo #{^:ref [:bar]}
   :bar "bar"
}

Results in the exception:

1. Unhandled clojure.lang.ExceptionInfo
   No reader for tag ref
   {:tag ref, :value [:bar]}

                  core.clj: 4725  clojure.core/ex-info
                  core.clj: 4725  clojure.core/ex-info
                 core.cljc:   35  aero.core$eval3991$fn__3992/invoke
              MultiFn.java:  238  clojure.lang.MultiFn/invoke
                  core.clj: 2598  clojure.core/partial/fn
                 core.cljc:  224  aero.core$resolve_tag_wrappers$fn__4163/invoke
                  walk.clj:   48  clojure.walk/walk
                  walk.clj:   52  clojure.walk/postwalk
                  walk.clj:   52  clojure.walk/postwalk
                  core.clj: 2597  clojure.core/partial/fn
                  core.clj: 2728  clojure.core/map/fn
              LazySeq.java:   40  clojure.lang.LazySeq/sval
              LazySeq.java:   49  clojure.lang.LazySeq/seq
                   RT.java:  525  clojure.lang.RT/seq
                  core.clj:  137  clojure.core/seq
             protocols.clj:   24  clojure.core.protocols/seq-reduce
             protocols.clj:   75  clojure.core.protocols/fn
             protocols.clj:   75  clojure.core.protocols/fn
             protocols.clj:   13  clojure.core.protocols/fn/G
                  core.clj: 6704  clojure.core/reduce
                  core.clj: 6771  clojure.core/into
                  walk.clj:   49  clojure.walk/walk
                  walk.clj:   52  clojure.walk/postwalk
                  walk.clj:   52  clojure.walk/postwalk
                  core.clj: 2597  clojure.core/partial/fn
                  core.clj: 2726  clojure.core/map/fn
              LazySeq.java:   40  clojure.lang.LazySeq/sval
              LazySeq.java:   49  clojure.lang.LazySeq/seq
                   RT.java:  525  clojure.lang.RT/seq
LazilyPersistentVector.java:   44  clojure.lang.LazilyPersistentVector/create
                  core.clj:  377  clojure.core/vec
                  walk.clj:   45  clojure.walk/walk
                  walk.clj:   52  clojure.walk/postwalk
                  walk.clj:   52  clojure.walk/postwalk
                  core.clj: 2597  clojure.core/partial/fn
                  core.clj: 2728  clojure.core/map/fn
              LazySeq.java:   40  clojure.lang.LazySeq/sval
              LazySeq.java:   49  clojure.lang.LazySeq/seq
                   RT.java:  525  clojure.lang.RT/seq
                  core.clj:  137  clojure.core/seq
             protocols.clj:   24  clojure.core.protocols/seq-reduce
             protocols.clj:   75  clojure.core.protocols/fn
             protocols.clj:   75  clojure.core.protocols/fn
             protocols.clj:   13  clojure.core.protocols/fn/G
                  core.clj: 6704  clojure.core/reduce
                  core.clj: 6771  clojure.core/into
                  walk.clj:   49  clojure.walk/walk
                  walk.clj:   52  clojure.walk/postwalk
                  walk.clj:   52  clojure.walk/postwalk
                 core.cljc:  221  aero.core$resolve_tag_wrappers/invokeStatic
                 core.cljc:  219  aero.core$resolve_tag_wrappers/invoke
                 core.cljc:  372  aero.core$read_config/invokeStatic
                 core.cljc:  361  aero.core$read_config/invoke
                 core.cljc:  374  aero.core$read_config/invokeStatic
                 core.cljc:  361  aero.core$read_config/invoke
                      REPL:   99  grafter-server.main/eval49500
                      REPL:   99  grafter-server.main/eval49500
             Compiler.java: 6978  clojure.lang.Compiler/eval
             Compiler.java: 6941  clojure.lang.Compiler/eval
                  core.clj: 3187  clojure.core/eval
                  core.clj: 3183  clojure.core/eval
                  main.clj:  242  clojure.main/repl/read-eval-print/fn
                  main.clj:  242  clojure.main/repl/read-eval-print
                  main.clj:  260  clojure.main/repl/fn
                  main.clj:  260  clojure.main/repl
                  main.clj:  176  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  657  clojure.core/apply
                  core.clj:  652  clojure.core/apply
                regrow.clj:   18  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  657  clojure.core/apply
                  core.clj: 1963  clojure.core/with-bindings*
                  core.clj: 1963  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   85  clojure.tools.nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   55  clojure.tools.nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  222  clojure.tools.nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
    interruptible_eval.clj:  190  clojure.tools.nrepl.middleware.interruptible-eval/run-next/fn
                  AFn.java:   22  clojure.lang.AFn/run
   ThreadPoolExecutor.java: 1142  java.util.concurrent.ThreadPoolExecutor/runWorker
   ThreadPoolExecutor.java:  617  java.util.concurrent.ThreadPoolExecutor$Worker/run
               Thread.java:  745  java.lang.Thread/run

Similarly you get the same exception if you use a #ref tagged literal too:

{
   :foo #{#ref [:bar]}
   :bar "bar"
}

Consider using namespace qualified tagged literals

I appreciate the succinct nature of the namespace free reader tags e.g. #ref but ideally these tags would be namespaced #juxt.aero/ref or perhaps just #aero/ref. clojure.org has this to say:

Reader tags without namespace qualifiers are reserved for Clojure.

And the edn-format spec re-echoes the same sentiment:

Tag symbols without a prefix are reserved by edn for built-ins defined using the tag system.

There may already be a collision here too, as integrant also defines #ref. How would one use both aero and integrant together?

#hostname broken sinсe 1.1.4

function env was renamed to get-env, but forgot to rename call (env "HOSTNAME")

#104

Is there any way to retain the information in a ref's path?

My use case has me trying to get at the path of my #ref without re-writing it somewhere else in the given context. For example:

;; Config
{:primitive-thing {:some   :shared
                   :values true}

 :other-thing #build-thing {:referenced-thing #ref [:primitive-thing]}}

;; An ideal reader
(defmethod aero/reader 'build-thing [_ _ thing]
  (->> thing
       :referenced-thing
       (meta)
       :ref-path
       (assoc thing :knows-about)))

;; End result
{:primitive-thing {:some   :shared
                   :values true}
 :other-thing     {:referenced-thing {:some   :shared
                                      :values true}
                   :knows-about      [:primitive-thing]}}

Is this possible today, perhaps through a different approach? If not, is this easy enough to add?

Is it possible to override config.edn with local configurations that is not tracked by git?

During development, we need to run the program with some configurations that is not tracked by git.

What's the most proper way to achieve that with aero?

Reader literal to split a value

I have a need to split (as per clojure.string/split) a single value (coming from #env) into a sequence, and implemented the following reader literal in my own code:

(defmethod aero.core/reader 'split
  [opts tag value]
  (let [[s re] value]
    (if (and s re)
      (s/split s (re-pattern re)))))

It seems to work well, and I thought it might be useful in aero core? Happy to prepare a PR if the authors are on board and that would be more helpful (or just copy this code directly - I'm happy either way).

Two disclaimers:

1. I have no experience with ClojureScript so this solution may not work in that environment.
2. I'm not sure if opening up the regex to the EDN layer might represent a risk (security, DoS, remote code execution, etc.). If so, it may be better to hardcode the regex inside the reader method (e.g. as "\\s*,\\s*" for comma delimited data), perhaps offering a fixed set of predefined regexes selectable via keywords in the second position of value?

Dead link to Juxt blog in readme

This link in the readme is dead: https://juxt.pro/blog/posts/component-meet-schema.html I couldn't find the article anywhere

#ref breaks #or

given:

{:foo :foo
 :bar #join [#ref [:foo]
             #or [nil 42]]}

i'd expect

{:foo :foo, :bar ":foo42"}

I get:

{:foo :foo, :bar :fooaero.core.TagWrapper@8ec8f862}

Tags do not resolve in an obvious order.

The two following code blocks will have different values associated with the key :result.

{:foo {:bar "my-value"}
:result #ref [:foo #keyword "bar"]}

will resolve to the map

{:foo {:bar "my-value"}
:result nil}

where as

{:foo {:bar "my-value"}
:my-key #keyword "bar"
:result #ref [:foo #ref [:my-key]]}

will resolve to the map

{:foo {:bar "my-value"}
:my-key :bar
:result "my-value"}

My expectation would be that the output maps should be the exact same. (other than for the :my-key :bar key-value pair) I ideally shouldn't have to create an intermediary key to work-around reader composition rules.

Feature Request: Set profile via environment variable

In the case of 12 factor applications, it can be useful to select the profile via a single environment variable. Perhaps defaulting profile to a parser reading the ENV environment variable?

ENV=production java my-awesome-app.jar

If aero wanted to be less opinionated, it could even be configured via the options map:

(aero/read-config "config.edn" {:profile-var "ENV"})

Happy to submit a PR if this is something that could be added. I also buy that this can be done in a per-project config namespace (sourcing the env var, parsing it to keyword, and using it) so maybe it's not worth it.

Cannot read-config from resource in uberjar

Hello. I'm getting this error when running an uberjar that uses [aero "1.0.0-beta3"].

Exception in thread "main" java.lang.IllegalArgumentException: Cannot open <nil> as a Reader

Works fine with [aero "1.0.0-beta2"]

Deprecate #cond in favor of #profile

Misleading information in README.md

In the env section it states: "[...] it is very easy for anyone to read a process's environment (e.g. via ps -ef)".
First of all, it's supposed to be ps ef, or probably even ps e -f, and not ps -ef.
And please correct me if I'm wrong but I think with ps ef you can only see environment variables of the processes that have been started by your user (or all users if you're root). More info: https://security.stackexchange.com/questions/14000/environment-variable-accessibility-in-linux/14009#14009

#ref breaks #or

Given the following config:

{:env #or [#env FOO :foo]
 :foo #ref [:env]}

I'd expect read-config to return (assuming the env var FOO isn't set):

{:env :foo, :foo :foo}

I get this instead:

{:env nil, :foo nil}

Getting an unexpected circular dependency

After looking at the example of #ref in the README.md I tried using it in my own config as follows:

{:some-top-level-key {:db-connection "datomic:dynamo://dynamodb"
                      :webserver {:db #ref [:some-top-level-key :db-connection]}}

This gives a circular dependency error.

Anything I'm missing here? Is #ref only valid if the path to the #ref doesn't share any of the path defined in the #ref vector?

Thank you :)

clojurescript and clojure has different behavior on missing files when using #include

The include tag behaves differently when it does not find a file to include.
In the case if clj it will return a map like {:aero/missing-include include}.
Cljs will just throw a error.

#file won't read resources from inside uberjar

using the #file reader condition areo fails to read files packaged in an uberjar

make a release including the 'require "fs"' commit?

I'm using aero for a node CLI app and everything works fine if the commit requiring fs is included (bit not the released 1.1.2). I was wondering if you would be willing to make a release of 1.1.3 including this change?

Thanks!

Do magic without being forced to load files

Is there a way to get the same cool functions as provided in

(read-config "config.edn")

by soley act on software defined map? Sth. like

(resolve-config {:hello "World"}) ?

Not being forced to handle files (either on filesystem-by-path, by-resource or by-resource-in-jar) would make testing a lot easier ...

Is aero still beta?

Readme stills says:

Please note that being a beta version indicates the provisional status of this library, and that features are subject to change

Is it still in experimentation phase? I feel it's API is not likely to change anymore.

Should support #include of absolute file paths

Has regressed since 81df59c

conditional if/else

Hi there, thanks so much for this awesome utility. I wonder if it possible to set field based on env variable. For example, if the current os is linux then set this field to x, if window then to y, else mac to z.

Thanks

^:secret tag and bespoke serializer to redact sensitive information for logging purposes #enhancement

Aero's ability to combine configuration sources from separate sources is a useful (and well-documented) way to satisfy the use case of keeping secrets out of git.

I'd like a way to log the resolved config without all my passwords ending up in plaintext though.

I hacked together a patched pretty printer dispatch table wrapper and a tag to match it for my immediate purposes, but I'm certain there must be cleverer ways to get this to work.

Or perhaps this is out-of-band for Aero and ought to be filed elsewhere?

Reader tag envf not present in 1.0.2

Hello, the README tells you to use 1.0.2 but also mentions #envf tag but that was only added in 1.0.3 (which seems to be not yet available from central repos).
Could you please update the README to be explicit about which aero versions the given tag supports?
Thank you.

[Enchancement proposal] Support for optional inclusions using #opt-include reader tag

It is a common pattern to include a default configuration with your application and then override those defaults from other sources(if found) during deployment. Aero'r #include tag seems particularly useful in such situations. The problem is, it does not handle missing files gracefully. Also, the course of action taken for a missing file varies greatly depending on the resolver in use.

After some discussion in clojure slack, someone suggested that a custom reader tag would do the job and it would be trivial to implement. As such, here's an implementation of the #opt-include tag,
which takes a filename along with a default value(which is used if the file is missing), passed as a 2-element vector:

(require 'clojure.java.io)
(require 'clojure.string)
(require 'aero.core)
(import [java.io])

(defn missing?
  "Missing file detection, compatible with relative-resolver."
  [file]
  (or (nil? file)
      (and (= (type file) java.io.StringReader)
           (clojure.string/includes? (slurp file) "missing-include"))))

(defmethod aero.core/reader 'opt-include
  [{:keys [resolver source] :as opts} tag [filename default-val]]
  (try
    (let [f (if (map? resolver)
              (get resolver filename)
              (resolver source filename))]
      (if (missing? f)
        default-val
        (aero.core/read-config f)))
    (catch java.io.FileNotFoundException e
      default-val)))

Example configuration file using #opt-include:

#merge [
   {:host "localhost"}
   #opt-include ["opt-config.edn" {:port 8080}]
]

I plan to submit a pull request later.

Doesn't work with planck

I tried using this with planck after installing it to my local Maven repo and then running:

planck -D aero:1.1.2
(Planck REPL starts up)

cljs.user=> (require '[aero.core :as aero])
No such namespace: cljs.nodejs, could not locate cljs/nodejs.cljs, cljs/nodejs.cljc, or JavaScript source providing "cljs.nodejs" in file aero/core.cljc

Set semantics for #profile (similar to #hostname or #user)

There are scenarios when staging and production profiles get all data from env variables, thus making some duplication.

Having them a set semantics will help with this.

Implement a reader for #keyword tag

Many clojure projects use keywords in their configuration maps,
a #keyword reader tag to convert string to keywords would be nice for those cases.

Provide support / API for loading a configuration directory - a la linux config.d ?

Hi,

Context: I am looking at aero for configuration loading but we have a use case that might not fit ?!

I know there is a reason why aero suggests using a single file but there are valid reasons for having configurations in a directory or loaded from multiple places.

A good write up is here, but I'm sure there are more: https://www.redhat.com/sysadmin/etc-configuration-directories

For example, in linux, files are loaded from a config.d/ directory in sorted order, because it allows users to ADD configs that will not be overwritten on app upgrade for example.

Another personal example: in one app I load configs from a directory because we keep secrets in HashiCorp Vault and we save those as a file, alongside the public config which we store in git.
Plus we generate some other data during container image build as config file.

For us, it made sense to load configuration from a conf.d/ directory.

Meta not being added to vector on v1.1.5 - works on v1.1.3

Meta not being added to vector on v1.1.5 - works on v1.1.3

Example:

{:post [^:interceptors
    [#interceptor :my.company.interceptors.auth/parse-jwt
     #interceptor :my.company.interceptors.auth/verify-signature
     #interceptor [:my.company.interceptors.auth/scope? {:scope "admin/csv-import"}]
     #interceptor [:my.company.interceptors.params/coerce-params {:body-params {:filename string?}}]]
    #handler :my.company.handlers.import/trigger-csv-import]}

Then when we query for the meta on the vector that is the value of :post, on aero v1.1.3 it returns :interceptors, as expected, however on aero v1.1.5 it returns nil

Faulty set lookups in #user and #hostname

First off, thanks for this library, I've found it very useful!

The set lookups in #user and #hostname readers use an or backoff instead of checking for key presence, and so will return the :default value when the desired key is present but has a falsy value.

https://github.com/juxt/aero/blob/master/src/aero/core.cljc#L92

One alternate approach is to expand set keys into individual entries, and then use get with default, eg:

(defn- expand-set-keys [m]
  (reduce-kv (fn [m k v]
               (if (set? k)
                 (reduce #(assoc %1 %2 v) m k)
                 (assoc m k v))) {} m))

(defmethod aero/reader 'hostname
  [{:keys [hostname]} tag value]
  (let [value (expand-set-keys value)]
    (get value hostname
         (get value :default))))

#or and #ref tags don't combine as expected in "large" maps

On aero 1.1.4 the #or and #ref tags do not combined as expected when the maps contain enough data.

The following config map

 {:x "foo"
  :y #or [#ref [:x] "bar"]}

will properly evaluate to

 {:x "foo"
  :y "foo"}

However the config map

 {:x "foo"
  :y #or [#ref [:x] "bar"]
  :junk0 "0"
  :junk1 "1"
  :junk2 "2"
  :junk3 "3"
  :junk4 "4"
  :junk5 "5"
  :junk6 "6"
  :junk7 "7"
  :junk8 "8"}

improperly evaluates to

 {:x "foo"
  :y nil
  :junk0 "0"
  :junk1 "1"
  :junk2 "2"
  :junk3 "3"
  :junk4 "4"
  :junk5 "5"
  :junk6 "6"
  :junk7 "7"
  :junk8 "8"}

`aero.core/deferred` does not transparently compose with subsequent data readers

Version

[aero "1.1.3"]

Problem

Custom data readers that use aero.core/deferred do not transparently compose with subsequent data readers.

Example

user=> (require '[aero.core :as aero])
nil

user=> (import java.io.StringReader)
java.io.StringReader

user=> (defmethod aero/reader 'custom [_ _ v]
  #_=>   (aero/deferred v))
#multifn[reader 0x74a1ace3]

user=> (defn read-str [s]
  #_=>   (aero/read-config (StringReader. s)))
#'user/read-str

user=> (read-str "{:x #boolean \"true\",
  #_=>             :y #boolean #custom \"true\"}")
{:x true, :y false}

user=> (read-str "{:x #long #custom \"123\"}")
Execution error (NumberFormatException) at java.lang.NumberFormatException/forInputString (NumberFormatException.java:68).
For input string: "aero.core.Deferred@e53bb4c1"

Aside: Catching this was made harder because I was using #boolean, which silently yields false when given values outside of #{"true" "false"}. This kind of laxity could be dangerous.

Work-around

Given that the use case for deferred is mainly to compose with #profile:

1. Remove the use of aero.core/deferred from the custom reader.
2. Structure the config such that #profile will be resolved prior to the custom tag, e.g.:
   {:x #long #custom #profile {:dev "foo" :prod "bar"}}

Consider a #aero/resolve tag

This would utilize requiring-resolve (backported from 1.10 OR not available if not present). This would make it easier to support things like extending a protocol via metadata:

^{foo.bar/some-proto-method #aero/resolve my.project/proto-method-ext} {:foo/bar :config}

This would be assistive to projects like clip and edge which allow defining the system in Aero.

Why are refs metadata rather than tags?

I'm curious why refs are metadata, and everything else uses tags. Is there a reason for this?

Suggestion: A value which should never be filled for dummy values (e.g. #throw)

Inspired by https://lobste.rs/s/chxlum/changeme_is_valid_base64#c_drqazo

The use case is for when setting up a config early in a project, and you're not sure what the value will be, something like this:

:api-token {:dev "dev-token" :prod "changeme"}

In order to assist with forgetting to change the token, you could instead do

:api-token {:dev "dev-token" :prod #throw "changeme"}

which would give you early warning that something was amiss. Ideally Aero's reporting would bubble out information such as what k-path the error was at.

How to read configuration from string or map?

Hello,

I saw that aero reads directly from a file but I would like to read from a string or build a configuration from a user provided map.

The use case is to integrate it with https://github.com/l3nz/cli-matic that already has file reading capabilities.

Another use case would be to build a configuration dynamically (from a db or user input).

All the examples show reading from file and I'm quite new to clojure so I can't really tell how easy it is to do that from the sources.

Thanks,