pastdev/clconf

clconf's Issues

getv --as-json flag non-functional on strings.

Description

When I use getv --as-json to extract a string value, I get the actual string rather than a JSON string.

How to repeat

clconf \
  --ignore-env \
  getv \
  --yaml-base64 "$(echo 'item: Book "Binder"' | base64 --wrap=0)" \
  "/item" \
  --as-json

Expected result:

"Book \"Binder\""

Actual result:

Book "Binder"

Migrate to spf13

Seems like the momentum is behind spf13/cobra. Plus I use it in other projects and prefer its approach.

Incorrect output for certain string values

The following behavior has changed from v3.0.6; when using the supplied config.yml, additional files are output when reading certain strings.

$ ./clconf-307 version
Version: v3.0.7
$ YAML_FILES=config.yml ./clconf-307 getv "example"
' This is a string. '
$ ./clconf-306 version
Version: v3.0.6
$ YAML_FILES=config.yml ./clconf-306 getv "example"
 This is a string. $
$ cat config.yml
example: '
  This is a string.
'
$

Note the quotes in the first, and the lack of newline in the second. I also saw a slight variation of this inside a container elsewhere:

$ ./clconf-307 version
Version: v3.0.7
$ YAML_FILES=example.yml ./clconf-307 getv "example"
|2
   This is a string value.
$ ./clconf-306 version
Version: v3.0.6
$ YAML_FILES=example.yml ./clconf-306 getv "example"
   This is a string value.
$ cat example.yml
example: '
   This is a string value.
'

Note the |2 inexplicably added with 3.0.7; this container was rockylinux 8 based.

As an aside, the reason for finding the strange formatting of the string is due to the yaml file being generated by python (pyyaml).

CLI cannot merge different types

Description

The clconf CLI application cannot merge different types.

How to repeat

1. Create two files:

number.yml:

number: 100

quoted.yml:

number: "500"

Now run clconf on these two files:

clconf --yaml number.yml --yaml quoted.yml

Expected Result

number: "500"

Actual Result

Error: cannot append two different types (int, string)

Workarounds

Don't do that!

Error: cannot append two different types (slice, string) (2.0.15)

Problem

Running clconf with multiple --yaml switches results in an error.

Steps to Reproduce

Create a secrets.yml file:

---
jira:
  password: [OMITTED]
  url: https://jira.mitre.org/
  username: [OMITTED]
nuget:
  username: [OMITTED]
  password: [OMITTED]
resource_manager_settings:
  next_sprint_pattern: '^ASIAS ETL \\d+(:? \\((?i)next(?-i)\\))?$'
ssh:
  knownHosts: [OMITTED]
  publicKey: [OMITTED]
  privateKey: [OMITTED]

Create a secrets.override.yml file with the omitted values from secrets.yml populated.

Call clconf with the two files mentioned above in the order of secrets.yml, then secrets.override.yml:

clconf --yaml secrets.yml --yaml secrets.override.yml

Expected Outcome

The merger of the two yaml files gets printed to the console.

Actual Outcome

Error: cannot append two different types (slice, string)
Usage:
  clconf [global options] command [command options] [args...] [flags]
  clconf [command]

Workarounds

2.0.13 was the last working version, so I'll be using that for now.

ToKvMap should produce /key/<index>/value for arrays

Currently, a list:

--- 
foo:
- a
- b
- c

Will result in:

/foo/a = ""
/foo/b = ""
/foo/c = ""

When run through ToKvMap. This would be useful for iterating over the key foo using confd getvs '/foo/*', but does not preserve order as would be expected in a list. Given that confd's file backend was modified to include the index as a subkey:

/foo/1 = "a"
/foo/2 = "b"
/foo/3 = "c"

This will preserve order... But will require users to know this conversion ahead of time so that when the ask for getvs "/foo/*" they will expect numeric values...

Convert to go 1.11 modules

Template command does not generate empty output

Problem

template command does not generate an ouput when that output file would be empty.

How to reproduce

Using the following two files, issue the command:

clconf template --yaml values.yml --in-place template.clconf

values.yml

---
my_array: []

template.clconf

{{- if gt (len (ls "/my_array")) 0 -}}
Lots
of
stuff
{{ end -}}

Expected results

An empty file: template

Actual results

No file generated

Workarounds

If the output file accepts comments, we can put a boilerplate comment in the template - but this only works for certain use-cases.

Resolution

None yet.

Templates don't like top-level lists

Command-level getv works fine with a top-level list:

$ clconf --pipe getv /0 <<'EOF'
[{"key": "value"}]
EOF
key: value

But a template is not:

$ clconf --pipe getv / --template-string '{{ getv "/0" }}' <<'EOF'
[{"key": "value"}]
EOF
Error: template execute: execute template: template: cli:1:3: executing "cli" at <getv "/0">: error calling getv: /0: key does not exist

Value resolution inconsistent when value specified multiple YAML files

Below are two YAML files: secrets.yaml and configMap.yaml. The configMap.yaml file contains the placeholder value, SET IN SECRETS for usernames and passwords - these values are all specified in the secrets.yaml file.

secrets.yaml

mongodb:
  service_account:
    user: mongodb-service-user
    password: mongodb-service-password
  public_account:
    user: mongo-public-user
    password: mongo-public-password
ldap:
  service_account:
    user: ldap-service-user
    password: ldap-service-password

configMap.yaml

mongodb:
  servers:
  - host: mongodb1.mycorp.com
    port: 27017
  - host: mongodb2.mycorp.com
    port: 27017
  - host: mongodb3.mycorp.com
    port: 27017
  service_account:
    user: SET IN SECRETS
    password: SET IN SECRETS
  public_account:
    user: SET IN SECRETS
    password: SET IN SECRETS
webapp:
  server_id: cluster1.mycorp.com
  cluster_name: Cluster 1
  external_clusters:
  - name: Cluster 2
    url: https://cluster2.mycorp.com
  - name: Cluster 3
    url: https://cluster3.mycorp.com
business_day:
  timezone: America/New_York
  start_of_day: "06:30:00"
  end_of_day: "19:30:00"
email:
  support_email: [email protected]
  default_email_domain: "@mycorp.com"
  smtp_host: smtp-main.mycorp.com
ldap:
  provider_uri: ldaps://ldap.mycorp.com
  search_base: ou=People,dc=mycorp,dc=com
  search_filter: (uid={0})

I set up my YAML_FILES environment variable thusly:

export YAML_FILES=$(readlink -e secrets.yaml),$(readlink -e configMap.yaml)

When I run clconf the following occurs:
clconf INCORRECTLY returns SET IN SECRETS (from the configMap.yaml file) for:

• /mongodb/public_account/password
• /mongodb/public_account/user
• /mongodb/service_account/password
• /mongodb/service_account/user

clconf correctly returns the user/password from the secrets.yaml file for:

• /ldap/service_account/password
• /ldap/service_account/user

Cannot retrieve items from arrays using getv

The following script will attempt to retrieve the first element of an array, but will fail:

#!/bin/sh
YAML="$(cat<<"__EOF__"
  applications:
  - a
  - b
  - c
  credentials:
    username: foo
    password: bar
__EOF__
)"
YAML_BASE_64="$(printf "%s" "${YAML}" | base64 -w 0)"

clconf getv --yaml-base64="${YAML_BASE_64}" "/applications/0"

I expect to get the value "a" but instead, I am getting the following error:

WARN[0000] value at [0] not a map: slice
Error: [/applications/0] does not exist

If I change it to grab "/applications", I get:

- a
- b
- c

clconf fails when YAML_FILES is set but empty

Example:

$ YAML_FILES="" clconf
Error: load conf: stat: stat : no such file or directory

We should probably handle that case better.

Running clconf on overlapping YAML files does not result in proper merge of boolean properties.

Problem

Running clconf on overlapping YAML files does not result in proper merge of boolean properties.

Steps to Reproduce

Create the following YAML files:

# foo.yml
---
foo: true

# bar.yml
---
foo: false

Run clconf with those files:

clconf --ignore-env --yaml foo.yml --yaml bar.yml

Expected Outcome

foo: false

Actual Outcome

foo: true

Workarounds

N/A

Resolution

TBD

lsdir template func sorts keys in lexicographical order

Hi, the lsdir func currently sorts numerical keys in lexicographical order:

clconf --yaml /tmp/foo.yml getv --template-string '{{ range $idx := lsdir "/foo" }}
{{- $idx }}
{{ end }}
'
0
1
10
11
12
13
14
15
16
17
18
19
2
20
21
22
23
24
25
26
27
28
29
3
30
4
5
6
7
8

This is due to the sort.Strings() call here call. I realize that this is part of clconf 's core, but is there any chance of natural sort in some form in the future?

--patch is ambiguous

The actual parameter for --patch takes a file, but the help makes no mention of that and the example is a string.

$ echo '{"foo": "bar"}' | clconf --pipe getv --patch '[{"op": "replace", "path": "/foo", "value": "bip"}]' /
Error: load conf: patch: reading [{"op": "replace", "path": "/foo", "value": "bip"}]: open [{"op": "replace", "path": "/foo", "value": "bip"}]: no such file or directory

$ echo '{"foo": "bar"}' | clconf --pipe --patch <(echo '[{"op": "replace", "path": "/foo", "value": "bip"}]') getv /
foo: bip

We should update the docs to be clear, and potentially support just string patches.

clconf example doesn't match definition

In the configuration section, the example doesn't match the documentation.

Using clconf requires one or more yaml files (or strings) to merge together. They are specified using either using environment variables or command line options, as either files or base64 encoded strings. The order they are processed in is as follows:

     --yaml: One or more files.
    YAML_FILES environment variable: A comma separated list of files.
    --yaml-base64: One or more base64 encoded strings containing yaml.
     YAML_VARS environment variable: A comma separated list of environment variable names, each a base64 encoded string 
 containing yaml.
    --stdin: One or more --- separated yaml files read from stdin.
    --var: One or more path overrides of the form /foo="bar". Key is a path, an value is json/yaml encoded.
     --patch: One or more rfc 6902 json/yaml patch files to apply to the result of merging all the config sources.
     --patch-string: One or more rfc 6902 json/yaml patches to apply to the result of merging all the config sources.

All of these categories of input will be appended to each other and the last defined value of any key will take precedence. For example:

 YAML_FILES="a.yml,b.yml"
 YAML_VARS="C_YML_B64,D_YML_B64"
 C_YML_B64"$(echo -e "c:\n  foo: bar" | base64 -w 0)
 D_YML_B64"$(echo -e "d:\n  foo: bar" | base64 -w 0)
 
 G_YML_B64="$(echo -e "g:\n  foo: bar" | base64 -w 0)
 H_YML_B64="$(echo -e "h:\n  foo: bar" | base64 -w 0)
 
 clconf \
   --yaml e.yml \
   --yaml f.yml \
   --yaml-base64 "$G_YML_B64" \
   --yaml-base64 "$H_YML_B64" \
   --var '/foo="bar"' \
   --patch patch.json \
   --patch-string '[{"op": "replace", "path": "/foo", "value": "baz"}]' \
   <<<"---\nfoo: baz"

Would be processed in the following order:

a.yml
b.yml
C_YML_B64
D_YML_B64
e.yml
f.yml
G_YML_B64
H_YML_B64
stdin
`/foo="bar"`
patch.json
`[{"op": "replace", "path": "/foo", "value": "baz"}]`

The documentation is correct, the example is incorrect.

A means to iterate through a range of numbers in a template

edit:

closing this one, and opening an issue on confd