petems/petems-hiera_vault

A hiera backend for access to secrets being stored in HashiCorp Vault

petems-hiera_vault's Issues

puppet lookup does not work as desired

Dear Peter,

Thank you very much for your great work. I have a setup issue which I am unable to resolve. I must say I am fairly new to vault.

1. Our puppet infrastructure is at version 6.15 and vault server is installed on puppet master host with puppet module jsok-vault 2.3.0.

vault::version: '1.5.3'
vault::storage:
 file:
   path: /mnt/vault_store

vault::listener:
 - tcp:
     address: 0.0.0.0:8200
     tls_client_ca_file: "/etc/puppetlabs/puppet/ssl/certs/ca.pem"
     tls_cert_file: "/etc/puppetlabs/puppet/ssl/certs/HOSTNAME.pem"
     tls_key_file: "/etc/puppetlabs/puppet/ssl/private_keys/HOSTNAME.pem"

vault::enable_ui: true
vault::api_addr: "http://HOSTNAME:8200"
vault::extra_config:
 log_level: "Debug"

2. Vault server is a part of the puppet certificate infrastructure and it uses puppet agents certificate. Vault cert auth is enabled and puppet server CA certificate is added as trusted cert as described here https://github.com/voxpupuli/puppet-vault_lookup#setup

3. puppet-vault_lookup 0.1.1 and hiera_vault 1.0.0 is installed with debouncer and vault gem. Global hiera 5 file /etc/puppetlabs/puppet/hiera.yaml is extended to:

- name: "Hiera-vault lookup"
    lookup_key: hiera_vault
    options:
      confine_to_keys:
        - '^vault_.*'
        - '^.*_password$'
        - '^password.*'
      ssl_verify: true
      ssl_ca_cert: /etc/puppetlabs/puppet/ssl/certs/ca.pem
      address: VAULT_SERVER
      token: TOKEN
      default_field: value
      mounts:
        puppet_secret:
          - "%{::trusted.certname}"
          - "test"

4. Enabled kv2 engine with puppet_secret path and then added secrets with certname/hostname e.g. vault kv put puppet_secret/certname my_password=foo123

5. I am able to fetch it with curl on puppet master host: curl --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem -H "X-Vault-Token:$TOKEN" -X GET https://VAULT_SERVER:8200/v1/puppet_secret/data/certname

6. But puppet lookup my_password --explain --compile does not find the key. I tested it with root token and also generated prod, test policies with read, list permissions and generated token from that. All tokens work with curl but from puppet lookup command don't.

Hierarchy entry "Hiera-vault lookup"
      No such key: "my_password"
      [hiera-vault] Client configured to connect to https://certname:8200
      [hiera-vault] Looking in path puppet_secret/certname for my_password
      [hiera-vault] Checking path: puppet_secret/certname/my_password
      [hiera-vault] Checking path: puppet_secret/certname/data/my_password
      [hiera-vault] Checking path: puppet_secret/data/certname/my_password
      [hiera-vault] Looking in path puppet_secret/test for my_password
      [hiera-vault] Checking path: puppet_secret/test/my_password
      [hiera-vault] Checking path: puppet_secret/test/data/my_password
      [hiera-vault] Checking path: puppet_secret/data/test/my_password
Function lookup() did not find a value for the name 'my_password'

7. I also enabled audit logging and I can see all the requests that were made to vault server but I don't see any errors.
8. Using deferred function in a manifest is also getting me no results.

$vault_notify = Deferred('vault_lookup::lookup', ['puppet_secret/data/certname/my_password', 'https://VAULT_SERVER:8200'])
  notify {"testing vault ${vault_notify}":}

Do you have any hint what might be going wrong? I am stuck with this problem for so many days :(. Any help will be much appreciated. Many thanks!

tag is not in sync with the version

Could you please create a tag to be sync with the module version?

Supporting k/v v2

It would be nice to be able to support the new k/v v2 backend, especially as it's now default for the main kv mount.

Blockers

Right now, the Vault gem does not have an easy way of using the new KV v2 backend

The following issues will need to be solved/merged before hiera_vault can support v2:

hashicorp/vault-ruby#195
hashicorp/vault-ruby#194
hashicorp/vault-ruby#196

returned secret merge behaviour

my hiera includes:-
mounts:
sandpit:
- function/%{function}
- job/%{job}
- climate/%{climate}
- product/%{product}
- role/%{role}
- stack/%{stack}
- platform/%{platform}
- common
I do not want 'merge: first' which appears to be the behaviour of hiera_vault at present.
I have modified my copy of hiera_vault.rb (version 0.3.0) to now accept
options:
<merge: (first|unique|hash|deep)>
assuming first if not present.

If the Vault token becomes unavailable to the Hiera/Vault module, a secret is substituted with an empty string

If the Vault token becomes unavailable, a secret is substituted with an empty string. If you have thousands of systems, all configuration files would be updated with an empty string which could be a huge issue.

Here is my Hiera configuration file. I was able to recreate the issue by moving the token file to a .old extension.

   - name: "Hiera-vault lookup"
    lookup_key: hiera_vault
    options:
      confine_to_keys:
        - '^vault_.*'
        - '^.*_password$'
        - '^password.*'
      address: https://hostname.example.com
      ssl_ca_cert: /etc/pki/tls/certs/example-root.pem
      token: /etc/puppetlabs/tokens/tokens_ro.txt
      default_field: value
      mounts:
        secrets/puppet_nogroup:
          - nodes/%{::trusted.certname}
          - common

In the nodes directory, I have a hostname1.net.yaml Hiera data file. This has the following secret key lookup using the Hiera/Vault integration:

virtual::secret_key: "%{lookup('password_virtual_key')}"

This is the example of a Puppet run where the secret was substituted for an empty Puppet string:

@@ -163,7 +163,7 @@
 # SECRET_KEY for all of them.
 #SECRET_KEY = secret_key.generate_or_read_from_file(
 #    os.path.join(LOCAL_PATH, '.secret_key_store'))
-SECRET_KEY = ‘REDACTED’
+SECRET_KEY = ''```

I am running Puppet 6.7 with petems-hiera_vault (v1.0.0).

Please advise on the best approach to resolve this.

Token Field does not Support JSON Files

Hi Peter

token: The token to authenticate with Vault, also read as ENV["VAULT_TOKEN"] or a full path to the file with the token (eg. /etc/vault_token.txt). When bootstrapping, you can set this token as IGNORE-VAULT and the backend will be stubbed, which can be useful when bootstrapping.

From vault agent and the sink the format for the file (/etc/vault/.vault_token) is the following:
{"token":"s.XXXXXXXXXXX","accessor":"XXXXXXXXXXXXXX","ttl":300,"creation_time":"2019-02-25T11:59:35.172939383Z","creation_path":"sys/wrapping/wrap","wrapped_accessor":""}

It can only read files with the plain token value in it not with the vault agent info from above.
https://www.vaultproject.io/docs/agent/autoauth/index.html

Easy to have another script reading that file and duping the info on another file but then you will have an inconsistency for couple of seconds,so it is better to make a change on the code.

BR

Joaquin

Vault agent Config

pid_file = "/etc/vault/pidfile"

auto_auth {
  method "approle" {
    mount_path = "auth/approle"
    config = {
      role_id_file_path = "/etc/vault/role_id.txt"
      secret_id_file_path = "/etc/vault/secret_id.txt"
      remove_secret_id_file_after_reading = false
    }
  }
  sink "file" {
    wrap_ttl = "5m"
    config = {
      path = "/etc/vault/.vault_token"
    }
  }
}

Issue with lookup function

Hi Peter,
I am trying to integrate puppet with vault.
I am using this hiera vault module for it.
After integration when I am running lookup function from the puppet master I am able to fetch data from the vault however when I am trying it from agent node its not working.
I have tested it by using puppet agent and lookup function.
Do I need to make any changes in the agent config?
I am using foreman as puppet master.

Setting token: 'ENV["VAULT_TOKEN"]' doesn't work

@petems How is the ENV["VAULT_TOKEN"] supposed to be set? I have exported the token as export VAULT_TOKEN=<token_here> and in my hiera.yaml file, I have token: 'ENV["VAULT_TOKEN"]' but when I run puppet, it fails to lookup the key in vault. Setting the token: '/path/to/token' works however.

Any thoughts on why setting token: 'ENV["VAULT_TOKEN"]' doesn't work? or I'm I doing something wrong?

IGNORE-VAULT doesn't work correctly if set in the file.

Consider:

vault.token

[root@puppettest01 ~]# cat /etc/puppetlabs/puppetserver/vault.token
IGNORE-VAULT

And in hiera.yaml

  - name: "Hiera-vault lookup"
    lookup_key: hiera_vault
    options:
      confine_to_keys: #Only lookup these
        - '^vault_.*'
        - '^.*password.*'
      ssl_verify: true
      address: <address>
      token: '/etc/puppetlabs/puppetserver/vault.token'
      default_field: value
      mounts:
        puppet:
          - 'nodes/%{facts.location}/%{trusted.certname}/'

I think this is because the check for Ignore is before the setting of the variable hiera_vault.rb#L56

And the to read the file into the variable comes way after hiera_vault.rb#L92

Unable to configure using socket instead of URL

I am still debugging this and I will try to submit a PR, but wanted to add a placeholder issue in case anyone has tried this and knows what is going on.

How to protect the token?

I'm reviewing this code as a potential move away from hiera-eyaml, but I'm not clear how to protect the token in the hiera.yaml file. Since this file is committed to the repo in my environment, I don't see an opportunity to template out this file due to the environments. Please, can you offer a bit of guidance here?

Testing - Running a real vault server is flaky

The tests don't seem to work on my new macbook:

5) FakeFunction#lookup_key when vault is unsealed should not cache the response when options changes
     Failure/Error: raise "Vault did not start in 10 seconds!"

     RuntimeError:
       Vault did not start in 10 seconds!
     # ./spec/support/vault_server.rb:80:in `rescue in wait_for_ready'
     # ./spec/support/vault_server.rb:71:in `wait_for_ready'
     # ./spec/support/vault_server.rb:49:in `initialize'

Probably best to move to use something like docker instead, and it'd be easy to set the root token and such with VAULT_DEV_ROOT_TOKEN_ID

Supporting AWS IAM/EC2 authentication

Is there a plan to support AWS IAM/EC2 authentication with Vault, as an alternative to a Vault token? It would be nice to have this functionality which would make it unnecessary to manually set up a Vault token for use by the puppet server.
It looks like Vault Ruby client https://github.com/hashicorp/vault-ruby supports AWS EC2 authentication
Has this been considered before?

When get the secrets if the secret contain a dollar (example$rest) sign only retrieve the secret until the dollar sign (example)

When I get a secret from the Vault Server using hiera_vault if the secret contains a dollar sign (exampe$more-characters) the value returned is until the dollar sign (example), I use a lot to render ini/config files.

I use puppet 6.x as code and make a lab with Puppet Server Open Source.

I try a lot of things like escaped the dollar signs but doesn't work!

Backend fails without gems

Ideally this function doesn't completely fail if the required gems are missing.

I think it would be better for it to just skip over the backend - that way a user can keep a default set of Vault information in their managed hiera.yaml file and not need to worry about having the required gems in place before that hiera.yaml file is deployed.

It's helped us internally as we deploy our Hiera file via r10k. Now I can r10k the same file to X amount of Puppetservers, and the ones with Vault installed will use the backend and the ones without can simply continue on as normal using the other backends defined.

Docs: More guidance on how to use this in yaml

So, I've been reading the README, and came across this:

And make yourself a favor and avoid lookup directly ;) Use

profile::ssl_role::key: "%{alias('vault_storage::ssl/params.key')}"

to inject value from key inside http://vault.foobar.com:8200/secret/data/ssl/params.

To me, this suggests that I can "just" use this as-is in my node.yaml files. However, for a given vault path puppet/data/nodes/node01.example.com/vault_gitlab with a key of token and a value of some_value, and a line in my yaml like this:

profiles::gitlab::runner::runners:
  '%{::fqdn}_shell':
    url: 'https://example.com'
    executor: 'shell'
    token: "%{alias('vault_storage::nodes/node01.example.com/vault_gitlab.token')}"

I get exactly nothing. If I use lookup() in my corresponding profile, I get the correct value back.

So, I am missing something, quite possibly something trivial. But what? My guess'd be that I have not defined a class vault_storage anywhere, but how would I go ahead and define the alias correctly?

Relevant part of my hiera.yaml looks like this:

---
version: 5

defaults:
  datadir: 'data'
  data_hash: 'yaml_data'

hierarchy:
  ## Hashicorp Vault
  - name: 'Hiera-vault lookup'
    lookup_key: hiera_vault
    options:
      confine_to_keys:
        - '^vault_.*'
      address: https://vault.example.com
      token: /etc/puppetlabs/code/vault_token.txt
      v2_guess_mount: false
      v1_lookup: false
      mounts:
        puppet/data:
          - 'nodes/%{::trusted.certname}'
          - 'global'

  ## Server
  - name: 'Machine specific data for one configured node served by puppet.'
    path: 'nodes/%{trusted.certname}.yaml'

Edit: I also tried with the following hiera.yaml:

---
version: 5

defaults:
  datadir: 'data'
  data_hash: 'yaml_data'

hierarchy:
  ## Hashicorp Vault
  - name: 'Hiera-vault lookup'
    lookup_key: hiera_vault
    options:
      confine_to_keys:
        - '^vault_storage::.*'
        - '^vault_.*'
      strip_from_keys:
        - 'vault_storage::'
      address: https://vault.example.com
      token: /etc/puppetlabs/code/vault_token.txt
      v2_guess_mount: false
      v1_lookup: false
      mounts:
        puppet/data:
          - 'nodes/%{::trusted.certname}'
          - 'global'

  ## Server
  - name: 'Machine specific data for one configured node served by puppet.'
    path: 'nodes/%{trusted.certname}.yaml'

Still to no avail.

Writing variables as key value pair

I wonder is it possible to create a kv secret called common and has key value pairs in it for variables? currently each variable needs to be written as secret and has field called value, right?

debouncer issue?

I see this was literally added yesterday. :)

Just installing vault:

Must install debouncer gem to use hiera-vault backend

so I:

puppetserver gem install debouncer (gets 0.2.2):

Internal Server Error: org.jruby.exceptions.RaiseException: (SyntaxError) /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/debouncer-0.2.2/lib/debouncer.rb:86: syntax error, unexpected tLABEL
def flush(id = EMPTY, and_join: false)

Dropping to debouncer 0.1.0:

Evaluation Error: Error while evaluating a Function Call, Function Load Error for function 'hiera_vault': undefined method `shutdown' for #Vault::Client:0x180f2b89

Not sure if this is a WIP, I'm doing something wrong, or ? :)

Thanks!

Caching vault object causes heisenbug on heavy-loaded puppetserver

I have a heavy-loaded puppetserver (about 2k nodes with default 30m checkin interval; 32 workers) with lots of environments (20-30).
Relatively frequently I receive different errors caused by $vault being shut down or nil'ed.
I guess this one occurs when $vault.shutdown happens at the same time as lookup is made for another node:

[hiera-vault] Skipping backend. Configuration error: Vault::ConnectionPool::PoolShuttingDownError

And this one, I guess, happens when $vault = nil during another lookup:

[hiera-vault] Skipping backend. Configuration error: undefined method 'request' for nil:NilClass
[hiera-vault] Skipping backend. Configuration error: undefined method `configure' for nil:NilClass

To be honest, for this one I'm not sure what causes it, but maybe it has something to do with simultaneous shutdown:

[hiera-vault] Skipping backend. Configuration error: no cipher match

For now I solved this problem by removing global vault caching object and changing it back to local variable; adding vault.shutdown instead of $shutdown.call seems to solve problem of CLOSE_WAIT sockets (although right now I see one such socket on my server).
Is there any better way to do this? Maybe fence these operations with mutex or something similar?

Alternative ways to specify Vault token (or secure it)

How to secure the token inside the hiera.yaml? The one that is used to connect to Vault?
Is there any option to user token_file instead of the clear text token in hiera.yaml?
Thanks!

can't setup puppet module

Hello.

I have a very strange problem with module installation.
I've installed vault gem, but I can't to install puppet module:

$ puppet module install davealden/hiera_vault
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Notice: Downloading from https://forgeapi.puppet.com ...
Error: Could not install 'davealden-hiera_vault' (latest)
No releases are available from https://forgeapi.puppet.com
Does 'davealden-hiera_vault' have at least one published release?

I think that my container hasn't internet but I'm not sure this.

Lots of CLOSE_WAITS to Vault

Hi,

Version:

Puppet: 6.1
Vault: 1.0.1
Gem Vault: 0.12.0
Hiera_vault: Latest repo version

Install the hiera_vault and everything working fine. BUT there is a huge number of CLOSE_WAITS. Seems the function doesn't close the connection once it finish. Noticing as well that after removing hiera.yaml all ESTABLISH move to CLOSE_WAIT but the connection still not close.

Before removing hiera.yaml (after removingthe difference is that all connections are CLOSE_WAIT)
10.100.10.101-> Puppetserver
10.100.2.157 -> Vault

netstat -an | grep 10.100.2.157
tcp        0      0 10.100.10.101:39110     10.100.2.157:8200       ESTABLISHED
tcp       54      0 10.100.10.101:37442     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38504     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38530     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38758     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38000     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38506     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38008     10.100.2.157:8200       CLOSE_WAIT 
tcp        0      0 10.100.10.101:39212     10.100.2.157:8200       ESTABLISHED
tcp        0      0 10.100.10.101:39076     10.100.2.157:8200       ESTABLISHED
tcp       54      0 10.100.10.101:37426     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38006     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:37638     10.100.2.157:8200       CLOSE_WAIT 
tcp        0      0 10.100.10.101:39072     10.100.2.157:8200       ESTABLISHED
tcp        0      0 10.100.10.101:39236     10.100.2.157:8200       ESTABLISHED
tcp        0      0 10.100.10.101:39234     10.100.2.157:8200       ESTABLISHED
tcp        0      0 10.100.10.101:39250     10.100.2.157:8200       ESTABLISHED
tcp       54      0 10.100.10.101:37888     10.100.2.157:8200       CLOSE_WAIT 
tcp        0      0 10.100.10.101:39078     10.100.2.157:8200       ESTABLISHED
tcp       54      0 10.100.10.101:38772     10.100.2.157:8200       CLOSE_WAIT 
tcp        0      0 10.100.10.101:39232     10.100.2.157:8200       ESTABLISHED
tcp       54      0 10.100.10.101:38502     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38792     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:37440     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:37886     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:37094     10.100.2.157:8200       CLOSE_WAIT 
tcp        0      0 10.100.10.101:39238     10.100.2.157:8200       ESTABLISHED
tcp       54      0 10.100.10.101:37822     10.100.2.157:8200       CLOSE_WAIT 
tcp        0      0 10.100.10.101:39240     10.100.2.157:8200       ESTABLISHED
tcp       54      0 10.100.10.101:37940     10.100.2.157:8200       CLOSE_WAIT 
tcp        0      0 10.100.10.101:39230     10.100.2.157:8200       ESTABLISHED
tcp       54      0 10.100.10.101:37928     10.100.2.157:8200       CLOSE_WAIT 
tcp        0      0 10.100.10.101:39074     10.100.2.157:8200       ESTABLISHED
tcp       54      0 10.100.10.101:37116     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:37460     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38764     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:37648     10.100.2.157:8200       CLOSE_WAIT 
tcp        0      0 10.100.10.101:39242     10.100.2.157:8200       ESTABLISHED
tcp       54      0 10.100.10.101:37270     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38498     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38766     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:37932     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38624     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38642     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:37448     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38500     10.100.2.157:8200       CLOSE_WAIT 
tcp        0      0 10.100.10.101:39070     10.100.2.157:8200       ESTABLISHED
tcp       54      0 10.100.10.101:37452     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38018     10.100.2.157:8200       CLOSE_WAIT 
tcp        0      0 10.100.10.101:39224     10.100.2.157:8200       ESTABLISHED
tcp       54      0 10.100.10.101:37620     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:37884     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38528     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38524     10.100.2.157:8200       CLOSE_WAIT 
tcp        0      0 10.100.10.101:39246     10.100.2.157:8200       ESTABLISHED
tcp       54      0 10.100.10.101:37130     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:37936     10.100.2.157:8200       CLOSE_WAIT 
tcp        0      0 10.100.10.101:39098     10.100.2.157:8200       ESTABLISHED
tcp       54      0 10.100.10.101:38770     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38768     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:37838     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38014     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:38756     10.100.2.157:8200       CLOSE_WAIT 
tcp        0      0 10.100.10.101:39208     10.100.2.157:8200       ESTABLISHED
tcp       54      0 10.100.10.101:37332     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:37938     10.100.2.157:8200       CLOSE_WAIT 
tcp       54      0 10.100.10.101:37852     10.100.2.157:8200       CLOSE_WAIT 

And here hiera

---
version: 5
hierarchy:
  - name: "Hiera-vault lookup"
    lookup_key: hiera_vault
    options:
      confine_to_keys:
        - '^vault_.*'
        - '^.*_password$'
        - '^password.*'
      ssl_verify: true
      ssl_ca_cert: /etc/pki/tls/cert.pem
      address: https://vault.server.com:8200
      token: blablabla
      default_field: value
      mounts:
        secret:
          - puppet/%{::trusted.certname}

Citing the Red Hat docs:

CLOSE_WAIT - Indicates that the server has received the first FIN signal from the client and the connection is in the process of being closed. This means the socket is waiting for the application to execute close(). A socket can be in CLOSE_WAIT state indefinitely until the application closes it. Faulty scenarios would be like a file descriptor leak: server not executing close() on sockets leading to pile up of CLOSE_WAIT sockets.

changelog not updated for 2.0.0

There is no changelog entry for release 2.0.0

Integration into hiera-eyaml

Any chance this could be turned into a plugin for hiera-eyaml?

get a specific version secret

How do I get a specific version secret from a vault? I try to do it like this,
puppet lookup vault::testsecret?version=2 --explain --compile --node mynode
but it doesn't work.

Sample vault secrets in README, Documentation of default_field_*

The documentation currently talks about creating a new mount for secrets that Puppet can access. It would be nice if the documentation also included some sample secrets that the supplied configuration could retrieve. It took me a long time to figure out how to set hiera.yaml and the secret being retrieved in the lookup() function to be able to access existing secrets, but if there were sample secrets in the documentation, we could just make sure we keep them like that.

Also as part of that searching, I stumbled across default_field_behavior and default_field_parse. I don't see any documentation of either of those two options. I can guess based on the code, but it would be nice to know for sure how they are intended to work.

Feature Request: Renew Token Key

Hi

It would be nice if there was an option like on consul-template to renew the token (once we have manually create it for the first time). Just an idea.

Thanks

Joaquin

Puppet 7 Support and New Forge Version

We're upgrading to Puppet 7, as 6 is EOL this month. This module indicates a requirement range for Puppet excluding 7, but considering that Hiera 5 is still in use in Puppet 7, I can't think of a reason this wouldn't continue to work.

Has this been tested with the latest version of Puppet? If so, can metadata.json get an update and a new dot-version be pushed to the Forge?

Thanks for your great work on this.

Lots of CLOSE_WAITS to Vault on petems-hiera_vault v2.0.0

Hi,

Version:
puppetserver version: 6.8.0
Puppet v6.22.1
petems-hiera_vault v2.0.0

Puppet agent:
Puppet v6.13.0

Vault 1.12.0

Haproxy 2.1.4

After the installing hiera_vault we begin to see a lot of CLOSE_WAITS, it looks like application does not close the connection properly.

#25 - we found the previous issue about it, but the problem begin to reproduce on the latest version also.

Puppet server does not send fin package from it side:

16:56:43.070098 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [S], seq 3899952215, win 43690, options [mss 65495,nop,nop,sackOK,nop,wscale 7], length 0
16:56:43.070120 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [S.], seq 1096888154, ack 3899952216, win 43690, options [mss 65495,nop,nop,sackOK,nop,wscale 7], length 0
16:56:43.070135 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [.], ack 1, win 342, length 0
16:56:43.109275 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [P.], seq 1:227, ack 1, win 342, length 226
16:56:43.109296 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [.], ack 227, win 350, length 0
16:56:43.150004 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [P.], seq 1:1950, ack 227, win 350, length 1949
16:56:43.150011 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [.], ack 1950, win 1365, length 0
16:56:43.206679 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [P.], seq 227:309, ack 1950, win 1365, length 82
16:56:43.206689 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [.], ack 309, win 350, length 0
16:56:43.206814 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [P.], seq 309:315, ack 1950, win 1365, length 6
16:56:43.206820 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [.], ack 315, win 350, length 0
16:56:43.207070 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [P.], seq 315:384, ack 1950, win 1365, length 69
16:56:43.207077 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [.], ack 384, win 350, length 0
16:56:43.207638 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [P.], seq 1950:2025, ack 384, win 350, length 75
16:56:43.207646 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [.], ack 2025, win 1365, length 0
16:56:43.211176 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [P.], seq 384:837, ack 2025, win 1365, length 453
16:56:43.212136 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [P.], seq 2025:2574, ack 837, win 359, length 549
16:56:43.219959 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [P.], seq 837:1338, ack 2574, win 1396, length 501
16:56:43.223363 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [P.], seq 2574:2883, ack 1338, win 367, length 309
16:56:43.262512 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [.], ack 2883, win 1426, length 0

vault try to close connection from it side and send tcp fin:
17:01:43.224629 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [F.], seq 2883:2936, ack 1338, win 367, length 53
17:01:43.264539 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [.], ack 2937, win 1426, length 0

and number of close_wait begin to increase from time to time:
CLOSE_WAIT
netstat -ntulpa | grep CLOSE_WAIT
tcp 0 0 x.x.x.1:33124 x.x.x.2:8200 CLOSE_WAIT 36977/haproxy
tcp 54 0 127.0.0.1:48228 127.0.0.1:5000 CLOSE_WAIT 42943/java

hiera config:

• name: Hiera-vault lookup
  lookup_key: hiera_vault
  options:
  confine_to_keys:
  - '^.*password$'
  ssl_verify: false
  ssl_ca_cert: <path_to_cert>
  address: <haproxy address, which balance and forward traffic to Vault>
  token:
  default_field: secret
  mounts:
  puppet:
  - ''

Release 2.0.0 is not tagged

There is no tag/release for 2.0.0 in the git repo

Secret search path joined into one path instead of iteration over paths

hierarchy:
  - name: "Hiera-vault lookup"
    lookup_key: hiera_vault
    options:
      confine_to_keys:
        - '^vault_.*'
      ssl_verify: false
      address: 'http://127.0.0.1:8200'
      token: '/etc/puppetlabs/puppetserver/vault_token'
      default_field: value
      mounts:
        puppet:
          - "%{::trusted.certname}"
          - common

Output:

  Environment Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppetlabs/code/environments/env_name/hiera.yaml"
    Hierarchy entry "Hiera-vault lookup"
      No such key: "vault_key"
      [hiera-vault] Client configured to connect to http://127.0.0.1:8200
      [hiera-vault] Looking in path puppet/certname/common/vault_key

Expecter behavior:

[hiera-vault] Looking in path puppet/certname/vault_key
[hiera-vault] Looking in path puppet/common/vault_key

Returned value should be not_found() if secret not found

If I do a Vault lookup for a secret, and it can't be found, currently the code returns nil, instead of returning the not_found method.

@hlindberg explained it well on the Puppet slack:

hlindbe [13:56]
yep - it finds a `nil`
the hiera-vault function should not return `nil` if it detects a “not found” - it should call the context method `not_found` instead
if it returns a `nil` it means that the value IS `undef` in puppet - i.e. a valid value

This means, with Puppet code like this:

$vault_notify = lookup({"name" => "vault_notify", "default_value" => "No Vault Secret Found"})
notify { "testing vault ${vault_notify}":}

It will return a string with an undef value, rather than seeing that it wasnt found and returning the default value:

Notice: /Stage[main]/Profile::Vault_message/Notify[testing vault ]/message: defined 'message' as 'testing vault '

Add note to readme about puppetserver JRUBY

Need to add to docs:

ini_setting { "Change jruby to 9k":
    ensure  => present,
    setting => 'JRUBY_JAR',
    path    => "/etc/sysconfig/puppetserver",
    key_val_separator => '=',
    section => '',
    value   => '"/opt/puppetlabs/server/apps/puppetserver/jruby-9k.jar"',
    show_diff => true,
    notify  => Class['puppetserver::service']
  }

See #5, #14 and possibly #18

Mark fields as Sensitive

Has anyone looked whether it's possible to mark values coming from this provider as Sensitive? I know this may be quite complex to configure correctly, but this may prevent passwords to show up in diffs 😦

References:

• https://puppet.com/docs/puppet/6.17/lang_data_sensitive.html
• https://puppet.com/blog/my-journey-securing-sensitive-data-puppet-code/

can't use standby node

I have a Vault cluster and, since hiera is read-onyl, I wanted to use either active and standby node.

I do not have an in depth knowledge of Vault, hence I am not sure if this is a limitation of Vault, a problem with my configuration a problem with hiera_vault, a problem the ruby gem... a problem with something else 😃

Problems with later versions of vault gem and puppetserver

In a Puppet 5 server environment using the latest version of the vault-ruby gem, Puppet can successfully perform lookups using puppet lookup from the command line using this module.

However, trying to do the same via a puppet run (i.e. puppet agent -t), which is served by puppetserver, fails.

We think this is due to an older version of Ruby still being used in the latest puppetserver; Puppet 5.2.0 has Ruby 2.4.1p111, puppetserver version 5.1.0 only has Ruby 1.9.3p551.

This is not the fault of this module - but to make it work for a puppet agent -t run, you may need to roll back your vault-ruby gem even as far as 0.6.0. I haven't investigated exactly at which version it becomes incompatible with puppetserver.

Other than by manually upgrading the Ruby version in puppetserver, we have to wait until newer Ruby is included by Puppet.

vault caching

hey @petems
out of curiosity, is caching supported as described here: Vault Agent Caching ?
is it supported by the vault gem?
If it's supported by the ruby gem, are we using it in hiera_vault ?

p.s.: this is somehow in between an information request and a feature request

question: using wildcards in mounts paths

Hi, I'm looking for the way to use wildcards in mounts paths.
I would need something like this to make work, but it doesn't with hiera_vault in my hiera5 config:

---
version: 5

hierarchy:
  - name: "Hiera-vault lookup"
    lookup_key: hiera_vault
    options:
      confine_to_keys:
        - "^vault_.*"
        - "^.*_password$"
        - "^password.*"
      ssl_verify: false
      address: https://vault.foobar.com:8200
      token: <insert-your-vault-token-here>
      default_field: value
      mounts:
        some_secret:
          - %{::trusted.certname}/*
          - common/*
        another_secret:
          - %{::trusted.certname}/*
          - common/*

Any hint, please?
Thanks

handshake_failure

when trying to read from vault I am getting following on puppet agent:
$ sudo puppet agent -tv --tags test
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Lookup of key 'libnss-ldap_secret' failed: [hiera-vault] Skipping backend. Configuration error: Received fatal alert: handshake_failure (file: /etc/puppetlabs/code/environments/stretch/modules/test/manifests/init.pp, line: 5, column: 13) on node kicktest.domain.com
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

tcpdump confirms it is handshake failure

puppetserver -v
puppetserver version: 5.3.9

puppet module list | grep vault
davealden-hiera_vault (v0.9.0) - that is the 0.1.0 version from git

puppetserver gem list

*** LOCAL GEMS ***

aws-sigv4 (1.0.3)
columnize (0.9.0)
debouncer (0.2.2)
deep_merge (1.0.1)
did_you_mean (default: 1.0.1)
fast_gettext (1.1.2)
gettext (3.2.2)
hiera-eyaml (3.0.0, 2.1.0)
hiera-vault (0.2.2)
highline (1.6.21)
hocon (1.2.5)
jar-dependencies (default: 0.3.12)
jruby-openssl (0.9.21 java)
jruby-readline (1.2.2 java)
json (1.8.3 java)
linecache (1.3.1 java)
locale (2.1.2)
minitest (default: 5.4.1)
multi_json (1.13.1)
net-telnet (default: 0.1.1)
optimist (3.0.0)
power_assert (default: 0.2.3)
psych (2.2.4 java)
puppetserver-ca (1.2.1)
rake (default: 10.4.2)
rdoc (default: 4.2.0)
ruby-debug (0.10.6)
ruby-debug-base (0.10.6 java)
semantic_puppet (0.1.3, 0.1.2)
test-unit (default: 3.1.1)
text (1.3.1)
trollop (2.9.9)
vault (0.11.0)

puppetserver ruby --version
jruby 9.1.16.0 (2.3.3) 2018-02-21 8f3f95a OpenJDK 64-Bit Server VM 25.222-b10 on 1.8.0_222-8u222-b10-1~deb9u1-b10 +jit [linux-x86_64]

/etc/puppetlabs/puppet/hiera.yaml
version: 5
hierarchy:

• name: "Hiera-vault lookup"
  lookup_key: hiera_vault
  options:
  confine_to_keys:
  - ".*secret$"
  - ".*vault$"
  ssl_verify: false
  ssl_pem_file: /etc/puppetlabs/puppet/ssl/ssl_pem_file.pem
  ssl_ca_cert: /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem
  address: https://vault.domain.com:8200
  token: s.x6kVhjgf67gjM2p4UXlzBB65jg876876bSFfC4hn
  default_field: value
  mounts:
  generic:
  - secret/puppet/%{::environment}/
  - secret/puppet/

cat /etc/puppetlabs/code/environments/stretch/modules/test/manifests/init.pp
class test {

$secret = lookup({"name" => "libnss-ldap_secret", "value_type" => String, "merge" => "first"})
notify {"testing vault ${secret}":}
file { '/tmp/test':
content => $secret,
}

notify {"testing vault1":}
file { '/tmp/test1':
content => lookup({"name" => "libnss-ldap_secret", "value_type" => String, "merge" => "first"}),

}
}

PUPPET LOOKUP AND CURL WORKS

puppet lookup libnss-ldap_secret --explain
Searching for "lookup_options"
Global Data Provider (hiera configuration version 5)
Using configuration "/etc/puppetlabs/puppet/hiera.yaml"
Hierarchy entry "Hiera-vault lookup"
No such key: "lookup_options"
[hiera-vault] Skipping hiera_vault backend because key 'lookup_options' does not match confine_to_keys
Environment Data Provider (hiera configuration version 5)
Using configuration "/etc/puppetlabs/code/environments/stretch/hiera.yaml"
Merge strategy hash
Hierarchy entry "Per-node data (yaml version)"
Path "/etc/puppetlabs/code/environments/stretch/data/nodes/sppuppet03test.domain.com.yaml"
Original path: "nodes/%{::trusted.certname}.yaml"
Path not found
Hierarchy entry "Other YAML hierarchy levels"
Path "/etc/puppetlabs/code/environments/stretch/data/common.yaml"
Original path: "common.yaml"
Path not found
Searching for "libnss-ldap_secret"
Global Data Provider (hiera configuration version 5)
Using configuration "/etc/puppetlabs/puppet/hiera.yaml"
Hierarchy entry "Hiera-vault lookup"
Found key: "libnss-ldap_secret" value: "test"
[hiera-vault] Client configured to connect to https://vault.domain.com:8200
[hiera-vault] Looking in path secret/puppet/stretch/libnss-ldap_secret
[hiera-vault] Looking in path secret/puppet/libnss-ldap_secret
[hiera-vault] Read secret: libnss-ldap_secret

curl -v https://vault.domain.com:8200 -E /etc/puppetlabs/puppet/ssl/ssl_pem_file.pem --cacert /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem

• Rebuilt URL to: https://vault.domain.com:8200/
• Trying 10.99.4.100...
• TCP_NODELAY set
• Connected to vault.domain.com (10.99.4.100) port 8200 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@strength
• successfully set certificate verify locations:
• CAfile: /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem
  CApath: /etc/ssl/certs
• TLSv1.2 (OUT), TLS header, Certificate Status (22):
• TLSv1.2 (OUT), TLS handshake, Client hello (1):
• TLSv1.2 (IN), TLS handshake, Server hello (2):
• TLSv1.2 (IN), TLS handshake, Certificate (11):
• TLSv1.2 (IN), TLS handshake, Server key exchange (12):
• TLSv1.2 (IN), TLS handshake, Server finished (14):
• TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
• TLSv1.2 (OUT), TLS change cipher, Client hello (1):
• TLSv1.2 (OUT), TLS handshake, Finished (20):
• TLSv1.2 (IN), TLS change cipher, Client hello (1):
• TLSv1.2 (IN), TLS handshake, Finished (20):
• SSL connection using TLSv1.2 / DHE-RSA-AES256-GCM-SHA384
• ALPN, server did not agree to a protocol
• Server certificate:
• subject: OU=Domain Control Validated; CN=*.domain.com
• start date: Feb 20 18:58:38 2019 GMT
• expire date: Apr 21 17:34:11 2020 GMT
• subjectAltName: host "vault.domain.com" matched cert's "*.domain.com"
• issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2
• SSL certificate verify ok.

GET / HTTP/1.1
Host: vault.domain.com:8200
User-Agent: curl/7.52.1
Accept: /

You have similar ticker, where lookup works but puppet agent doesn't, not sure if related.
Would you be able to help to found out the root cause ?

Lookup-error in strict_mode in case of not found secret

Hello,
we use hiera_vault v2.0.0 and we have had recently issues with our vault-server, because vault-server has answered some requests with HTTP 500 errors. HTTP 500 errors have been caught from hiera_vault in version v2.0.0 without special handling and we have had some consequential error.
Anyway, in a current project I have seen promising new flag strict_mode with following description: “When enabled, the lookup function fail in case of http errors when looking up a secret.”
I tested strict_mode with HTTP-errors and behaviour is exact what I am looking for, but I do not understand the lookup-error if hiera_vault does not find values. Below please find the relevant source-code:
raise Puppet::DataBinding::LookupError, "[hiera-vault] Could not find secret #{key}" if answer.nil? and strict_mode
This behaviour does not reflect above mentioned description of strict_mode and I query why is this relevant for strict_mode at all? Everyone can decide with puppet-lookup if lookup should fail without value, right? Or have I missed something?
Cheers,
Andreas

No value returned by hiera-vault backend

When I try to look up a key from vault using hiera-vault, I get no output back:

# puppet lookup vault_password
#

I ran puppet lookup with debug mode enabled:

# puppet lookup vault_password --debug
...snip...
  Searching for "vault_password"
    Global Data Provider (hiera configuration version 5)
      Using configuration "/etc/puppetlabs/puppet/hiera.yaml"
      Hierarchy entry "Vault"
        No such key: "vault_password"
        [hiera-vault] Client configured to connect to http://localhost:8200
        [hiera-vault] Looking in path secret/puppet/ubuntu-bionic/vault_password
        [hiera-vault] Looking in path secret/puppet/common/vault_password
...snip...

However, I can get the value back when I run vault kv get secret/puppet/common/vault_password:

# vault kv get secret/puppet/common/vault_password
====== Metadata ======
Key              Value
---              -----
created_time     2018-10-04T16:55:16.423687622Z
deletion_time    n/a
destroyed        false
version          5

=== Data ===
Key    Value
---    -----
foo    bar

Here's my vault key/value:

# vault kv get secret/puppet/common/vault_password
====== Metadata ======
Key              Value
---              -----
created_time     2018-10-04T16:55:16.423687622Z
deletion_time    n/a
destroyed        false
version          5

=== Data ===
Key    Value
---    -----
foo    bar

Here's my hiera.yaml:

hierarchy:
  - name: Vault
    lookup_key: hiera_vault
    options:
      confine_to_keys:
        - '^vault_.*'
      ssl_verify: false
      address: http://localhost:8200
      token: <omitted>
      default_field: value
      mounts:
        generic:
          - secret/puppet/%{::trusted.certname}/
          - secret/puppet/common/

I'm running the following software versions:

# vault --version
Vault v0.11.2 ('2b1a4304374712953ff606c6a925bbe90a4e85dd')

# puppet --version
5.5.6

# puppetserver --version
puppetserver version: 5.3.5

# puppetserver ruby --version
jruby 9.1.16.0 (2.3.3) 2018-02-21 8f3f95a OpenJDK 64-Bit Server VM 25.181-b13 on 1.8.0_181-8u181-b13-0ubuntu0.18.04.1-b13 +jit [linux-x86_64]

# puppetserver gem list

*** LOCAL GEMS ***

aws-sigv4 (1.0.3)
debouncer (0.2.2)
deep_merge (1.0.1)
did_you_mean (default: 1.0.1)
facter (2.5.1)
fast_gettext (1.1.2)
gettext (3.2.2)
hiera-eyaml (2.1.0)
highline (1.6.21)
hocon (1.2.5)
jar-dependencies (default: 0.3.12)
jruby-openssl (0.9.21 java)
jruby-readline (1.2.2 java)
json (1.8.3 java)
locale (2.1.2)
minitest (default: 5.4.1)
multi_json (1.13.1)
net-telnet (default: 0.1.1)
power_assert (default: 0.2.3)
psych (2.2.4 java)
puppetserver-ca (0.3.1)
rake (default: 10.4.2)
rdoc (default: 4.2.0)
semantic_puppet (0.1.3, 0.1.2)
test-unit (default: 3.1.1)
text (1.3.1)
trollop (2.1.3, 2.1.2)
vault (0.12.0)

# /opt/puppetlabs/puppet/bin/ruby --version
ruby 2.4.4p296 (2018-03-28 revision 63013) [x86_64-linux]

# /opt/puppetlabs/puppet/bin/gem list

*** LOCAL GEMS ***

aws-sigv4 (1.0.3)
bigdecimal (default: 1.3.2)
debouncer (0.2.2)
deep_merge (1.0.1)
did_you_mean (1.1.0)
facter (3.11.4, 2.5.1)
fast_gettext (1.1.2)
gettext (3.2.2)
gettext-setup (0.30)
hiera (3.4.4)
hiera-eyaml (2.1.0)
highline (1.6.21)
hocon (1.2.5)
io-console (default: 0.4.6)
json (default: 2.0.4)
locale (2.1.2)
minitest (5.10.1)
multi_json (1.13.1)
net-ssh (4.2.0)
net-telnet (0.1.1)
openssl (default: 2.0.7)
power_assert (0.4.1)
psych (default: 2.2.2)
puppet (5.5.6)
puppetserver-ca (0.3.1)
rake (12.0.0)
rdoc (default: 5.0.0)
semantic_puppet (0.1.2)
stomp (1.4.4)
test-unit (3.2.3)
text (1.3.1)
trollop (2.1.2)
vault (0.12.0)
xmlrpc (0.2.1)

I'm just running Vault in -dev mode locally at the moment to kick the tires on this backend, so there shouldn't be anything fancy.

What does "No such key" mean in this context? Have I misconfigured something?

Any help appreciated. :)