Comments (21)
Upgrade is possible, we did that. Read the documentation for v2 store, there's instructions for upgrading: https://www.vaultproject.io/docs/secrets/kv/kv-v2.html
Note: you have to modify your policies and clients to work with new paths.
from petems-hiera_vault.
@petems is it possible to upgrade the keystore on Vault from v1 to v2? Or do we need to copy the keys one by one
from petems-hiera_vault.
see PR #36 regarding one approach to supporting kv2 (among a bunch of other things), still needs a modification to hashicorp/vault-ruby (see PR 201)
from petems-hiera_vault.
See also PR #41 which propose another approach to be v1 and v2 compatible.
from petems-hiera_vault.
See also PR #41 which propose another approach to be v1 and v2 compatible.
@arcenik can you please check the #41 PR, please? It feels like the data
part should be constructed differently, resp. it should be put right behind the mount
part and then the rest of the path. Current version of the code doesn't work for me (even though it was merged into master already). Simple hack (sorry, not a rubyist) made it work for me:
--- /opt/puppetlabs/puppet/cache/lib/puppet/functions/hiera_vault.rb 2019-11-12 12:48:18.320664160 +0000
+++ /tmp/puppet-file20191112-396-1jzwdd7 2019-11-12 13:56:24.046216794 +0000
@@ -125,15 +125,15 @@
kv_mounts.each_pair do |mount, paths|
paths.each do |path|
- secretpath = context.interpolate(File.join(mount, path))
-
- context.explain { "[hiera-vault] Looking in path #{secretpath} for #{key}" }
-
begin
+ secretpath = context.interpolate(File.join(mount, path))
+ context.explain { "[hiera-vault] Looking in path #{secretpath} for #{key}" }
secret = get_kv_v1(secretpath, key)
if secret.nil?
- secret = get_kv_v2(secretpath, key)
+ secretpath_v2 = context.interpolate(File.join(mount, 'data', path))
+ context.explain { "[hiera-vault] Looking in path #{secretpath_v2} for #{key} via KV version 2" }
+ secret = get_kv_v2(secretpath_v2, key)
end
rescue Vault::HTTPConnectionError
@@ -189,7 +189,7 @@
end
def get_kv_v2(secretpath, key)
- res = $vault.logical.read(File.join(secretpath,'data',key))
+ res = $vault.logical.read(File.join(secretpath,key))
if ! res.nil?
res=res.data[:data]
end
from petems-hiera_vault.
Hello @jhejl
I think that you should make you own PR or create a new issue instead of putting file diff in an issue concerning an already merge PR.
Although, I don't see the point of moving the 'data' part of the path outside of get_kv_v2 function.
As it does not appear in the web UI of Vault, there no point to display it on the context.explain.
Best regards.
from petems-hiera_vault.
wonder if #41 is usable and if we can create a tag.
from petems-hiera_vault.
I just tested the change in #41 and it looks like it didnt work (I tried in my test environment at the time and it had but I think it was misconfifured)
I might be able to give this a go over the weekend and re-work it more
from petems-hiera_vault.
I can confirm that the changes from #41 do not work with KV2.
I created to mounts - hiera_backend (as kv1) and hiera_backend (as kv2) with the same parameters inside.
---
version: 5
defaults:
datadir: data
data_hash: yaml_data
hierarchy:
- name: 'Vault kv2 backend'
lookup_key: hiera_vault
options:
confine_to_keys:
- '^vault_.*'
- '^.*_password$'
- '^password.*'
- 'webserver::php_fpm_enable'
address: 'https://dev01.example.tld:8200/'
token: /etc/puppetlabs/puppet/vault-token
default_field: value
mounts:
hiera_backend2:
- '%{trusted.certname}'
- 'common'
- name: 'Vault kv backend'
lookup_key: hiera_vault
options:
confine_to_keys:
- '^vault_.*'
- '^.*_password$'
- '^password.*'
- 'webserver::php_fpm_enable'
address: 'https://dev01.example.tld:8200/'
token: /etc/puppetlabs/puppet/vault-token
default_field: value
mounts:
hiera_backend:
- '%{trusted.certname}'
- 'common'
Current result:
-bash# puppet lookup webserver::php_fpm_enable --explain --compile --node=web02.example.tld
Searching for "webserver::php_fpm_enable"
Global Data Provider (hiera configuration version 5)
No such key: "webserver::php_fpm_enable"
Environment Data Provider (hiera configuration version 5)
Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
Hierarchy entry "Vault kv2 backend"
No such key: "webserver::php_fpm_enable"
[hiera-vault] Client configured to connect to https://dev01.example.tld:8200/
[hiera-vault] Looking in path hiera_backend2/web02.example.tld for webserver::php_fpm_enable
[hiera-vault] Looking in path hiera_backend2/common for webserver::php_fpm_enable
Hierarchy entry "Vault kv backend"
Found key: "webserver::php_fpm_enable" value: false
Would be great if someone could provide a working solution for KV2.
from petems-hiera_vault.
Can you also provides corresponding audit log from vault ?
from petems-hiera_vault.
Can you also provides corresponding audit log from vault ?
Yep, vault responses with an error 404. Seems like something is wrong in the api call.
{
"time": "2019-11-20T23:47:54.805571348Z",
"type": "response",
"auth": {
"client_token": "hmac-sha256:fab71a05a86763d6c3988d32a3ed91cf9adc114b831eecc64902e435f586cefa",
"accessor": "hmac-sha256:99664f96d2c3f373ae5e8e541f599b0d88ca61108d1a223072e8bfe317c6a120",
"display_name": "root",
"policies": [
"root"
],
"token_policies": [
"root"
],
"metadata": null,
"entity_id": "",
"token_type": "service"
},
"request": {
"id": "ee9f4b23-c593-410f-a2fd-c59aa818e6ee",
"operation": "read",
"client_token": "hmac-sha256:fab71a05a86763d6c3988d32a3ed91cf9adc114b831eecc64902e435f586cefa",
"client_token_accessor": "hmac-sha256:99664f96d2c3f373ae5e8e541f599b0d88ca61108d1a223072e8bfe317c6a120",
"namespace": {
"id": "root",
"path": ""
},
"path": "hiera_backend2/common/data/webserver::php_fpm_enable",
"data": null,
"policy_override": false,
"remote_address": "127.0.0.1",
"wrap_ttl": 0,
"headers": {}
},
"response": {
"data": {
"http_content_type": "hmac-sha256:ed1d318d9cbb5093fabe96bc06c826d045201a8f32913aee1c722023f3b23f47",
"http_raw_body": "hmac-sha256:c7ef757aeaf9e7d3dc1ee2e644ff470d3e7d0afd46efe8bc380d9b77ba4aa098",
"http_status_code": 404
},
"headers": null
},
"error": ""
}
from petems-hiera_vault.
This is wrong:
$vault.logical.read(File.join(secretpath,'data',key))
but it's fairly easy to fix. I write down a possible solution, as a reminder (and I can try a PR myself, tomorrow):
key_path = File.dirname(key)
key_name = File.basename(key)
res = $vault.logical.read(File.join(secretpath,'data,key_path)).data[:data][:"#{key_name}"]
from petems-hiera_vault.
First of all, my previous message was slightly wrong, however, I managed the create #43 (on top of #41) which works for me.
- One issue was a trailing slash, which used to work with v1, but it doesn't with v2. I have added a
chomp
, in case someone re-uses an existing configuration. - the second part, consists of handling the hash in a different way compared to v1. The value is no longer a key called value, but a key whose name is the name of the initial key.
Please test it, improve it.
from petems-hiera_vault.
For your reference: Asciinema preview: Vault secrets shuffler
this is how I am using.
IMO it's only matter of agreeing if the key name should be value
of the name of the key
. In this case I'm using the name of the key.
p.s.: despite the name of the mount, this is v2, and not v1
from petems-hiera_vault.
I'm using the default_field set to value and your solution does not work with my setup as you do not use the same mapping between hiera and vault.
My mapping:
hiera | vault |
---|---|
mount + path | secret engine name |
lookup name | secret name |
default_field = "value" | key |
Your mapping:
hiera | vault |
---|---|
mount | secret engine name |
path | secret name |
lookup name | key |
Depending of the mapping, you need to place the 'data' item (for v2) need to be placed on a different place.
from petems-hiera_vault.
I'll try to look at this ... now I'm commuting: Cambridge (UK), to Netherlands :)
Because we really need to get this working :)
The fact that "it crashes", I believe is fine and it does it with every backend, if the default value is not supplied.
from petems-hiera_vault.
Ok, I've merged #43 and in my tests, the code now works for v1 and v2 backends.
I've got one last PR with added tests #44, as the other code didn't take into account people not using the default_field
option, where if you want to use data not being value
it didn't work.
from petems-hiera_vault.
ping :)
@petems thanks for the awesome module! Can you tell me about the current state for v2 support? It seems to be somehow almost kind of readyish?
from petems-hiera_vault.
Oh it's been ready and merged for ages now, I forgot to close this issue
from petems-hiera_vault.
@petems can you release latest version to forge please ?
from petems-hiera_vault.
Hmm, I thought my pipeline should've done that already, let me check
from petems-hiera_vault.
Related Issues (20)
- Setting token: 'ENV["VAULT_TOKEN"]' doesn't work HOT 10
- If the Vault token becomes unavailable to the Hiera/Vault module, a secret is substituted with an empty string HOT 6
- Mark fields as Sensitive HOT 1
- vault caching HOT 3
- IGNORE-VAULT doesn't work correctly if set in the file. HOT 2
- puppet lookup does not work as desired HOT 2
- Caching vault object causes heisenbug on heavy-loaded puppetserver HOT 14
- Supporting AWS IAM/EC2 authentication HOT 1
- When get the secrets if the secret contain a dollar (example$rest) sign only retrieve the secret until the dollar sign (example) HOT 2
- Release 2.0.0 is not tagged HOT 1
- changelog not updated for 2.0.0 HOT 2
- get a specific version secret HOT 1
- Testing - Running a real vault server is flaky
- Writing variables as key value pair HOT 1
- question: using wildcards in mounts paths
- Integration into hiera-eyaml
- Lots of CLOSE_WAITS to Vault on petems-hiera_vault v2.0.0 HOT 4
- Puppet 7 Support and New Forge Version
- Alternative ways to specify Vault token (or secure it) HOT 1
- Lookup-error in strict_mode in case of not found secret
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from petems-hiera_vault.