Comments (13)
@jasonodonnell Which consul helm chart I should install for HA option? Can you guide me in terms of installing consul and vault helm for HA and any specific configuration changes that I may need to do for Aws EKS.
from vault-helm.
Hey @msidd111, sorry for the delay but HashiConf is this week so we've been busy! :)
Which consul helm chart I should install for HA option?
The latest version of consul-helm
is fine.
Unfortunately Hashicorp Helm chart does not have an example for EKS.
I agree we should have an example of EKS with auto-unseal in the documentation. I will add this soon.
In the meantime here's how it can be done:
EKS and Vault Helm Unseal Example
First, create a secret with your KMS access key/secret:
Note: I used environment variables to populate the values from my workstation
kubectl create secret generic vault-aws \
--from-literal=AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID?}" \
--from-literal=AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY?}"
Next, clone the repo:
mkdir ~/vault-eks && cd ~/vault-eks
git clone [email protected]:hashicorp/vault-helm.git
Next, create a custom values-eks.yaml
file outside of the vault-helm
directory so we can just inject our custom values:
Note: You need to fill out the seal parameters below.
cat >~/vault-eks/values-eks.yaml <<EOL
server:
extraSecretEnvironmentVars:
- envName: AWS_ACCESS_KEY_ID
secretName: vault-aws
secretKey: AWS_ACCESS_KEY_ID
- envName: AWS_SECRET_ACCESS_KEY
secretName: vault-aws
secretKey: AWS_SECRET_ACCESS_KEY
ha:
enabled: true
config: |
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
}
seal "awskms" {
region = "KMS_REGION_HERE"
kms_key_id = "KMS_KEY_ID_HERE"
}
storage "consul" {
path = "vault"
address = "HOST_IP:8500"
}
EOL
Assuming consul-helm is already deployed and ready, we can now deploy Vault:
helm install --name=vault -f values-eks.yaml ~/vault-eks/vault-helm
Once deployed you can initialize vault-0
and it will auto unseal using AWS KMS:
kubectl exec -ti vault-0 -- vault operator init
from vault-helm.
@msidd111 At this time changing the type of service is not supported. You can make your own service though:
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
name: vault-lb
labels:
app: vault-lb
spec:
ports:
- port: 8200
targetPort: 8200
selector:
app.kubernetes.io/name: vault
component: server
type: LoadBalancer
EOF
from vault-helm.
Thank you for your quick response. Much Obliged. Will let you if I run into any issues or questions I may have.
from vault-helm.
@jasonodonnell I was able to configure vault on consul fine but consul seems to fail resilience test when I restarted all of kubernetes nodes. One of the consul node will stuck with pending state and other will not transition to ready state. Here is error message from consul pod that was stuck in pending state.
Events:
Type Reason Age From Message
Warning FailedScheduling 43m (x2 over 43m) default-scheduler 0/2 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules, 1 node(s) had volume node affinity conflict.
Warning FailedScheduling 42m (x6 over 42m) default-scheduler 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules, 1 node(s) had taints that the pod didn't tolerate, 1 node(s) had volume node affinity conflict.
Warning Fa
from vault-helm.
@msidd111 This is not a Vault Helm issue but a Consul Helm issue. That being said, Vault Helm and Consul Helm use affinity rules to spread out pods onto multiple Kubernetes worker nodes. You are failing the affinity check (it appears you don't have enough worker nodes). You would need at least 3 worker nodes and enough volumes to support each pod.
You could remove consul-helms affinity rules by removing the rule but that's not recommended as it weakens the HA deployment strategy. It's recommended to satisfy the affinity rule if possible.
from vault-helm.
Thanks. This is issue with consul affinity rules so I removed podAntiAffinity and I do get three pods distributed on 2 nodes instead three which podAntiAffinity should enforce so this addresses the issue of consul nodes going to pending state after restarting nodes with caveat that I will have weak HA.
from vault-helm.
@jasonodonnell In next test, I deleted vault-0 which was brought back by cluster but during this time around 10 seconds I was getting error: unable to upgrade connection: container not found ("vault"). Is this expected behavior or one of other two nodes vault-1 and vault-2 should start serving traffic without downtime once vault-0 is down
from vault-helm.
BTW, I was able to access vault when consul-0 pod was deleted.
from vault-helm.
@msidd111 Failover is not instant and could take a few seconds before the lock is released. Where are you seeing the error: unable to upgrade connection: container not found ("vault")
error?
from vault-helm.
https://github.com/hashicorp/vault/pull/7759/files EKS documentation has been added to the official doc website. Closing.
from vault-helm.
trying this setup on EKS
Here is my consul helm config:
syncCatalog:
consulNamespaces:
consulDestinationNamespace: "opx-vault"
client:
enabled: true
Storage: 1Gi
StorageClass: standard-encrypted
server:
replicas: 2
bootstrapExpect: 1
disruptionBudget:
maxUnavailable: 0
Using consul helm chart version: 0.18.0
helm install consul -f consul-eks/helm-consul-values.yaml vault-consul/consul-helm/
Vault config yaml:
server:
extraSecretEnvironmentVars:
- envName: AWS_ACCESS_KEY_ID
secretName: vault-aws
secretKey: AWS_ACCESS_KEY_ID
- envName: AWS_SECRET_ACCESS_KEY
secretName: vault-aws
secretKey: AWS_SECRET_ACCESS_KEY
ha:
enabled: true
replicas: 2
config: |
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
}
seal "awskms" {
region = "cn-north-1"
kms_key_id = "XXXXXXXXXXX"
endpoint = "kms.cn-north-1.amazonaws.com.cn"
}
storage "consul" {
path = "vault"
address = "HOST_IP:8500"
}
Using vault 0.4.0 chart
helm install vault -f vault-eks/values-eks.yaml vault-eks/vault-helm/
At time of launch vault unseal but after that it start failing with below error :
==> Vault server configuration:
AWS KMS Endpoint: kms.cn-north-1.amazonaws.com.cn
AWS KMS KeyID: XXXXXXXXXXXXXXXXX
AWS KMS Region: cn-north-1
Seal Type: awskms
Api Address: http://10.0.1.4:8200
Cgo: disabled
Cluster Address: https://10.0.1.4:8201
Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Log Level: info
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: consul (HA available)
Version: Vault v1.3.2
==> Vault server started! Log data will stream in below:
2020-04-21T10:50:29.411Z [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
2020-04-21T10:50:29.411Z [WARN] storage.consul: appending trailing forward slash to path
2020-04-21T10:50:29.525Z [INFO] core: stored unseal keys supported, attempting fetch
2020-04-21T10:50:29.582Z [INFO] core.cluster-listener: starting listener: listener_address=[::]:8201
2020-04-21T10:50:29.582Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
2020-04-21T10:50:29.582Z [INFO] core: entering standby mode
2020-04-21T10:50:29.586Z [INFO] core: vault is unsealed
2020-04-21T10:50:29.586Z [INFO] core: unsealed with stored keys: stored_keys_used=1
2020-04-21T10:50:29.595Z [INFO] core: acquired lock, enabling active operation
2020-04-21T10:50:29.657Z [INFO] core: post-unseal setup starting
2020-04-21T10:50:29.664Z [INFO] core: loaded wrapping token key
2020-04-21T10:50:29.664Z [INFO] core: successfully setup plugin catalog: plugin-directory=
2020-04-21T10:50:29.671Z [INFO] core: successfully mounted backend: type=system path=sys/
2020-04-21T10:50:29.671Z [INFO] core: successfully mounted backend: type=identity path=identity/
2020-04-21T10:50:29.671Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-04-21T10:50:29.697Z [INFO] core: successfully enabled credential backend: type=token path=token/
2020-04-21T10:50:29.697Z [INFO] core: restoring leases
2020-04-21T10:50:29.697Z [INFO] rollback: starting rollback manager
2020-04-21T10:50:29.701Z [INFO] expiration: lease restore complete
2020-04-21T10:50:29.707Z [INFO] identity: entities restored
2020-04-21T10:50:29.710Z [INFO] identity: groups restored
2020-04-21T10:50:29.770Z [INFO] core: post-unseal setup complete
after initiated failed error logs on vault:
2020-04-21T10:53:30.199Z [ERROR] core: error performing key upgrades: error="error reloading master key: error reloading master key: failed to read master key path: decryption failed: cipher: message authentication failed"
2020-04-21T10:53:30.199Z [INFO] core: marked as sealed
2020-04-21T10:53:30.205Z [INFO] core: stopping cluster listeners
2020-04-21T10:53:30.205Z [INFO] core.cluster-listener: forwarding rpc listeners stopped
2020-04-21T10:53:30.662Z [INFO] core.cluster-listener: rpc listeners successfully shut down
2020-04-21T10:53:30.663Z [INFO] core: cluster listeners successfully shut down
2020-04-21T10:53:30.663Z [INFO] core: vault is sealed
2020-04-21T10:53:32.840Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:35.840Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:38.842Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:41.846Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:44.842Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:47.839Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:50.836Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:53.843Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:56.832Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:59.830Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:54:02.838Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:54:05.838Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:54:08.842Z [INFO] core.autoseal: seal configuration
from vault-helm.
@jasonodonnell : any idea about it.
from vault-helm.
Related Issues (20)
- Latest vault helm chart (0.27.0) does not work with GCPCKMS
- Add a way to create Secrets in the values.yaml
- allow to pin IPs of vault services HOT 3
- json formatted server config converts to a freak vault-config k8s secret which is both hcl and json HOT 1
- Chart prevents synchronisation with ArgoCD when using custom sync label HOT 3
- Add support to external Vault running with tls HOT 2
- Configuring vault ha with raft and ingress HOT 1
- [Feature] Allow the vault sidecar injector to be configured to point to the vault-active service
- storage.raft.fsm: failed to store data: error="input/output error"
- Expose ports for the injector so that a podmonitor can scrape the metrics
- Got unseal error HOT 8
- Bug: Error Creating a raft cluster on OpenShift
- Would violate PodSecurity "restricted:latest" when installing Helm chart
- Please release a new version of helm chart with the current vault versions HOT 4
- Ability to have top level label on StatefullSet
- Cannot use HOSTNAME env var in VAULT_API_ADDR env var
- helm value server.logLevel does not set the log level but just logs all entries using this value
- Sidecar agent in CSI can't estabish a TLS connection with an external vault using a custom CA
- Deploying vault on OCI gives seal type Shamir not OCIKMS HOT 1
- Tests Assert that HA Should not be able to set the dataStorage StorageClass
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vault-helm.