HashCorp Vault Helm with consul should have example for Aws EKS

[hashicorp]

HashCorp Vault Helm with consul should have example for Aws EKS

[msidd111]

@jasonodonnell Which consul helm chart I should install for HA option? Can you guide me in terms of installing consul and vault helm for HA and any specific configuration changes that I may need to do for Aws EKS.

[jasonodonnell]

Hey @msidd111, sorry for the delay but HashiConf is this week so we've been busy! :)

Which consul helm chart I should install for HA option?

The latest version of consul-helm is fine.

Unfortunately Hashicorp Helm chart does not have an example for EKS.

I agree we should have an example of EKS with auto-unseal in the documentation. I will add this soon.

In the meantime here's how it can be done:

EKS and Vault Helm Unseal Example

First, create a secret with your KMS access key/secret:

Note: I used environment variables to populate the values from my workstation

kubectl create secret generic vault-aws \ 
  --from-literal=AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID?}" \
  --from-literal=AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY?}"

Next, clone the repo:

mkdir ~/vault-eks && cd ~/vault-eks
git clone [email protected]:hashicorp/vault-helm.git

Next, create a custom values-eks.yaml file outside of the vault-helm directory so we can just inject our custom values:

Note: You need to fill out the seal parameters below.

cat >~/vault-eks/values-eks.yaml <<EOL
server:
  extraSecretEnvironmentVars:
  - envName: AWS_ACCESS_KEY_ID
    secretName: vault-aws
    secretKey: AWS_ACCESS_KEY_ID
  - envName: AWS_SECRET_ACCESS_KEY
    secretName: vault-aws
    secretKey: AWS_SECRET_ACCESS_KEY

  ha:
    enabled: true
    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      seal "awskms" {
        region     = "KMS_REGION_HERE"
        kms_key_id = "KMS_KEY_ID_HERE"
      }

      storage "consul" {
        path = "vault"
        address = "HOST_IP:8500"
      }
EOL

Assuming consul-helm is already deployed and ready, we can now deploy Vault:

helm install --name=vault -f values-eks.yaml ~/vault-eks/vault-helm

Once deployed you can initialize vault-0 and it will auto unseal using AWS KMS:

kubectl exec -ti vault-0 -- vault operator init

[jasonodonnell]

@msidd111 At this time changing the type of service is not supported. You can make your own service though:

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  name: vault-lb
  labels:
    app: vault-lb
spec:
  ports:
  - port: 8200
    targetPort: 8200
  selector:
    app.kubernetes.io/name: vault
    component: server
  type: LoadBalancer
EOF

[msidd111]

Thank you for your quick response. Much Obliged. Will let you if I run into any issues or questions I may have.

[msidd111]

@jasonodonnell I was able to configure vault on consul fine but consul seems to fail resilience test when I restarted all of kubernetes nodes. One of the consul node will stuck with pending state and other will not transition to ready state. Here is error message from consul pod that was stuck in pending state.

Events:
Type Reason Age From Message

---

Warning FailedScheduling 43m (x2 over 43m) default-scheduler 0/2 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules, 1 node(s) had volume node affinity conflict.
Warning FailedScheduling 42m (x6 over 42m) default-scheduler 0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules, 1 node(s) had taints that the pod didn't tolerate, 1 node(s) had volume node affinity conflict.
Warning Fa

[jasonodonnell]

@msidd111 This is not a Vault Helm issue but a Consul Helm issue. That being said, Vault Helm and Consul Helm use affinity rules to spread out pods onto multiple Kubernetes worker nodes. You are failing the affinity check (it appears you don't have enough worker nodes). You would need at least 3 worker nodes and enough volumes to support each pod.

You could remove consul-helms affinity rules by removing the rule but that's not recommended as it weakens the HA deployment strategy. It's recommended to satisfy the affinity rule if possible.

[msidd111]

Thanks. This is issue with consul affinity rules so I removed podAntiAffinity and I do get three pods distributed on 2 nodes instead three which podAntiAffinity should enforce so this addresses the issue of consul nodes going to pending state after restarting nodes with caveat that I will have weak HA.

[msidd111]

@jasonodonnell In next test, I deleted vault-0 which was brought back by cluster but during this time around 10 seconds I was getting error: unable to upgrade connection: container not found ("vault"). Is this expected behavior or one of other two nodes vault-1 and vault-2 should start serving traffic without downtime once vault-0 is down

[msidd111]

BTW, I was able to access vault when consul-0 pod was deleted.

[jasonodonnell]

@msidd111 Failover is not instant and could take a few seconds before the lock is released. Where are you seeing the error: unable to upgrade connection: container not found ("vault") error?

[jasonodonnell]

https://github.com/hashicorp/vault/pull/7759/files EKS documentation has been added to the official doc website. Closing.

[ankitkl]

trying this setup on EKS

Here is my consul helm config:

syncCatalog:
  consulNamespaces:
    consulDestinationNamespace: "opx-vault"

client:
  enabled: true

Storage: 1Gi
StorageClass: standard-encrypted

server:
  replicas: 2 
  bootstrapExpect: 1
  disruptionBudget:
    maxUnavailable: 0

Using consul helm chart version: 0.18.0

helm install consul -f consul-eks/helm-consul-values.yaml vault-consul/consul-helm/

Vault config yaml:

server:
  extraSecretEnvironmentVars:
  - envName: AWS_ACCESS_KEY_ID
    secretName: vault-aws
    secretKey: AWS_ACCESS_KEY_ID
  - envName: AWS_SECRET_ACCESS_KEY
    secretName: vault-aws
    secretKey: AWS_SECRET_ACCESS_KEY

  ha:
    enabled: true
    replicas: 2
    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      seal "awskms" {
        region     = "cn-north-1"
        kms_key_id = "XXXXXXXXXXX"
        endpoint = "kms.cn-north-1.amazonaws.com.cn"
      }

      storage "consul" {
        path = "vault"
        address = "HOST_IP:8500"
      }

Using vault 0.4.0 chart

helm install vault -f vault-eks/values-eks.yaml vault-eks/vault-helm/

At time of launch vault unseal but after that it start failing with below error :

==> Vault server configuration:
    AWS KMS Endpoint: kms.cn-north-1.amazonaws.com.cn
      AWS KMS KeyID: XXXXXXXXXXXXXXXXX
     AWS KMS Region: cn-north-1
        Seal Type: awskms
       Api Address: http://10.0.1.4:8200
           Cgo: disabled
     Cluster Address: https://10.0.1.4:8201
       Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
        Log Level: info
          Mlock: supported: true, enabled: false
      Recovery Mode: false
         Storage: consul (HA available)
         Version: Vault v1.3.2
==> Vault server started! Log data will stream in below:
2020-04-21T10:50:29.411Z [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
2020-04-21T10:50:29.411Z [WARN] storage.consul: appending trailing forward slash to path
2020-04-21T10:50:29.525Z [INFO] core: stored unseal keys supported, attempting fetch
2020-04-21T10:50:29.582Z [INFO] core.cluster-listener: starting listener: listener_address=[::]:8201
2020-04-21T10:50:29.582Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
2020-04-21T10:50:29.582Z [INFO] core: entering standby mode
2020-04-21T10:50:29.586Z [INFO] core: vault is unsealed
2020-04-21T10:50:29.586Z [INFO] core: unsealed with stored keys: stored_keys_used=1
2020-04-21T10:50:29.595Z [INFO] core: acquired lock, enabling active operation
2020-04-21T10:50:29.657Z [INFO] core: post-unseal setup starting
2020-04-21T10:50:29.664Z [INFO] core: loaded wrapping token key
2020-04-21T10:50:29.664Z [INFO] core: successfully setup plugin catalog: plugin-directory=
2020-04-21T10:50:29.671Z [INFO] core: successfully mounted backend: type=system path=sys/
2020-04-21T10:50:29.671Z [INFO] core: successfully mounted backend: type=identity path=identity/
2020-04-21T10:50:29.671Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-04-21T10:50:29.697Z [INFO] core: successfully enabled credential backend: type=token path=token/
2020-04-21T10:50:29.697Z [INFO] core: restoring leases
2020-04-21T10:50:29.697Z [INFO] rollback: starting rollback manager
2020-04-21T10:50:29.701Z [INFO] expiration: lease restore complete
2020-04-21T10:50:29.707Z [INFO] identity: entities restored
2020-04-21T10:50:29.710Z [INFO] identity: groups restored
2020-04-21T10:50:29.770Z [INFO] core: post-unseal setup complete

after initiated failed error logs on vault:

2020-04-21T10:53:30.199Z [ERROR] core: error performing key upgrades: error="error reloading master key: error reloading master key: failed to read master key path: decryption failed: cipher: message authentication failed"
2020-04-21T10:53:30.199Z [INFO] core: marked as sealed
2020-04-21T10:53:30.205Z [INFO] core: stopping cluster listeners
2020-04-21T10:53:30.205Z [INFO] core.cluster-listener: forwarding rpc listeners stopped
2020-04-21T10:53:30.662Z [INFO] core.cluster-listener: rpc listeners successfully shut down
2020-04-21T10:53:30.663Z [INFO] core: cluster listeners successfully shut down
2020-04-21T10:53:30.663Z [INFO] core: vault is sealed
2020-04-21T10:53:32.840Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:35.840Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:38.842Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:41.846Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:44.842Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:47.839Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:50.836Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:53.843Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:56.832Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:53:59.830Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:54:02.838Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:54:05.838Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2020-04-21T10:54:08.842Z [INFO] core.autoseal: seal configuration 

[ankitkl]

@jasonodonnell : any idea about it.