not a general issue. Couldn't find any docs.

[hashicorp]

not a general issue. Couldn't find any docs.

[tomhjp]

Thanks for the detailed configs. A couple of issues I've spotted first:

• vaultCACertPath: "/vault/tls/vault.crt" should be vaultCACertPath: "/vault/tls/vault.ca"; this field is setting the trusted CA, which should be the Kubernetes CA if you have created your certificates as per the linked docs.
• The Vault CSI provider doesn't need any TLS certificates, it only needs the CA that Vault's certificate was signed by so that it knows to trust it. The Vault CSI provider communicates with the secrets store CSI driver over a unix socket and doesn't use TLS for that, so it's only a TLS client, never a TLS server.

Otherwise it looks like you're mostly on the right track. In particular, you've recognised that the SecretProviderClass TLS parameters are referring to files on the CSI provider pod's file system, which is a bit of a gotcha.

Currently, the best worked example is in the e2e tests in this repo, but I'm definitely planning to add a documentation example that addresses TLS specifically.

• The TLS CA + private key + certificate bundle is created by helm using this. If you're not doing mutual TLS, the vault-client-tls secret would only need the CA entry.
• TLS secret is mounted to Vault CSI provider here.
• Those two helm charts are installed here.
• TLS is configured in the SPC here. Again, if not doing mutual TLS, you'll only need the CA entry on the CSI provider side.

[tomhjp]

Well done tracking down the issue. For anyone else stumbling across similar, when you get a * claim "iss" is invalid error, it most likely means the kubernetes auth mount configuration needs updating, in particular issuer.

Default service account tokens are created by the Kubernetes Service Account admission controller, and typically the JWTs that creates have kubernetes/serviceaccount as an issuer, which is also the default value for issuer in the k8s auth method on Vault, so everything normally "just works" when using pre-existing tokens. However, the service account tokens created and used by the Vault CSI provider for auth use the value of kube-apiserver's --service-account-issuer flag as the issuer, and the k8s auth mount needs to have a matching value for issuer validation. I noted a few common values in the tests, but I think we probably should create a more prominent note about this in the docs as well.

EDIT: The simpler alternative of course is to set disable_iss_validation=true, but that's not recommended.

[hanem100k]

thanks, @eyenx @tomhjp

Hi, you need to set the issuer correctly when creating the vault-auth binding (auth/kubernetes) as done here:

Double checked this, no help, unfortunately, however, I think if this would be the issue, the API would just return a 403 or something along those lines, I mean the request would not be short-circuited right?

• vaultCACertPath: "/vault/tls/vault.crt" should be vaultCACertPath: "/vault/tls/vault.ca"; this field is setting the trusted CA, which should be the Kubernetes CA if you have created your certificates as per the linked docs.

The manifest i copied above was actually outdated, I had the right reference to the CA, however, I had the key pair in addition. Fixed all that and this is the current state.

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  namespace: heimdall-dev
  name: heimdall-config
spec:
  provider: vault
  secretObjects:
  - secretName: heimdall-dev-secrets
    type: Opaque
    data:
    - objectName: DATABASE_URL # References dbUsername below
      key: DATABASE_URL          # Key within k8s secret for this value
    - objectName: test
      key: test
  parameters:
    roleName: "heimdall-dev"
    vaultAddress: "https://vault.vault:8200"
    vaultCACertPath: /vault/tls/vault.ca

    objects: |
      - objectName: "DATABASE_URL"
        secretPath: "heimdall-dev/config/env"
        secretKey: "DATABASE_URL"
      - objectName: "test"
        secretPath: "heimdall-dev/config/env"
        secretKey: "test"

Via this setting I'm still getting 500 and the same error.

Some other details. For the csi provider certificate I have the following CSR

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = vault-csi
DNS.2 = vault-csi-provider
DNS.3 = vault.csi-*

IP.1 = 127.0.0.1

Im thinking it might be a dns issue at this point.
Csr for vault is almost the same except the DNS values ofc.

Is there a way I could say more verbose logs from the vault? Or what would be a next step solving this issue? (now vault just logs tls handshake error)

Also here is the error from the csi provider

Thanks a lot guys for the support!

Edit:

Found the issue.
One of the policies issuer had a typo in the cluster name. It was really annoying to track it down, because it could be so many things with all the certs and policies. (maybe we have separate errors for all these possibilities)

Still, getting more and more familiar with vault and its ecosystem, thank you guys for all of your hard work.
Closing the issue.