using Vault Issuer

[cert-manager]

using Vault Issuer

[JoshVanL]

Hey @adnankobir, apologies this isn't working right quite yet.

Would you be able to verify a couple things?

• The propagated root CA is the same CA you are expecting (the root CA). You can set this manually with certificate.rootCA
• Istiod has picked up and mounted its cert from the secret, and it also has its CA Server disabled. There should be some indication in the istiod logs of what its CA is.
• Checking the envoy logs that they too are using the same CA that you are expecting, and they are requesting certs from istio-csr
• Check the logs in istio-csr, to make sure it is receiving requests from envoys in the cluster

[adnankobir]

thanks for the prompt response @JoshVanL

I've verified the following (sensitive and unnecessary info redacted):

• The propagated root CA is the same CA you are expecting

The propagated root CA (intermediate CA in Vault) looks correct:
(I'm uncertain if we require the root-ca from ACM-PCA here or both/full chain)

kubectl get cm istio-ca-root-cert -o jsonpath="{.data['root-cert\.pem']}" | step certificate inspect -

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 68566204885263584472517345669233231743 (0x33955ee740f4bafe9f963b7b151bc77f)
    Signature Algorithm: SHA512-RSA
        Issuer: O=redacted Inc.,ST=New York,CN=redacted.works,L=New York
        Validity
            Not Before: Feb 1 15:55:15 2021 UTC
            Not After : Feb 1 16:55:15 2026 UTC
        Subject: ST=New York,L=New York,O=redacted.,CN=v7-test-use1.redacted.works
        Subject Public Key Info:
            Public Key Algorithm: RSA
                Public-Key: (2048 bit)
                Modulus:
                    ...
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:v7-test-use1.redacted.works, DNS:vault-v7-test-use1, DNS:vault-v7-test-use1.vault, DNS:vault-v7-test-use1.vault.svc
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            ...
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
            ...

• Istiod has picked up and mounted its cert from the secret, and it also has its CA Server disabled

Istiod has in-fact picked up the correct cert from istiod-tls secret and we can see the cert and its chain to Intermediate CA in Vault:

2021-02-02T13:37:46.234341Z     info    Istiod certificates are reloaded
2021-02-02T13:37:46.234785Z     info    x509 cert [0] - Issuer: "CN=v7-test-use1.redacted.works,O=redacted,L=New York,ST=New York", Subject: "", SN: 3af9313443a2a2de75c177625d5918287ef72716, NotBefore: "2021-02-02T13:36:38Z", NotAfter: "2021-02-03T13:37:08Z"

2021-02-02T13:37:46.234929Z     info    x509 cert [1] - Issuer: "CN=redacted.works,O=redacted,L=New York,ST=New York", Subject: "CN=v7-test-use1.redacted.works,O=redacted,L=New York,ST=New York", SN: 33955ee740f4bafe9f963b7b151bc77f, NotBefore: "2021-02-01T15:55:15Z", NotAfter: "2026-02-01T16:55:15Z"

The CA server is disabled in istiod:

        - name: ENABLE_CA_SERVER
          value: "false"

Whats interesting of note here is that istio-operator overlay for ca-root-cert defines secretName: istiod-tls here is ignored/is invalid and we are left with:

     - configMap:
          defaultMode: 420
          name: istio-ca-root-cert
       name: ca-root-cert

• Checking the envoy logs that they too are using the same CA that you are expecting, and they are requesting certs from istio-csr

The envoy proxy on the sleep container does appear to be requesting certs from istio-csr and the CA serial matches the intermediate CA in vault:

2021-02-02T18:44:52.902086Z	info	citadelclient	Citadel client using custom root: cert-manager-istio-csr.cert-manager.svc:443 

The CA serial matches that of root-cert.pem in cm istio-ca-root-cert:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 68566204885263584472517345669233231743 (0x33955ee740f4bafe9f963b7b151bc77f)
    Signature Algorithm: SHA512-RSA
        Issuer: O=redacted,ST=New York,CN=redacted.works,L=New York
        Validity
            Not Before: Feb 1 15:55:15 2021 UTC
            Not After : Feb 1 16:55:15 2026 UTC
        Subject: ST=New York,L=New York,O=redacted.,CN=v7-test-use1.redacted.works
        ...
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:v7-test-use1.redacted.works, DNS:vault-v7-test-use1, DNS:vault-v7-test-use1.vault, DNS:vault-v7-test-use1.vault.svc

            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign

• Check the logs in istio-csr, to make sure it is receiving requests from envoys in the cluster

There doesn't seem to be any logs indicating that it is serving requests from envoys in the cluster (i've tested this by deleting the sleep pod and tailing the logs), I've got the following logs from the container:

I0202 01:37:12.356930       1 controller.go:206] ca-root-controller/controller/namespace "msg"="Starting workers" "configmap-name"="istio-ca-root-cert" "reconciler group"="" "reconciler kind"="Namespace" "worker count"=1
I0202 17:37:12.182980       1 tls.go:116] serving_certificate "msg"="renewing serving certificate"
I0202 17:37:12.597222       1 tls.go:230] serving_certificate "msg"="created serving CertificateRequest" "name"="cert-manager-istio-csr-22hq7" "namespace"="istio-system"
I0202 17:37:13.113882       1 tls.go:238] serving_certificate "msg"="serving CertificateRequest ready" "name"="cert-manager-istio-csr-22hq7" "namespace"="istio-system"
I0202 17:37:13.115853       1 tls.go:148] serving_certificate "msg"="fetched new serving certificate"
I0202 17:37:13.115890       1 tls.go:102] serving_certificate "msg"="renewing serving certificate"  "renewal-time"=57600000000000
2021-02-02T17:37:13.114573Z	info	spiffe	Added 1 certs to trust domain cluster.local in peer cert verifier
I0202 17:37:13.168318       1 tls.go:248] serving_certificate "msg"="deleted serving CertificateRequest" "name"="cert-manager-istio-csr-22hq7" "namespace"="istio-system"

Finally to add a bit more context I performed an openssl s_client connect to httpbin from sleep:

❯ kubectl exec -it sleep-f8cbf5b76-gtn4s -c istio-proxy -- openssl s_client -showcerts -connect httpbin:8000
CONNECTED(00000005)
depth=1 ST = New York, L = New York, O = redacted, CN = v7-test-use1.redacted.works
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
 0 s:
   i:ST = New York, L = New York, O = redacted, CN = v7-test-use1.redacted.works
-----BEGIN CERTIFICATE-----
redacted
-----END CERTIFICATE-----
 1 s:ST = New York, L = New York, O = redacted, CN = v7-test-use1.redacted.works
   i:O = redacted, ST = New York, CN = redacted.works, L = New York
-----BEGIN CERTIFICATE-----
redacted
-----END CERTIFICATE-----
 2 s:ST = New York, L = New York, O = redacted., CN = v7-test-use1.redacted.works
   i:O = redacted, ST = New York, CN = redacted.works, L = New York
-----BEGIN CERTIFICATE-----
redacted
-----END CERTIFICATE-----
---
Server certificate
subject=

issuer=ST = New York, L = New York, O = redacted, CN = v7-test-use1.redacted.works

---
Acceptable client certificate CA names
ST = New York, L = New York, O = redacted, CN = v7-test-use1.redacted.works
Requested Signature Algorithms: ECDSA+SHA256:RSA-PSS+SHA256:RSA+SHA256:ECDSA+SHA384:RSA-PSS+SHA384:RSA+SHA384:RSA-PSS+SHA512:RSA+SHA512:RSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:RSA-PSS+SHA256:RSA+SHA256:ECDSA+SHA384:RSA-PSS+SHA384:RSA+SHA384:RSA-PSS+SHA512:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4721 bytes and written 419 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
---
139814336737728:error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required:../ssl/record/rec_layer_s3.c:1528:SSL alert number 116
command terminated with exit code 1

Note here that I'm unsure of: cert 1 and 2 are identical

My hunch is that there is an issue with the cert chain or the KeyUsage fields

Thanks again!

[JoshVanL]

Thanks for all the detailed info @adnankobir !

I agree that it looks suspicious that it is duplicating the certificate chain here, and could be causing the issue.

(I'm uncertain if we require the root-ca from ACM-PCA here or both/full chain)

Ideally it should be the root certificate that should be populated here, and the intermediate is kept in the chain when making connections. I wonder if overriding the propagating certificate to the root CA would solve the issue- we likely won't have this duplication in the chain I believe.

If you can increase the log level of istio-csr it should give you some more info about what it is signing (agent.logLevel --v=3). You can also set the flag so that CertificateRequests are preserved, which should give us a bit more info on what is actually getting signed and passed to envoys (certificate.preserveCertificateRequests: true --preserve-certificate-requests=true). The CertificateRequest resources will also have a label that matches which identity the CR is for.

[adnankobir]

I believe this issue is related to cert-manager/cert-manager#2166. I've gone ahead and added both the root and intermediate to istio-csr via certificate.rootCA but upon inspection of the CertificateRequest for the workloads, I can see in the status field that there is only a single CA, at this point I'm not sure if this is feasible until the vault-issuer can successfully return the full CA chain

Update: upon switching cert-manager to v1.2.0-alpha.2 which includes cert-manager/cert-manager#3433, the propagated root CA is now populated correctly and we have the intermediate cert appended to the certificate.

Thanks @JoshVanL for pointing me in the right direction

[adnankobir]

now just patiently waiting for a stable cert-manager v1.2.0