[doc] confusion with `ca.pem` and Readiness probe failed on ingress and egress gateways

[cert-manager]

[doc] confusion with `ca.pem` and Readiness probe failed on ingress and egress gateways

[JoshVanL]

Hi @nicop311, the ca.pem file referenced should contain the root CAs that you would like your istio cluster to trust (including and likely only the CA of your issuer). If you are using the Issuer of type ca, then this would be the CA within the Secret as referenced in the Issuer config.

Propagating a different CA to that used by the Issuer will make istio workloads not trust the CA which signed their certificates.

[nicop311]

Thank you @JoshVanL for your answer, your explanation is clear. But I am still confused by the documentation.

See my quetion below: "What is the relation between CA_FROM_cert-manager-CA_Issuer_istio-system from the istio-system namespace and the file ca.pem from the cert-manager namespace?

If they are the same, I don't understand the steps and the procedure explained in the documentation.

---

If I try to follow istio-csr documentation, here is the details that could
be added to the documentation.

Step 0. Have a K8s cluster and istioctl

Step 1. Install cert-manager with OLM

1. Go to: https://operatorhub.io/operator/cert-manager.
2. Click Install.
3. Install OLM (curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.19.1/install.sh | bash -s v0.19.1)
4. Install cert-manager (kubectl create -f https://operatorhub.io/install/cert-manager.yaml)

Step 2. Create CA Issuer or ClusterIssuer

In the istio-csr documentation Issuer or ClusterIssuer,
you are advise to create a Cert-manager CA Issuer
in the istio-system namespace.

1. I use istioctl operator init which creates an istio-system namespace if it does not exist.

istioctl operator init

Operator controller is already installed in istio-operator namespace.
Upgrading operator controller in namespace: istio-operator using image: docker.io/istio/operator:1.11.3
Operator controller will watch namespaces: istio-system
✔ Istio operator installed
✔ Installation complete

Now the istio-system ns exist. We can also create it by hand it does not matter.

2. I create a cert-manager CA Issuer thanks to the suggested example:

kubectl apply -n istio-system -f https://raw.githubusercontent.com/cert-manager/istio-csr/v0.3.0/docs/example-issuer.yaml

issuer.cert-manager.io/selfsigned unchanged
certificate.cert-manager.io/istio-ca configured
issuer.cert-manager.io/istio-ca unchanged

This creates some secrets in the istio-system namespace:

kubectl get secrets -n istio-system

NAME                  TYPE                                  DATA   AGE
default-token-pwsx4   kubernetes.io/service-account-token   3      5d
istio-ca              kubernetes.io/tls                     3      5d
istiod-tls            kubernetes.io/tls                     3      5d

Now this is the result (I replace keys and cert by names/const to save space)

thedetective@k8s-kind-monitoring-target-114:~$ kubectl get secrets -n istio-system istiod-tls -o yaml > istiod-tls.yaml

# istiod-tls.yaml
apiVersion: v1
data:
  ca.crt:
  CA_FROM_cert-manager-CA_Issuer_istio-system
  tls.crt:
  istiod-tls_tls.crt
  tls.key:
  istiod-tls_tls.key
kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: istiod.istio-system.svc
    cert-manager.io/certificate-name: istiod
    cert-manager.io/common-name: istiod.istio-system.svc
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-group: cert-manager.io
    cert-manager.io/issuer-kind: Issuer
    cert-manager.io/issuer-name: istio-ca
    cert-manager.io/uri-sans: spiffe://cluster.local/ns/istio-system/sa/istiod-service-account
  creationTimestamp: "2021-10-22T08:56:48Z"
  name: istiod-tls
  namespace: istio-system
  resourceVersion: "2387761"
  uid: 58ee187a-f233-4f39-b9e5-00ffbc28ea58
type: kubernetes.io/tls

thedetective@k8s-kind-monitoring-target-114:~$ kubectl get secrets -n istio-system istio-ca -o yaml > istio-ca.yaml

# istio-ca.yaml
apiVersion: v1
data:
  ca.crt:
  CA_FROM_cert-manager-CA_Issuer_istio-system
  tls.crt:
  CA_FROM_cert-manager-CA_Issuer_istio-system
  tls.key:
  istio-ca-tls.key
kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: ""
    cert-manager.io/certificate-name: istio-ca
    cert-manager.io/common-name: istio-ca
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-group: cert-manager.io
    cert-manager.io/issuer-kind: Issuer
    cert-manager.io/issuer-name: selfsigned
    cert-manager.io/uri-sans: ""
  creationTimestamp: "2021-10-22T08:54:53Z"
  name: istio-ca
  namespace: istio-system
  resourceVersion: "114380"
  uid: efaceac5-d398-4c84-bfaf-d13635d54d8e
type: kubernetes.io/tls

Step 3: Load root CAs from file ca.pem (Preferred)

I create a ca.pem file since I need one and there are no explanation from where this CA should come.

openssl req -x509 -sha512 -nodes -extensions v3_ca -newkey rsa:4096 -keyout ca-cert-and-key.pem -days 7320 -out ca-cert-and-key.pem

I create the ca.pem from ca-cert-and-key.pem.

Then I follow the documentation:

$ helm repo add jetstack https://charts.jetstack.io
$ helm repo update
$ kubectl create namespace istio-system
$ kubectl create secret generic istio-root-ca --from-file=ca.pem=ca.pem -n cert-manager
$ helm install -n cert-manager cert-manager-istio-csr jetstack/cert-manager-istio-csr \
  --set "app.tls.rootCAFile=/var/run/secrets/istio-csr/ca.pem" \
  --set "volumeMounts[0].name=root-ca" \
  --set "volumeMounts[0].mountPath=/var/run/secrets/istio-csr" \
  --set "volumes[0].name=root-ca" \
  --set "volumes[0].secret.secretName=istio-root-ca"

Step 4: Installing Istio

I follow the documentation Installing Istio.

I put spec.meshConfig.trustDomain: cluster.local.

Question

I assume the CA CA_FROM_cert-manager-CA_Issuer_istio-system is generated by the Cert-manager CA Issuer or ClusterIssuer.
This CA_FROM_cert-manager-CA_Issuer_istio-system is used in both istio-ca and istiod-tls secrets int the istio-system namespace.

QUESTION: What is the relation between CA_FROM_cert-manager-CA_Issuer_istio-system and the file ca.pem from kubectl create secret generic istio-root-ca --from-file=ca.pem=ca.pem -n cert-manager (notice this is cert-manager namespace, not istio-system) from section Load root CAs from file ca.pem (Preferred) ?

If CA_FROM_cert-manager-CA_Issuer_istio-system and --from-file=ca.pem=ca.pem are supposed to be the same, then I don't understand what is happening in the doc procedure and I don't understand why one secret is in istio-system namespace and the other is in cert-manager.

It seems really weird to me to create the ca.pem file out of the CA_FROM_cert-manager-CA_Issuer_istio-system auto generated by cert-manager.

Maybe cert-manager/istio-csr makes sense as soon as we do a certificate certificate rotation? Because for the moment, I do not see the improvement compared to the istio plug in CA certificates and the cacerts secret, if we have to create a secret by hand.

Of course if I use CA_FROM_cert-manager-CA_Issuer_istio-system to create ca.pem, my error x509: certificate signed by unknown authority disappear.

I think the README could mention this.

---

Disclaimer: I know that in real life situation, cert-manager would be pluged into a PKI or tool like Hashicorp Vault.