I followed these steps up to the point of having the vault issuer:
https://learn.hashicorp.com/tutorials/vault/kubernetes-cert-manager?in=vault/kubernetes
then I provided that issuer to istio-csr, by setting values, certificate.name to vault-issuer, so instead of self-signed issuer I have vault.
So I did not do this:
https://github.com/cert-manager/istio-csr/blob/master/hack/demo/cert-manager-bootstrap-resources.yaml
instead I have the issuer from the vault documentation, and the cert
https://github.com/cert-manager/istio-csr/blob/master/deploy/charts/istio-csr/templates/certificate.yaml
is made from that (see below it does show up) and ready True status.
istio-csr comes up, and everything looks like its in place
certificate seems to have gone through and the vault issuer looks ok:
kubectl get certificate -A
NAMESPACE NAME READY SECRET AGE
istio-system istiod True istiod-tls 9m37s
kubectl get certificaterequest -A
NAMESPACE NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
istio-system istiod-1609388919 True vault-issuer system:serviceaccount:cert-manager:cert-manager 9m48s
kubectl get issuer -A
NAMESPACE NAME READY AGE
istio-system vault-issuer True 12m
then I submit the istioOperator, based on the docs here and it does accept the manifest with the istio-csr modifications, istiod seems to come up but ingresgateway does not:
kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istio-ingressgateway-76cd4879b4-4pnp9 0/1 Running 0 29m
istiod-6756549fcd-xpdnm 1/1 Running 0 29m
from istio ingressgateway:
2021-04-30T21:17:48.050660Z info PilotSAN []string{"istiod.istio-system.svc"}
2021-04-30T21:17:48.050674Z info MixerSAN []string{"spiffe://cluster.local/ns/istio-system/sa/istio-mixer-service-account"}
2021-04-30T21:17:48.050708Z info sa.serverOptions.CAEndpoint == cert-manager-istio-csr.cert-manager.svc:443
2021-04-30T21:17:48.050715Z info Using user-configured CA cert-manager-istio-csr.cert-manager.svc:443
2021-04-30T21:17:48.050718Z info istiod uses self-issued certificate
2021-04-30T21:17:48.050759Z info the CA cert of istiod is: -----BEGIN CERTIFICATE-----
<redacted>
-----END CERTIFICATE-----
...
...
warning envoy config gRPC config for type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret rejected: Failed to load certificate chain from <inline>
2021-04-30T21:17:49.002741Z warning envoy main there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
2021-04-30T21:17:49.003369Z error sds resource:default received error: code:13 message:"Failed to load certificate chain from <inline>". Will not respond until next secret update
...
...
then this just repeats forever:
2021-04-30T21:37:40.008063Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2021-04-30T21:37:40.816847Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-04-30T21:37:42.815700Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-04-30T21:37:44.815994Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-04-30T21:37:46.817274Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-04-30T21:37:48.815762Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-04-30T21:37:50.818775Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-04-30T21:37:52.815827Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-04-30T21:37:54.635857Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2021-04-30T21:37:54.815743Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-04-30T21:37:56.816362Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-04-30T21:37:58.820236Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
from istiod:
http: TLS handshake error from 10.66.15.64:33293: remote error: tls: bad certificate
secret shows up as well:
kubectl get secrets -n istio-system
NAME TYPE DATA AGE
...
...
istiod-tls kubernetes.io/tls 3 33m
...
When Ive researched others who are getting that "envoy proxy is NOT ready spam", it had something to do with the way the cert was presented/formatted, but all I did was follow the instructions and have it generated by vault, which means either it didnt like how vault formatted it, or istiod didnt like how istio-csr formatted it?
I'm not sure what else it could be, are there certain minimal versions I should be aware of? the only thing I see is istio needs to be 1.7+, which mine is 1.7.2, is there a minimal version for cert manager?
or if theres any steps I may have missed please let me know but Ive tried to look at these docs back and forth and I can't figure out what it is.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent