Istio stable-mesh test is broken

[nak3]

Istio stable-mesh test is broken

[tcnghia]

cc @mattmoor

[nak3]

I am suspecting this Istio's issue istio/istio#23463. The DestinationRule and some behavior were different whether external service has the port or not.

[tcnghia]

@nak3 I think this is fixed in 1.6?

[tcnghia]

cc @ZhiminXiang @JRBANCEL @Cynocracy

[JRBANCEL]

It is still broken.
The webhook Container of webhook Pod never gets ready (example), because the probes defined here:

2020-10-02T01:16:54.849646Z	error	Request to probe app failed: Get "https://localhost:8443/": remote error: tls: internal error, original URL path = /app-health/webhook/readyz app URL path = /
2020-10-02T01:16:55.347043Z	error	Request to probe app failed: Get "https://localhost:8443/": remote error: tls: internal error, original URL path = /app-health/webhook/livez app URL path = /

It might be an Istio issue, see this similar issue.

[JRBANCEL]

What seems to be happening is that Kubelet does indeed call the modified probes (on Istio Proxy) and then the proxy calls the Webhook probes, but this fails with the not useful remote error: tls: internal error.
I suspect that it fails because this is endpoint uses a self-signed certificate.

[JRBANCEL]

Istio skips TLS validation: https://github.com/istio/istio/blob/master/pilot/cmd/pilot-agent/status/server.go#L421
So it is failing for another reason. In the webhook logs I see 2020/10/02 01:20:23 http: TLS handshake error from 127.0.0.1:55830: server key missing.
Someone complained about this recently: knative/eventing#4165

[nak3]

Closing. There are still some failures but they are tracked by #340.