Istio sidecar can only request new cert using istio-token

[lukassup]

Istio sidecar can only request new cert using istio-token

[RicHincapie]

This is creating issues with Istio's mesh-onboarded VMs. VMs in Istio work different that pods, because the sidecar agent uses K8s provided SA token only to onboard into the mesh, and from there on it relies exclusively on certs:

Virtual machines handle certificates differently than Kubernetes Pods, which use a Kubernetes-provided service account token to authenticate and renew mTLS certificates. Instead, existing mTLS credentials are used to authenticate with the certificate authority and renew certificates. Istio docs source.

While istiod supports both mTLS and JWT to renew certs here:

    // Notice that the order of authenticators matters, since at runtime
    // authenticators are activated sequentially and the first successful attempt
    // is used as the authentication result.
    authenticators := []security.Authenticator{
        &authenticate.ClientCertAuthenticator{},

istio-csr only supports JWT here:

    auther := kubeauth.NewKubeJWTAuthenticator(mesh.NewFixedWatcher(meshcnf), kubeClient, cluster.ID(opts.ClusterID), nil, jwt.PolicyThirdParty)

This creates the following issue with istio-onboarded VMs when trying to renew certs:

error    klog    "msg"="failed to authenticate request" "error"="failed to validate the JWT from cluster \"k8s\": the service account authentication returns an error: [invalid bearer token, service account token has expired, unknown]" "logger"="grpc-server"

I think this needs (1) a warning in docs before more people hit it; (2) support for mTLS for VMs to authenticate with istio-csr.