ESP on Compute Engine : JWT validation failed: Unable to fetch verification key

[eshwarprasadhu]

ESP on Compute Engine : JWT validation failed: Unable to fetch verification key

[qiwzhang]

You are reporting two issues:

1. failed to fetch JWKS: jwt public keys, the url is specified in x-google-jwks_url in openapi spec.
2. failed to connect to Google API, e.g. Google service control API

Unless x-google-jwks_uri specified a Google API, they are two different problems; accessing Google API may require different configuration than accessing other remote endpoints.

for 1), I think your egress need to allow to access that jwks_uri.
for 2), It seems that Google service control is still not allowed. report call is rejected.

You may have to check your egress settings to allow jwks_uri and Google API.

[eshwarprasadhu]

for 1) - I have this url https://www.googleapis.com/robot/v1/metadata/x509/<service_account> as x-google-jwks-uri in openapi spec.
for 2) - The Google Service Control API also has the domain ending with *.googleapis.com

The url 1) & 2) is translating into an IP address that is in the IP range 199.36.153.8/30

It even connects, when I do a telnet to www.googleapis.com and servicecontrol.googleapis.com apis from host VM and esp container.

Please refer the below screenshot, I have allowed egress to the same IP range in my firewall settings.

Not sure what is going wrong here. Please advise.

[qiwzhang]

Hmm, you may be using private VPC. or your GCE VM only has internal IP address. If so, you need to follow some special setup to access Google services.

I am not expert in this area. I just happened to read this doc. You may need to find some help from other Google supports.

[nareddyt]

I'm not convinced about your IP range 199.36.153.8/30. Are you forming this based on your own observations? Here is the IP address I get from my local machine.

nareddyt-macbookpro% ping servicecontrol.googleapis.com
PING servicecontrol.googleapis.com (172.217.0.42): 56 data bytes

As Wayne said, this seems to be out of scope for us, we are not experts in this.

[eshwarprasadhu]

@nareddyt It's as per gcp documentation. Please refer this link Private Google Access

[eshwarprasadhu]

@qiwzhang Yes, We are using private VPC and we don't use external address for our GCE VM.

[eshwarprasadhu]

@qiwzhang It looks like a bug in connectivity between google service apis managing the endpoints. Please direct to the support team who can help in this regard

[qiwzhang]

Could you post your evidence here so that I can create a external bug for you? Or you can create a bug for GCP youself?

[eshwarprasadhu]

I filed a bug for GCP Cloud APIs. Link for your reference https://issuetracker.google.com/issues/169349072

[eshwarprasadhu]

The problem got resolved by overriding ESP container's default DNS Resolver (8.8.8.8) with the correct DNS Resolver

[qiwzhang]

That make sense. For a GCE with private IP, it has to use GCP metadata server as DNS resolver.