SSL certificate error when using gsutil on machine in a private subnet and wildcard CNAME

[googleapis]

SSL certificate error when using gsutil on machine in a private subnet and wildcard CNAME

[lizardslair]

I work with @swvjeff and just wanted to share our findings in the unique issue in case anyone else happens to have this same problem in the future.

I found that there are different distros of gsutil - one seems to be "standalone" and one shipped with the "google-cloud-sdk".

When the cloud version is invoked, it appears that it attempts to lookup "metadata.google.internal" to determine if the machine that is running the command is a part of Google's cloud infrastructure and on the network (hense the .intenral TLD would resolve to Google's internal IPs).

As a feature of DNS resolution, you can set a set of "search" domains that will be suffixed to any entry in the case the entry doesn't exist alone. For example, if your device is configured to search "testing.com".. and you try to lookup simply "db"... it will fall-back to looking up "db.testing.com" if the initial lookup failed. Similar if you try to look up "server1.db" ... it will finally try "server1.db.testing.com".

So in our case... we set up a wildcard cname for *.mydomain.com. Similarly, our "db" server has "mydomain.com" as a search domain by default.

Therefore, when the SDK version of gsutil was invoked, it attempted to resolve "metadata.google.internal" to see if it was on Google's internal network. Then the fall-back domain search was used in the lookup, which essentially became "metadata.google.internal.mydomain.com", and because of the wildcard "*.mydomain.com" it resolved to our primary website IP.

So, now gsutil tried connecting to the IP, which was our webserver that had an SSL certificate for "*.mydomain.com", which does not match the expected target of "metadata.google.internal" and threw a SSL connection exception.

So, we came up with two solutions:

1. Use the standalone version of gsutil that does not query for the ".internal" domain name to resolve.

2. Update our machines' /etc/resolv.conf line for "search" domains to not include our primary domain name that now will answer to anything because of the wildcard entry (such as "a.b.c.d" would translate to "a.b.c.d.mydomain.com" which would resolve our website target IP)

[frankyn]

This should be filed against https://github.com/googlecloudplatform/gsutil/issues/

Closing as it's not part of this library.