Add Retry Strategy to manual libraries.

[googleapis]

Add Retry Strategy to manual libraries.

[RunOrVeith]

Since these retry-able errors happen quite frequently, I'd really like for this to be a feature sooner than later.
Thanks for your efforts!

[dylancaponi]

We implemented this in production and have seen zero errors after 5 days.

Previously we were seeing ~10 errors a day.

This solved our issues. Thanks @micimize & @pietrodn

def add_client_retries(client):
    """ Retry connection to prevent connectionResetError
    """
    adapter = requests.adapters.HTTPAdapter(max_retries=5)
    client._http.mount("https://", adapter)
    client._http._auth_request.session.mount("https://", adapter)
    return client

in main.py:

from google.cloud import storage
from google.cloud import bigquery
storage_client = add_client_retries(storage.Client())
bq_client = add_client_retries(bigquery.Client())

Also, remember to only initialize these once outside of your main() function and reuse them in main().

[andrewsg]

This is now resolved for storage. Retries are supported by default, documentation work for configuring them is pending (for now, see docstrings).

[micimize]

@dylancaponi does it work in the same environment without the _http argument? 401 means unauthorized, so you probably need to "bind the credentials" like the docs say the default does:

If not passed, an _http object is created that is bound to the credentials for the current object

[micimize]

They broke the monorepo up, but I found where they set credentials for reference:

        if self._http_internal is None:
            self._http_internal = google.auth.transport.requests.AuthorizedSession(
                self._credentials,
                refresh_timeout=_CREDENTIALS_REFRESH_TIMEOUT,
            )

[dylancaponi]

Thanks! I think I want to do something like:

storage_client = storage.Client()
adapter = requests.adapters.HTTPAdapter(max_retries=5)
storage_client._http.mount("https://", adapter)
storage_client._http._auth_request.session.mount("https://", adapter)

as mentioned here:
googleapis/python-bigquery#59 (comment)

After 8 days of testing, this function did not terminate once due to connectionResetError!

Adding to production code and will report back.

[micimize]

This might be possible with the gcs client currently via

session = requests.Session()
session.mount("https://storage.googleapis.com", requests.adapters.HTTPAdapter(max_retries=5))
client = Client(project=env.GOOGLE_PROJECT, _http=session)

Haven't tested though

[dylancaponi]

@micimize This is giving me errors (below) on deployment. Any idea why?

I'm assuming Client here is really:
from google.cloud import storage
storage.Client(...) ?

Code snippet:

from google.cloud import storage
import requests
session = requests.Session()
session.mount("https://storage.googleapis.com", requests.adapters.HTTPAdapter(max_retries=5))
client = storage.Client(project=PROJECT_ID, _http=session)

Error when attempting blob.download_as_string():

ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Function failed on loading user code. Error message: 401 GET https://storage.googleapis.com/download/storage/v1/b/<redacted file name>/o/secrets?alt=media: ('Request failed with status code', 401, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

[dylancaponi]

@micimize Yes, the code runs when we don't pass _http. That is how our normal (non-retrying) connection is established.

builtins.storage_client = storage.Client(project='<redacted>')

However, that means it's not passing the session that defines 5 retries which I really want to do. Do you know what the proper approach to that is?

[micimize]

@dylancaponi I don't, but what I would do is keep digging through the source until you find google.auth.transport.requests.AuthorizedSession. You basically want to extract the auth logic they're using and figure out how to combine it with the requests.adapters.HTTPAdapter(max_retries=5) solution, and then pass the resulting session in as _http

[dylancaponi]

The solution above worked for a month or two but we are now experiencing this issue again ...
How is everyone else faring?

[jackkwok]

@andrewsg : What release version of google-cloud-storage is retries supported by default?