Allow the tap to continue on server side timeout (error 502)

[meltanolabs]

Allow the tap to continue on server side timeout (error 502)

[laurentS]

I opened https://gitlab.com/meltano/sdk/-/issues/282 to suggest a solution at the SDK level.

[ericboucher]

• MAX_PER_PAGE was updated in #53

• Implementation idea:
  Similar to our list of tolerated_error, we could have a list of continue_on_errors. When a partition exits on such an error, it would add a flag in its context, error_occured: true. If such a flag is raised, we could then override [Stream]._increment_stream_state to check this flag and NOT update the state for this partition if it has been raised.

[laurentS]

A couple of ideas to let the tap continue upon errors (assuming backoff has tried N times and failed, like we've seen a number of times in our logs):

Tolerated errors

We could add all sorts of error codes to the tolerated_http_errors list for a stream, but this has the downside of losing data, as the tap will update the bookmark to past the erroneous data.

Override stream request decorator

The RestStream class has a request_decorator which we could override in the tap when a specific config option is passed to continue_on_error. The decorator could then build a mock Response so that the rest of the stream moves on to the next partition. The state bookmark should be set to "before" the error, so it can be retried on the next run.

The difficulty would be in figuring out which errors are temporary, and which are not:

• the example URL above is still returning the same 502 after 10s more than 3 months later. It's unlikely it will ever work. At what point do we consider an error "permanent"?
• on the other hand, we had a 500 on /repos/opensourcedesign/opensourcedesign.github.io/commits yesterday which worked fine about 24h later.

A possible option would be to add a retried_count in the state bookmark which keeps track of how many times (or over how long) a specific partition+stream has been retried, and gives up after N attempts over multiple runs.