Comments (2)
@ericboucher - For the SDK generic handling, I'm thinking a multi-facet approach along these lines:
- Add a public and overridable method in
RESTStream
calledraise_if_error()
(or similar), which would abstract the case logic in_request_with_backoff()
. By default, theraise_if_error()
would check status codes just as we do today, and then raise one of a specific set of exception types to be handled by_request_with_backoff()
and the SDK internals. - Predefined exception types would be documented in the dev guide, to be optionally sent by custom implementations, and then caught and handled by the SDK.
- We probably would want to introduce both "Fatal" and "Retryable" exception classes, and also add exception classes to signify rate limits.
- Rate limit exceptions could similarly be categorized as retryable or fatal, depending on period (per-minute limits being retryable and per-day limits not, for instance).
- The set of status codes that roll up into each exception type should also be overridable by the developer.
- The example here uses
tolerated_http_errors
. We could support aRESTStream.tolerated_http_errors
property or perhaps just allow users to overridefatal_http_errors
to exclude the error code in question. - If the user overrides the numeric code list (
RESTStream.tolerated_http_errors
orRESTStream.fatal_http_errors
, for instance), then they may or may not actually need to also overrideraise_if_error()
. Presumably, they'd only need to do both when the status code is not sufficient information or when the response needs to be parsed in order to properly determine the nature of the exception message. - Potential flaw in this plan is that if we want an entire series of codes to represent a failure, it may require adding all the codes in the sequence, something like
range(start=400, stop=500)
. I'm not sure how much of an issue this would be, but wanted to call it out.
- The example here uses
@ericboucher - What do you think of this approach? It should give you less you would need to override in cases such as these.
from tap-github.
Linked issue on the SDK - Error Handling - https://gitlab.com/meltano/sdk/-/issues/134
from tap-github.
Related Issues (20)
- Passing a username as "organizations" config value crashes the tap HOT 5
- KeyError: `commit_timestamp` HOT 5
- Field `fetched_at` in stream `extra-metrics` can be formatted as a date-time string
- Releases stream has 10,000 record limit HOT 3
- The 'pull_number' field not being populated for the 'pull_request_commits' stream HOT 5
- If a member is part of multiple teams, they will only be listed once HOT 2
- ValueError: not enough values to unpack (expected at least 1, got 0) in repository_streams HOT 1
- Incremental replication doesn't respect the current state HOT 1
- Use pre-commit.ci to lint project
- Stream `extra_metrics` fails on repos with large number of issues/PRs HOT 1
- Drop support for python 3.7 HOT 1
- Invalid SCHEMA messages are produced for deselected streams HOT 3
- Replace use of `get_next_page_token` in the tap HOT 2
- Workflow streams incorrectly claim to support incremental loading
- Hard to tell if API token is valid or not HOT 1
- Add `files` property to `CommitsStream` HOT 1
- Experiencing 401 Bad Credentials when credentials are valid
- Document `api_url_base` setting for Enterprise Server installations
- SDK Version pointing to a specific commit HOT 1
- Loader 'target-jsonl' is not known to Meltano.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tap-github.