Giter Site home page Giter Site logo

Comments (2)

aaronsteers avatar aaronsteers commented on July 20, 2024 1

@ericboucher - For the SDK generic handling, I'm thinking a multi-facet approach along these lines:

  1. Add a public and overridable method in RESTStream called raise_if_error() (or similar), which would abstract the case logic in _request_with_backoff(). By default, the raise_if_error() would check status codes just as we do today, and then raise one of a specific set of exception types to be handled by _request_with_backoff() and the SDK internals.
  2. Predefined exception types would be documented in the dev guide, to be optionally sent by custom implementations, and then caught and handled by the SDK.
  3. We probably would want to introduce both "Fatal" and "Retryable" exception classes, and also add exception classes to signify rate limits.
    • Rate limit exceptions could similarly be categorized as retryable or fatal, depending on period (per-minute limits being retryable and per-day limits not, for instance).
  4. The set of status codes that roll up into each exception type should also be overridable by the developer.
    • The example here uses tolerated_http_errors. We could support a RESTStream.tolerated_http_errors property or perhaps just allow users to override fatal_http_errors to exclude the error code in question.
    • If the user overrides the numeric code list (RESTStream.tolerated_http_errors or RESTStream.fatal_http_errors, for instance), then they may or may not actually need to also override raise_if_error(). Presumably, they'd only need to do both when the status code is not sufficient information or when the response needs to be parsed in order to properly determine the nature of the exception message.
    • Potential flaw in this plan is that if we want an entire series of codes to represent a failure, it may require adding all the codes in the sequence, something like range(start=400, stop=500). I'm not sure how much of an issue this would be, but wanted to call it out.

@ericboucher - What do you think of this approach? It should give you less you would need to override in cases such as these.

cc @edgarrmondragon

from tap-github.

ericboucher avatar ericboucher commented on July 20, 2024

Linked issue on the SDK - Error Handling - https://gitlab.com/meltano/sdk/-/issues/134

from tap-github.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.