Add `requests-cache` implementation in CI to eliminate rate limit failures

[meltanolabs]

Add `requests-cache` implementation in CI to eliminate rate limit failures

[laurentS]

Relates to #14
I think the approach you suggest makes sense. I was pondering using a sort of snapshot system saved in the repo (I think vcr.py does that), but using a cache like you suggest is probably less maintenance.

Github's API rate limiting has a 1h window, after which it resets, so we could have a daily job, or even twice daily scheduled job that busts/primes the cache. I'm thinking of what would happen if github's API format suddenly changes in a way that breaks the tap, the less time we wait for CI to fail the better. If the scheduled job refreshes the caches and then runs the CI job on it, we'd get notified of failures quickly. In that case, we can even run all "regular" CI jobs from cache, so they would be much faster as well, which never hurts (and there's still the risk that 2-3 eager devs are pushing code multiple times within a 1h window and hit limits anyway 😉 )

[aaronsteers]

I was pondering using a sort of snapshot system saved in the repo (I think vcr.py does that), but using a cache like you suggest is probably less maintenance.

I agree. In the related SDK issue, we imagine a dev lifecycle where eventually (if not in the first iteration due to PII), the caches are stored in yaml within the repo. This can power CI tests (except when we intentionally ignore/refresh the cache) and gives us --demo behavior down the road without having to require creds that connect to the source.