Comments (4)
Idea: store the cache file somewhere else? In S3, or at least somewhere that gives us a bit more control?
https://github.com/marketplace/actions/s3-cache-for-github-actions or similar?
from tap-github.
One year later, I'm reopening this, as the cache is clearly not used.
This screenshot from https://github.com/MeltanoLabs/tap-github/actions/caches confirms why:
Cache is python version specific, which defeats the purpose of our caching attempts.
A local run of the test suite consumes about 500 rest requests and 25 graphql points. As we run 5 versions of python, we're still at only 50% of quota on rest, and much less on graphql. So there's probably some other consumer of quota outside of CI.
Storage of cache outside of github seems like the only option at this point. I will follow up on the suggestion above.
from tap-github.
Actually, I think I misunderstood what is happening. The cache key is api-cache-v4
which is python version agnostic.
But the cached file is 3 weeks old in the screenshot above, so our code ignores it (it expires the cache after 24h) so it hits the API again.
My guess at this point is that the cache works the first day we use it, but it then becomes stale and ignored by requests-cache
. 2 options:
- change code to expire the cache after much longer than 24h. The problem being that tests use a
start_date
of more or less "today". So beyond the first day, it's likely that requests might change - use a cache key based on the current date, which should make the cache file renew every 24h, in line with the cache expiry and api requests.
from tap-github.
To expand on the above, this is all because cached files are read-only, so the cache file is not updated beyond its first creation time.
See https://github.com/actions/cache/blob/main/tips-and-workarounds.md#update-a-cache for details.
from tap-github.
Related Issues (20)
- Passing a username as "organizations" config value crashes the tap HOT 5
- KeyError: `commit_timestamp` HOT 5
- Field `fetched_at` in stream `extra-metrics` can be formatted as a date-time string
- Releases stream has 10,000 record limit HOT 3
- The 'pull_number' field not being populated for the 'pull_request_commits' stream HOT 5
- If a member is part of multiple teams, they will only be listed once HOT 2
- ValueError: not enough values to unpack (expected at least 1, got 0) in repository_streams HOT 1
- Incremental replication doesn't respect the current state HOT 1
- Use pre-commit.ci to lint project
- Stream `extra_metrics` fails on repos with large number of issues/PRs HOT 1
- Drop support for python 3.7
- Invalid SCHEMA messages are produced for deselected streams HOT 3
- Replace use of `get_next_page_token` in the tap HOT 2
- Workflow streams incorrectly claim to support incremental loading
- Hard to tell if API token is valid or not HOT 1
- Add `files` property to `CommitsStream` HOT 1
- Experiencing 401 Bad Credentials when credentials are valid
- Document `api_url_base` setting for Enterprise Server installations
- SDK Version pointing to a specific commit
- Loader 'target-jsonl' is not known to Meltano.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tap-github.