Comments (1)
visch
commented on July 4, 2024
Another example hit today.
To repeat
- Provide an invalid token
- You get an error like this
2024-03-28 09:33:10,071 | WARNING | tap-github | A token was dismissed. 401 Client Error: b'{"message":"Bad credentials","documentation_url":"https://docs.github.com/rest"}' (Reason: Unauthorized)
2024-03-28 09:33:10,071 | INFO | tap-github | Tap will run with 0 auth tokens
2024-03-28 09:33:10,072 | INFO | tap-github | No auth token detected. For higher rate limits, please specify `auth_token` in config.
2024-03-28 09:33:10,225 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.153058, "tags": {"stream": "tempStream", "endpoint": "", "http_status_code": 403, "status": "failed"}}
2024-03-28 09:33:10,225 | INFO | singer_sdk.metrics | METRIC: {"type": "counter", "metric": "http_request_count", "value": 0, "tags": {"stream": "tempStream", "endpoint": "", "context": {}}}
2024-03-28 09:33:10,225 | ERROR | tap-github | An unhandled error occurred while syncing 'repositories'
Traceback (most recent call last):
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1179, in sync
for _ in self._sync_records(context=context):
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1054, in _sync_records
context_list = [context] if context is not None else self.partitions
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 175, in partitions
augmented_repo_list += self.get_repo_ids(
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 119, in get_repo_ids
for record in temp_stream.request_records({}):
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/rest.py", line 397, in request_records
resp = decorated_request(prepared_request, context)
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
ret = target(*args, **kwargs)
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/rest.py", line 276, in _request
self.validate_response(response)
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 103, in validate_response
super().validate_response(response)
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/client.py", line 440, in validate_response
super().validate_response(response)
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/client.py", line 213, in validate_response
self.authenticator.get_next_auth_token()
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/authenticator.py", line 214, in get_next_auth_token
raise RuntimeError(
RuntimeError: All GitHub tokens have hit their rate limit. Stopping here.
Traceback (most recent call last):
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/bin/tap-github", line 8, in <module>
sys.exit(cli())
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 500, in invoke
tap.sync_all()
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 459, in sync_all
stream.sync()
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1186, in sync
raise ex
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1179, in sync
for _ in self._sync_records(context=context):
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/core.py", line 1054, in _sync_records
context_list = [context] if context is not None else self.partitions
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 175, in partitions
augmented_repo_list += self.get_repo_ids(
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 119, in get_repo_ids
for record in temp_stream.request_records({}):
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/rest.py", line 397, in request_records
resp = decorated_request(prepared_request, context)
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
ret = target(*args, **kwargs)
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/singer_sdk/streams/rest.py", line 276, in _request
self.validate_response(response)
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/repository_streams.py", line 103, in validate_response
super().validate_response(response)
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/client.py", line 440, in validate_response
super().validate_response(response)
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/client.py", line 213, in validate_response
self.authenticator.get_next_auth_token()
File "/home/visch/git/meltano-projects/github_2_postgres/.meltano/extractors/tap-github/venv/lib/python3.10/site-packages/tap_github/authenticator.py", line 214, in get_next_auth_token
raise RuntimeError(
RuntimeError: All GitHub tokens have hit their rate limit. Stopping here.
This runtime error is highly misleading. The github token that got a 401 should most likely fail the tap (Unless there's some reason a 401 is given for something that's needed to retry). Maybe this is related to #243 ?
from tap-github.
Related Issues (20)
- Passing a username as "organizations" config value crashes the tap HOT 5
- KeyError: `commit_timestamp` HOT 5
- Field `fetched_at` in stream `extra-metrics` can be formatted as a date-time string
- Releases stream has 10,000 record limit HOT 3
- The 'pull_number' field not being populated for the 'pull_request_commits' stream HOT 5
- If a member is part of multiple teams, they will only be listed once HOT 2
- ValueError: not enough values to unpack (expected at least 1, got 0) in repository_streams HOT 1
- Incremental replication doesn't respect the current state HOT 1
- Use pre-commit.ci to lint project
- Stream `extra_metrics` fails on repos with large number of issues/PRs HOT 1
- Drop support for python 3.7
- Invalid SCHEMA messages are produced for deselected streams HOT 3
- Replace use of `get_next_page_token` in the tap HOT 2
- Workflow streams incorrectly claim to support incremental loading
- Add `files` property to `CommitsStream` HOT 1
- Experiencing 401 Bad Credentials when credentials are valid
- Document `api_url_base` setting for Enterprise Server installations
- SDK Version pointing to a specific commit
- Loader 'target-jsonl' is not known to Meltano.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tap-github.