Comments (25)
Also experiencing this consistently at 5 minute mark. Noticed after these however, I wouldn't receive a 429, so wrapped a try/except around the iter_lines() and continued with a while loop.
try:
for response_line in response.iter_lines():
if response_line:
json_response = json.loads(response_line)
print(json_response)
except requests.exceptions.ChunkedEncodingError:
encoding_count += 1
A crude patch until Twitter Devs figure out what's happening on their end?
from twitter-api-v2-sample-code.
Thanks for all the information here and the workarounds to reconnect. We'll see what we can do to integrate those changes into future versions of the sample code as needed.
from twitter-api-v2-sample-code.
For anyone still following/escaping this error, it would appear the error was on the server side, and was patched yesterday:
https://twittercommunity.com/t/filtered-stream-request-breaks-in-5-min-intervals/153926
Good to know we weren't all going mad!
from twitter-api-v2-sample-code.
I encounter similar issue with "filtered_stream.py" which was working for hours without any problem a month ago.
from twitter-api-v2-sample-code.
@andypiper I've submitted a pull request for the Python one at #36
Thanks for all the information here and the workarounds to reconnect. We'll see what we can do to integrate those changes into future versions of the sample code as needed.
from twitter-api-v2-sample-code.
Okay I tested this on JavaScript and seems to be fixed.
from twitter-api-v2-sample-code.
Glad someone else can confirm this. In production I reconnect, but it seems an imperfect solution as it likely still means I lose data. Furthermore, nodejs seems to behave awkward here – in v12 the stream ends and I can reconnect, but in v14 and v16 nodejs just silently stops code execution: no error to catch, no loop to continue.
from twitter-api-v2-sample-code.
Does anyone have a workaround for the NodeJS version? I've attached listeners to every possible event and the stream just stops without triggering any of my logs.
from twitter-api-v2-sample-code.
Only workaround I found is to reconnect immediately. This only works properly with node v 12.x since in later versions the stream does not stop immediately, but hangs for a considerable amount of time before it disconnects. Another alternative is to use python as python at least throws an error which you can handle (and reconnect).
from twitter-api-v2-sample-code.
Only workaround I found is to reconnect immediately. This only works properly with node v 12.x since in later versions the stream does not stop immediately, but hangs for a considerable amount of time before it disconnects. Another alternative is to use python as python at least throws an error which you can handle (and reconnect).
Thanks for the reply, I am working with python but I did not fully understand the structure. I want to ask you if you can help. If we consider the example python code (https://github.com/twitterdev/Twitter-API-v2-sample-code/blob/master/Filtered-Stream/filtered_stream.py) where should I put try/except block? And what is the meaning of reconnection in terms of the code should I call get_stream function in the exception.
from twitter-api-v2-sample-code.
I ended up switching to the https
module and that works fine, must be an issue with needle
.
from twitter-api-v2-sample-code.
I can reproduce the problem with needle, fetch, python and also https. Using https in the following function as a replacement for the example code function streamConnect(retryAttempt)
still breaks after five minutes:
function streamConnect(retryAttempt) {
https.get(streamURL, {
headers: {
"User-Agent": "v2FilterStreamJS",
"Authorization": `Bearer ${token}`
},
timeout: 20000
}, (stream) => {
console.log(stream.statusCode)
stream.on('data', data => {
try {
const json = JSON.parse(data);
console.log(json);
// A successful connection resets retry count.
retryAttempt = 0;
} catch (e) {
if (data.detail === "This stream is currently at the maximum allowed connection limit.") {
console.log(data.detail)
process.exit(1)
} else {
// Keep alive signal received. Do nothing.
}
}
}).on('err', error => {
if (error.code !== 'ECONNRESET') {
console.log(error.code);
process.exit(1);
} else {
// This reconnection logic will attempt to reconnect when a disconnection is detected.
// To avoid rate limits, this logic implements exponential backoff, so the wait time
// will increase if the client cannot reconnect to the stream.
setTimeout(() => {
console.warn("A connection error occurred. Reconnecting...")
streamConnect(++retryAttempt);
}, 2 ** retryAttempt)
}
});
return stream;
});
}
Do you mind sharing your code, @jahvi (and the node version you are on since this seems to have some impact on the error?)
from twitter-api-v2-sample-code.
@wiertz Actually I just noticed it does stop as well after ~5mins, the only reason I thought it didn't was because it doesn't exit like with needle
but it still stops nonetheless..
I'll try a few more things but I may end up having to switch to the python version.
from twitter-api-v2-sample-code.
I think I managed to fix it now by adding the reconnection logic to the end
event as well (still using https
, I tried the same thing with needle
and it didn't work).
It's been running for 20+ minutes now and it still stops but it reconnects automatically:
stream.on('end', () => {
setTimeout(() => {
console.warn(
'A connection error occurred. Reconnecting...'
);
streamConnect(++retryAttempt);
}, 2 ** retryAttempt);
});
EDIT: Ignore this, it doesn't work anymore somehow... I switched to using python + @Da1ne fix above as it's more consistent.
from twitter-api-v2-sample-code.
I have the same issue with python, any solution?
Also i get this error with code 104:
Traceback (most recent call last):
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/site-packages/urllib3/response.py", line 437, in _error_catcher
yield
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/site-packages/urllib3/response.py", line 519, in read
data = self._fp.read(amt) if not fp_closed else b""
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/http/client.py", line 461, in read
n = self.readinto(b)
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/http/client.py", line 495, in readinto
return self._readinto_chunked(b)
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/http/client.py", line 590, in _readinto_chunked
chunk_left = self._get_chunk_left()
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/http/client.py", line 558, in _get_chunk_left
chunk_left = self._read_next_chunk_size()
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/http/client.py", line 518, in _read_next_chunk_size
line = self.fp.readline(_MAXLINE + 1)
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/ssl.py", line 1071, in recv_into
return self.read(nbytes, buffer)
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/ssl.py", line 929, in read
return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/site-packages/TwitterAPI/TwitterAPI.py", line 373, in _iter_stream
buf += self.stream.read(1)
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/site-packages/urllib3/response.py", line 541, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/contextlib.py", line 130, in exit
self.gen.throw(type, value, traceback)
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/site-packages/urllib3/response.py", line 455, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/xxx/repos/2themoon/news-classifier/newsclassifier/websource/api/impl/twitter/v2.py", line 276, in producer_streaming
for line in stream:
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/site-packages/TwitterAPI/TwitterAPI.py", line 409, in iter
for item in self._iter_stream():
File "/home/xxx/anaconda3/envs/ai/lib/python3.7/site-packages/TwitterAPI/TwitterAPI.py", line 399, in _iter_stream
raise TwitterConnectionError(e)
TwitterAPI.TwitterError.TwitterConnectionError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
Thanks in advance!
from twitter-api-v2-sample-code.
Hey Everyone, it's a problem that arises with some defective http servers that can't be handled by httplib (dependency of requests). So I came up with the following method to tackle it.
while True:
try:
response = requests.get("https://api.twitter.com/2/tweets/search/stream?expansions=author_id", headers=self.headers, stream=True)
if(response.status_code!=200):
logger.error(f"SearchStream {response.json()}")
break
self.on_stream_trigger(response)
except KeyboardInterrupt as e:
print("\nSTREAM CLOSED!")
raise SystemExit(e)
except Exception as e:
logger.exception(e)
continue
You can use loggers to keep track of all the exceptions that are occuring. I know that this is an ugly fix but hey, it works!
from twitter-api-v2-sample-code.
Guys any workaround for NodeJS ?
from twitter-api-v2-sample-code.
It seems we can use PM2 to keep the server running if it crashes
from twitter-api-v2-sample-code.
@twitterdev is anyone going to look into this? I have seen this problem described across node, ruby and python so I don't think it's a problem with client implementations which are consuming stream api. First reports are dated back to October 2020
https://twittercommunity.com/t/rate-limit-on-tweets-stream-api/144389
from twitter-api-v2-sample-code.
If this is such a long issue then how people are making bots with it ? @arekgotfryd
from twitter-api-v2-sample-code.
@ShivamJoker I would love to know. Do you happen to know?
from twitter-api-v2-sample-code.
Also experiencing this consistently at 5 minute mark. Noticed after these however, I wouldn't receive a 429, so wrapped a try/except around the iter_lines() and continued with a while loop.
try: for response_line in response.iter_lines(): if response_line: json_response = json.loads(response_line) print(json_response) except requests.exceptions.ChunkedEncodingError: encoding_count += 1
A crude patch until Twitter Devs figure out what's happening on their end?
Hi @Da1ne,
I appreciate the work-around for catching the error. What are you doing with the error though once you've caught it, and what is the "encoding_count" used for? How are you restarting the script once it bonks? Do you have this inside a while loop that can keep the script running after each exception?
from twitter-api-v2-sample-code.
Thank you @andypiper for taking a look at it. Nevertheless I think it has become pretty clear by now that this is not a client side issue. All solutions I have seen thus far just mean catching the error and reconnecting. Afaics this is a backend issue and it would be awesome if this would get some attention there.
To be really honest I never expected a solution here, but was trying to raise attention and see if others have this issue. So feel free to close this if it is more appropriately addressed elsewhere.
from twitter-api-v2-sample-code.
Also experiencing this consistently at 5 minute mark. Noticed after these however, I wouldn't receive a 429, so wrapped a try/except around the iter_lines() and continued with a while loop.
try: for response_line in response.iter_lines(): if response_line: json_response = json.loads(response_line) print(json_response) except requests.exceptions.ChunkedEncodingError: encoding_count += 1
A crude patch until Twitter Devs figure out what's happening on their end?
Hi @Da1ne,
I appreciate the work-around for catching the error. What are you doing with the error though once you've caught it, and what is the "encoding_count" used for? How are you restarting the script once it bonks? Do you have this inside a while loop that can keep the script running after each exception?
Hi ctilly,
Really depends what you're doing with your implementation. For me:
if __name__ == "__main__":
encoding_count = 0
while encoding_count < 5:
try:
main()
except Exception as e:
if encoding_count == 4:
# do something here. I send an alert
raise SystemExit
else:
print(e) # typically 429
# do something to wait for the alert.
time.sleep(1000) # this is enough for me, however a better implementation is a backoff method (see forums)
encoding_count +=1
Hope this helps!
from twitter-api-v2-sample-code.
Also experiencing this consistently at 5 minute mark. Noticed after these however, I wouldn't receive a 429, so wrapped a try/except around the iter_lines() and continued with a while loop.
try: for response_line in response.iter_lines(): if response_line: json_response = json.loads(response_line) print(json_response) except requests.exceptions.ChunkedEncodingError: encoding_count += 1
A crude patch until Twitter Devs figure out what's happening on their end?
Hi @Da1ne,
I appreciate the work-around for catching the error. What are you doing with the error though once you've caught it, and what is the "encoding_count" used for? How are you restarting the script once it bonks? Do you have this inside a while loop that can keep the script running after each exception?Hi ctilly,
Really depends what you're doing with your implementation. For me:
if __name__ == "__main__": encoding_count = 0 while encoding_count < 5: try: main() except Exception as e: if encoding_count == 4: # do something here. I send an alert raise SystemExit else: print(e) # typically 429 # do something to wait for the alert. time.sleep(1000) # this is enough for me, however a better implementation is a backoff method (see forums) encoding_count +=1
Hope this helps!
Ah! I didn't think of putting the error trap in the "if name" block. I'm trapping the error and then resetting the connection within the main function and that worked really well today. However, I like this idea of trapping the error outside main() because I can reset the whole script if I need to. Thanks for the tip!
from twitter-api-v2-sample-code.
Related Issues (20)
- Authentication Error (403 Forbidden) when Running Twitter Bot HOT 4
- Was the users lookup endpoint removed from the free tier
- Add JS examples in Direct-Messages-Lookup folder
- Where's the media upload script in this repository? How can I attach an image to a tweet here? HOT 6
- Filtered stream example can cause 429 error
- delete_tweet.py returns 403 in free tier HOT 1
- How to do a Webhook connection using Twitter activity Api v2 to create a chatbot and access DM? using BASIC plan.
- you must use keys and tokens from a Twitter developer App that is attached to a Project
- the whole x dev bs needs fixing
- 'Enviornment?' Oy.
- How to fix "Uncaught Error Error [ERR_REQUIRE_ESM]: require() of ES Module not supported"? HOT 1
- When authenticating requests to the Twitter API v2 endpoints, you must use keys and tokens from a Twitter developer App that is attached to a Project. You can create a project via the developer portal HOT 2
- Twitter V2 Get Tweets API's Query Parameter not working HOT 1
- Getting Tweets with Bearer Token is not working
- How to solve the bug about issue #69
- Error to get tweets from my account HOT 1
- ConnectionError
- Error adding rules: 403 Forbidden When authenticating requests to the Twitter API v2 endpoints, you must use keys and tokens from a Twitter developer App that is attached to a Project. You can create a project via the developer portal.
- Error adding rules: 403 Forbidden When authenticating requests to the Twitter API v2 endpoints, you must use keys and tokens from a Twitter developer App that is attached to a Project. You can create a project via the developer portal HOT 1
- Sd
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from twitter-api-v2-sample-code.