Stream stops receiving tweets without any errors

[draftbit]

Stream stops receiving tweets without any errors

[plu5]

You’ve reached the wrong conclusion. It is a twitter-lite issue, or rather oversight. The stream times out after a bit of time -- stops sending keepalives. Some other libraries let you pass through a timeout parameter which is passed along to the request, and the library they use for requests handles it: if it doesn’t get any response after that time emits an error so that it can be handled.

The endpoint provides a 20-second keep alive heartbeat (it will look like a new line character). Use this signal to detect if you’re being disconnected.

1. Your code should detect when fresh content and the heartbeat stop arriving.

(Source)

[Admittedly it seems like most other libraries don’t handle this properly either. I think because the maintainers themselves don’t use streams beyond the short amount of time it takes to test them, so overlook this issue.]

I see that you are using cross-fetch, I am not familiar with it but I think it does not let you pass a timeout:

• lquixada/cross-fetch#62
• https://stackoverflow.com/questions/46946380/fetch-api-request-timeout
• node-fetch/node-fetch#831 “Remove timeout option”
• node-fetch/node-fetch#885

Related issue: #114

---

twitter-lite is kind enough to emit the keepalives at least, so a workaround is to keep track of when the last response from the server (data or ping) was received and run a function on a timer to check if it has been too long.

const timeout = 31000;
let backoffExponential = 0;
let timeLastResponseReceived;

function startStream() {
  const stream = twitterClient.stream('statuses/filter', parameters)
        .on('start', (response) => {
          console.log('Started stream');

          // Stall handling
          const intervalId = setInterval(function() {
            if (Date.now() - timeLastResponseReceived > timeout) {
              console.log('timeout');
              clearInterval(intervalId);
              reconnect(stream);
            }
          }, timeout);
        })
        .on('data', (tweet) => {
          console.log("data", tweet.text); // Whatever you use to handle the data
          timeLastResponseReceived = Date.now();
        })
        .on('ping', () => {
          console.log('ping');
          timeLastResponseReceived = Date.now();
        })
        .on('error', (error) => console.log('error', error))
        .on('end', (response) => console.log('end'));
}

async function reconnect(stream) {
  backoffExponential++;
  if (stream.destroy) stream.destroy();
  await sleep(2 ** backoffExponential * 1000);
  startStream();
  // If you use this code, remember to also:
  // 1. Reset backoffExponential at some point
  // 2. Check if connected to network before starting stream
  // 3. Make sure you do not start a stream if one is already running
}

async function sleep(delay) {
  return new Promise((resolve) => setTimeout(() => resolve(true), delay));
}

For the reconnect logic I looked at https://developer.twitter.com/en/docs/tutorials/building-an-app-to-stream-tweets

Note how he uses request library which lets you specify a timeout, so all he has to do is handle errors. This is ideally what we should be able to do with twitter-lite as well.

[plu5]

Twitter sends a new line character every 20 seconds if no content arrives, which twitter-lite notifies us about with the 'ping' event. So if you update timeLastResponseReceived (or whatever equivalent variable you are using) on ping, and your timeout is greater than 20 seconds, there should be no difference between active or inactive streams.

What twitter-lite should do: as I said in my comment on November 2nd,

[let us] pass through a timeout parameter which is passed along to the request [..] if it doesn’t get any response after that time emits an error so that it can be handled.

[butaca]

According to https://developer.twitter.com/en/docs/tutorials/consuming-streaming-data#reconnecting is expected to receive 1) "a zero-byte chunk" and 2) "no new Tweets AND no keep-alive carriage return signals – for more than 30 seconds"

As far as I can tell (I apologize in advance if I'm wrong, this is the first time I use Node, twitter-lite or the Twitter API):

Case 1) I think this isn't handled by twitter-lite (maybe is handled by Node?)
Case 2) The keep-alive carriage return is the ping event emitted by the twitter-lite stream, but I guess twitter-lite isn't handling the case in which no ping is received after X seconds (something like #59 (comment))

Regarding 2) in my scenario it happens a few times per day. But it is common to receive pings with a time interval greater than 30 seconds, so I would use a greater threshold to consider the stream dead.

I'm willing to contribute to the project but I'm a Node newbie so it may take a while.

[jxlarrea]

Well, it seems like twitter-lite is not at fault here. I basically replicated the same twitter-lite project I had but this time on .NET using the Tweetinvi library and the same behavior is there: streams are randomly stopped with no errors raised whatsoever.

[ShivamJoker]

Actually the Twitter Dev's fixed this internally so it doesn't disconnect 😀

Update here - twitterdev/Twitter-API-v2-sample-code#34 (comment)

Blog link - https://twittercommunity.com/t/filtered-stream-request-breaks-in-5-min-intervals/153926

[ShivamJoker]

@T0TProduction updated the comment.

[Pinta365]

I have not experienced this problem but I have not had my program on for several hours yet so that could be why :)
Just a thought I had, are you still receiving "ping" events after the halt or not?

[dylanirlbeck]

@atomheartother Are you still having this issue? If so, we'll work with you to try and get it fixed.

This may be related to #48, but I'm not entirely sure.

[atomheartother]

@dylanirlbeck I am indeed working on fixing this but as I think I mentioned i need to wait upwards to days for the bug to occur so debugging is extremely tough :/

I have a backoff exponential algorithm set up for reconnecting, and again no events are being triggered but I'll go back through how I do Twitter connection in case it is #48.

[dylanirlbeck]

Okay gotcha --- do let me know if #48 proves helpful. If not, I might need to re-investigate how we do streaming and/or your code.

[dylanirlbeck]

@atomheartother Did you ever figure this out?

[atomheartother]

@dylanirlbeck Hey, actually no, it still happens from time to time, however I've now realized why it started happening recently - it's because I used to run an older node version which kept crashing, so the stream constantly restarted and this bug very rarely occurred. But I recently refactored a lot of stuff and this bug kind of emerged out of less crashes! So it's very possible this has been around for a long time - or maybe not, either way the plot thickens.

I put on messages every time my stream was destroyed (on purpose at least) and nothing weird showed up, and still, everytime, no callbacks are being triggered... I was thinking of forking twitter-lite and setting up some logging messages in some spots to see what info i can gather, but obviously I'm not familiar with the codebase so any advice from you on where to plant my little messages would be helpful 😉

[atomheartother]

Actually yknow what I'm going to write a small bit of code that connects to a stream and just listens to it, no side effects, nothing complicated, & run it on my server for a few days, see if the bug occurs there too. If so, it's one of 2 things, my environment or twitter-lite itself.
If the bug doesn't occur then the problem is definitely in my codebase and I'll close this issue. I don't want to waste your time.

[dylanirlbeck]

I was thinking of forking twitter-lite and setting up some logging messages in some spots to see what info i can gather

I would totally recommend doing this if the issue doesn't resolve itself. The stream.js file would be the way to go, since that's where all the streaming logic is. Also, PRs are most certainly welcome! If, after forking, you see somewhere where we're doing something dumb then please let us know.

Keep me posted on the bit of code you're going to write. If the bug persists, I'll try to replicate on my end.

[atomheartother]

OK BIG NEWS!

I was able to reproduce this bug with very minimal code, so either there's something wrong with the way I set up streams or there's indeed something wrong with twitter-lite! It has now been 3h since I got a tweet from this program (which gets about a tweet every 2-3min).

You can check the code out here, create a .env file copied from .example.env with your twitter keys and run docker-compose up -d and you're good to go. Unless you can find something blatantly wrong with this code, I can now confirm that this is an issue with twitter-lite. I'll be checking out stream.js.

[atomheartother]

(Or, obviously, there's something that goes wrong with twitter-lite in the environment i set up, either way this narrows things down quite a lot!)

[dylanirlbeck]

@atomheartother So I'm not the best with Docker, so I'll need some help with reproducing this issue. Should I just docker attach to the container to see the stdout? docker-compose up -d worked fine, and it looks like the container was installed correctly (running docker ps says the container status is Restarting (1) X seconds ago (where the X is always a different number).

[atomheartother]

Did you enter your keys in .env? Restarting is a sign that it keeps failing actually, but either way you can check out the output with docker-compose logs

[atomheartother]

The plot thickens - but i still haven't given up on this bug, just letting you know.

So since last time, I moved servers, which appears to have affected the bug: it happens even less often, now it's once every 4-5 days. Bad news for me!

Now, this new VPS has more RAM than the previous one, that's my only explanation? It's not much of an explanation though. Anyway my users are still reporting the bug happening on their machines as well, one guy even had it occur after 2h.

I still have my modified version of twitter-lite running on my twitter-lite-test repo, it's printing any kind of buffer that reaches the lib from Twitter, but i haven't been able to catch the elusive bug yet. In short this bug is slowly driving me insane but at least it's happening way less often now! (For reasons i can't comprehend).

[dylanirlbeck]

Hi @atomheartother, sorry for the delay. Things got a bit hectic the last two weeks with Corona, but I'm still committed to figuring this out. It really stinks that you're still experiencing this bug.

I'm going through your twitter-lite-test project and instructions for Docker, and I'm having some trouble getting the container up and running. When I do docker-compose up -d everything seems to work fine but running docker-compose logs gives me the following output:

app_1  |   code: 'MODULE_NOT_FOUND',
app_1  |   path: '/usr/src/app/node_modules/twitter-lite/package.json',
app_1  |   requestPath: 'twitter-lite'
app_1  | }
app_1  | error Command failed with exit code 1.
app_1  | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
app_1  | yarn run v1.21.1
app_1  | $ node --max-old-space-size=512 --optimize-for-size --gc-interval=10 -r esm main.js
app_1  | [/usr/src/app/main.js:1
app_1  | Error: Cannot find module 'twitter-lite'. Please verify that the package.json has a valid "main" entry]

Any ideas on how to solve this? It just looks like yarn/npm isn't installing the packages on the container.

[atomheartother]

@dylanirlbeck This is actually because twitter-lite only creates the dist package on release, and for twitter-lite-test I used a github repo (which doesn't contain dist) so on install it fails.

If you want the version that logs everything and works then update your repo and re-install your packages, I ended up just versioning dist to fix this.

If you don't want my custom version just reset to 5bd3f99962c66aed341abf4aba65adc8786a64a1 which uses the upstream twitter-lite (yours).

[atomheartother]

I got the bug using my modified twitter-lite which logs activity in the parse function... That one took 4 days, which puts in perspective how annoying this is to debug. No breakthrough unfortunately, here's the info i got:

• No pings from Twitter reach twitter-lite once the bug occurs, nothing - more precisely, the parse function of Stream simply never receives any buffer.
• There doesn't appear to be any abnormal activity before the bug occurs
• I triple-checked, I still have plenty of RAM, there's no unusual CPU usage which would indicate an infinite loop or that kind of stuff.

So yeah, no real new data, I'm still in the dark regarding this... I'll keep investigating. I'll join the log for the last few tweets but there's simply not much information :/

twitterlitetest_logs.txt

Edit: Nope, wait a second, I actually found something very interesting, it appears that the container has simply stopped without any errors! This is very strange, but it explains everything, the lack of error messages in particular - twitter-lite simply isn't running anymore! I never noticed because logs are still available for stopped containers and I never thought to check... Well now I'm even more confused though, why are these node containers stopping randomly?? Especially since they are specifically told to restart on failure by docker-compose.

The plot thickens... but also thinnens! This could be unrelated to twitter-lite and have to do with node and Docker. Will keep you updated. Either way I've set my docker-compose to restart: always my service which should "fix" it, I just still don't know why it exits but I have no reason to believe it has anything to do with twitter-lite anymore.

[ImTheDeveloper]

I've had this issue myself also. I think it's a node inside docker issue. I have been running alpine node versions and it seemed to just cut the stream at very random times.

I set an interval timer to destroy the stream and recreate it every 15 minutes and it seems to have solved the issue... But now I sometimes hit 'Enhance your calm' .. FML.

@atomheartother are you able to share your back off strategy? I'll give it a go with that and see if I can run continuous on this known broken setup I have.

[ImTheDeveloper]

Started to see this issue more frequently today, maybe within a few hours. I have log output showing when I'm receiving pings and they just seem to stop. The stream hasn't been destroyed as when I check for existence it's still there. To get around the issue I'm now storing the last time I received a ping. If its over 90 seconds ago I will then kill the stream and restart it with a back off period.

Also worth mentioning, there is nothing received from the twitter stream to suggest why the ping/closure should happen. I've received a couple of status 200 closes in the past but these are fine to restart from, the random stops though not so easy.

Managed to capture the stream dieing on me with the new check in place fixing the issue:

Ping event stores the last Date.now captured. Every 3 minutes Im checking my DB to see if there are any new subscriptions I need to listen for, but also in this process I check to see when the last ping ran. If that ping was over 90 seconds ago then I recreate the stream. 👍

Edit:

I noticed that node v14 has made a change to streams. Whilst reviewing the documentation they have mentioned from 0.10 there has been an edge case with streams being paused.
https://nodejs.org/api/stream.html#stream_compatibility_with_older_node_js_versions

[butaca]

I'm experiencing the same issue. I'm running my bot in a Raspberry Pi, node v8.11.1 (no docker).
I'm handling the error and end events (they aren't called, only start and data). I have to restart the bot every day because it stops receiving events (no errors received).

In my scenario the users I'm following post each day at the same time, and don't post anything in about 24 hours.

[ImTheDeveloper]

I've had mine running for over a week now and it's been a mixed bag of things going on.

I store the last time I received a ping and run an interval to check that this is not greater than 90 seconds ago. If it is I restart.

During restart I also wait 90 seconds before hitting the API to start the stream again. However sometimes earlier in the week I would hit the calming errors. VERY annoying.

I would need to manually stop and calm for 15 minutes to get running again. However since those initial issues the first couple of days and changing no code I've been running with the setup restarting as needed without any problems.

I still lose connection but the restart startegy has been going fine. I also have been running through luminati proxy to ensure my IP address gets changed up as I did suffer rate limiting when the API would continually restart a few times in a row.

I'm thinking there's a couple of issues going on. One with streams in node but also one with updates at twitter's side. Maybe some distruption of their service as there isn't a pattern to the restarts at all. It can happen 1 after another and then nothing for hours or days.

[maybeanerd]

Just barging in to confirm we've also been experiencing this behavior, roughly once or twice a day the stream stops with "200 OK" but simply restarting it that moment works for us.
I assumed it was "intended" by Twitter, but after reading this it seems it's not?
We do have exponential backoff in place if we were to get rate-limited, but we never actually need to wait to start a new stream.

Our product is https://switch.flint.gg; we have tweets coming in constantly (around 80 / second ) since we listen to hashtags. The environment is node12 hosted on AWS with the amazon Linux dist, so I would also assume that this issue is either Twitter, Node, or this lib, nothing within the environment.

[dandv]

v0.9.4 has been used in production (DO VPS running Ubuntu) for over a year on this project, following a set of 1000+ users. The stream has been "stalling" randomly, sometimes every 2 hours, other times running for 4-5 days straight. We guarded against this by exiting the script and restarting it when no tweets had been received for over 30 seconds. I spent a couple days tweaking the library and trying to recreate the stream programmatically, but that didn't work 🤷‍♂️, so I had to process.exit(), lacking a better solution.

[atomheartother]

I just want to say it's quite a fuzzy feeling as a dev to see other people are getting this bug after I was kinda alone on it for a month :P

Edit:

I noticed that node v14 has made a change to streams. Whilst reviewing the documentation they have mentioned from 0.10 there has been an edge case with streams being paused.
nodejs.org/api/stream.html#stream_compatibility_with_older_node_js_versions

Has anyone tried to run their project under v14 yet? Does this fix the issue?

[ImTheDeveloper]

After weeks of no issues I hit it again today. Had the odd 200 closure along the way but this morning I woke to see I needed to once again enhance my calm. 😟

[jxlarrea]

Any progress on this? I've encountered the same issue, streams dying with a 200 response status code after just a few minutes. Not sure why, but the ping event never seems to be triggering.

[atomheartother]

I've recently updated to node v15 and this is still happening. Twitter has to be at fault, and they won't fix this since they're gonna deprecate streams anyway. I'm closing this.

[atomheartother]

@plu5 this is what i'm currently doing, but it's causing 420 rate limits sometimes when my users don't subscribe to very active streams unfortunately.

The problem is - what is twitter-lite supposed to do then, destroy & reconnect the stream itself, when it stops receiving keepalives?

[ShivamJoker]

Is this library getting any fix for this ?

[rlueder]

@ShivamJoker this is not a bug per se, though one could argue it'd be nice for twitter-lite to support stalls/disconnects/reconnects. See this comment for more information on why the 420 is likely happening.

Clients which do not implement backoff and attempt to reconnect as often as possible will have their connections rate limited for a small number of minutes. Rate limited clients will receive HTTP 420 responses for all connection requests.

It shouldn't be hard for you to track outside of twitter-lite when the stream has updated last and if more than 90s have passed without an update as suggested by Twitter disconnect and reconnect, keeping in mind that if you're receiving 420 status responses you should exponentially wait (e.g. 1min, 2min, 3min so on before attempting to reconnect).

If your application is eager to reconnect by, say attempting every 10s, within a minute you'll potentially have received six 420 responses which will add a total of six minutes to how long you'd have to wait before attempting again, you can see how this can easily get out of hand and cause the application to stop streaming completely.

[maybeanerd]

Actually the Twitter Dev's fixed this internally so it doesn't disconnect 😀

@ShivamJoker
Any source/blog/update-log for this? I would like to rely on this fact, because that would make things a lot more comfortable in general.