Filtered stream disconnected every 5 minutes

[hunterlarco]

Filtered stream disconnected every 5 minutes

[AnthonyFuller]

This isn't too helpful to the issue but I am using a different module (Twit) and I believe that uses the v1 of the API and it is also disconnecting with the messages aborted every 5 minutes without fail. This must be an issue with twitter.

[wiertz]

Thanks, Anthony. Somewhat reassuring that I am not the only one seeing this. As a follow-up:

• I updated my node version from 12 to 14 LTS. Since then, the stream does not end without error, but times out. This is not only odd, but from a data collection point of view even worse, because I wait for up to 30 seconds before reconnecting and probably miss tweets in the meantime. And it makes me wonder whether there is some low-level interaction problem between nodejs and twitter.
• I also created a very simple test script to check if the error is particular to twitter-v2, using needle instead of fetch as http client (as do most Twitter examples). Behavior is exactly the same. The script below for me closes reliably after ~5 min with "Stream unresponsive". (Would appreciate if someone else could confirm this.)

const needle = require('needle');

// Create 30 second timer to catch stream timeouts
// (Twitter sends 'keep alive' signal ~every 20 minutes)
let timer
refreshTimeout = function () {
    if (timer) clearTimeout(timer)
    timer = setTimeout(() => {
        console.log(`${new Date()}: Stream unresponsive.`)
        process.exit(0)
    }, 30 * 1000)
}


// Send request to API filtered stream endpoint
// Ues TWITTER_BEARER_TOKEN from environment
const stream = needle.get('https://api.twitter.com/2/tweets/search/stream', {
    headers: { "Authorization": `Bearer ${process.env.TWITTER_BEARER_TOKEN}` }
});

// Create listeners for stream events and log activity
stream
    .on('data', function (data) {
        refreshTimeout()
        process.stdout.clearLine();
        process.stdout.cursorTo(0);
        try {
            const json = JSON.parse(data)
            if (json.data) {
                process.stdout.write(`${new Date()} <- Time of last tweet.`)
            } else {
                process.stdout.write(`${new Date()} <- Message from Twitter: ${json}`)
            }
        } catch (error) {
            process.stdout.write(`${new Date()} <- Keep alive signal received.`)
        }
    })
    .on('err', function (error) {
        console.log(`\n${new Date()}: Error (${error.message}).`)
    })
    .on('end', function () {
        console.log(`\n${new Date()}: Stream ended.`)
    });


``

[wiertz]

The issue also affects python (as tested with twitters own python example). I close this here since a solution will probably have to come from elsewhere.