Giter Site home page Giter Site logo

twitter-stream-api's Introduction

Twitter Stream API (v2)

Tests Formats Version Total Downloads codecov

Consume the Twitter Stream API v2 in real-time.

This package is the spiritual successor of fennb/phirehose. It also uses some of salsify/jsonstreamingparser and maxakawizard/json-collection-parser.

Getting started

You need an approved developer account. If you don't have one, apply here. Once you are in, create an "Application" in the Developer Portal and generate a new bearer token.

Requires PHP 8.1+

You can install the package via composer:

composer require redwebcreation/twitter-stream-api

Then, create a connection:

$connection = new \Felix\TwitterStream\TwitterConnection(
    bearerToken: '...' # the one you got from the Developer Portal
);

Usage

Streams

Creating a stream

$stream = new \Felix\TwitterStream\Streams\VolumeStream();
// or
$stream = new \Felix\TwitterStream\Streams\FilteredStream();

Configuring a stream

  • withTweetLimit(int) - Limit the number of tweets a connection should process
    $stream->withTweetLimit(100_000);
  • fields(string[]) - See Fields
  • expansions(...string) - See Expansions

For advanced use

  • withBufferSize(int = 85) - How many bytes should the parser store before trying to parse the JSON, on very high-volume streams, using a larger buffer size is recommended (2500, 10000, depending on the volume). Setting to a big value > 2000 on a low-volume stream would result in 0 tweets being processed until there are enough tweets in the buffer.

Interacting with a stream

  • stopListening() - Stops listening to the stream.
  • createdAt(): int - The UNIX timestamp at which you started listening
  • timeElapsedInSeconds(): int - How much time passed since you started listening
  • tweetsReceived(): int - How much the stream sent

For advanced use

  • response(): Psr\Http\Message\ResponseInterface - The response sent by Twitter

Listening to a stream

$stream->listen($connection, function (object $tweet) {
    echo $tweet->data->text . PHP_EOL;
});

Filtering the stream

This part only applies if you're interested in the filtered stream.

Building a rule

Note, If you change your rules while connected to the stream, Twitter will use the new rules immediately.

Save, read and delete rules

You can not update rules.

use Felix\TwitterStream\Rule\RuleManager;

$rule = new RuleManager($connection);

Let's create a rule:

$rule->save(
	# tweets must contain the word cat and have at least one image
	"cat has:images",
	"images of cats"
);

You may now retrieve your newly saved rule:

$rule->all();

Which returns an array of Felix\TwitterStream\Rule\Rule:

[
	0 => Felix\TwitterStream\Rule\Rule{
		+value: "cat has:images",
		+tag: "images of cats",
		+id: "4567654567654567654"
	}
]

Note, the Felix\TwitterStream\Rule\Rule is merely a Data Object, it does not contain any method.

To delete the rule pass its ID to the delete method:

$rule->delete('4567654567654567654');
Batch Processing

To save many rules at once:

use Felix\TwitterStream\Rule\Rule;

$rule->saveMany([  
   new Rule("cats has:images", "cat pictures"),  
   new Rule("dogs has:images", "dog pictures"),  
   new Rule("horses has:images", "horse picture"),  
]);

To delete these new rules,

$rule->delete([
	'[RULE ID]',
	'[RULE ID]',
	'[RULE ID]',
]);

Validating your rules

You can either use the validate() method:

# returns a list of errors
$errors = $rule->validate('cats ha:images');

Or, the save and saveMany method both have a dryRun parameter:

$rule->save('...', '...', dryRun: true);

$rule->saveMany([...], dryRun: true);

Rule Builder

Every operator is available, here's an example:

$rule->new('listening to music')
    ->raw('#nowplaying')
    ->isNotRetweet()
    ->lang('en')
    ->save();

You may also use and[Operator], or[Operator], for example orNotFrom('ID') or andBioLocation('location').

Compiling this would produce the following:

#nowplaying -is:retweet lang:en sample:10
Tips
  • To directly add a string to the rule, use raw(string)
  • You may call dump() or dd() to quickly debug your rule.
  • and is the default operator, you may omit it. For example, andIsNotRetweet() is the same as isNotRetweet().

Fields

Fields allow for more customization regarding the payload returned per tweet. Let's see that in an example below:

$stream
    ->fields([
        'tweet' => 'author_id'
         // or,
         // 'tweet' => ['author_id', '...']
    ])
    ->listen(...);

Which could return:

{
  "data": {
    "id": "1234321234321234321",
    "text": "Hello world!",
    "author_id": "5678765678765678765"
  }
}

Here's the list of all the available field types and their respective object model (last updated: Aug. 2022):

  • Tweet
  • User
  • Media
  • Poll
  • Place

You can also check out Twitter’s documentation for more details.

Expansions

Expansions let you expand ids to their complete object, for example, if you request an extra author_id field, you may expand it using the author_id expansion:

$stream
    ->fields(['tweet' => 'author_id'])
    ->expansions('author_id')
    ->listen(...);

Which could return:

{
  "data": {
    "id": "1234321234321234321",
    "text": "Hello world!",
    "author_id": "5678765678765678765"
  },
  "includes": {
    "users": [
      {
        "id": "5678765678765678765",
        "name": "John Doe",
        "username": "johndoe"
      }
    ]
  }
}

The list of expansions is quite extensive and not all expansions work the same, please check out Twitter's documentation. on the subject.

Testing

composer test

Twitter Stream API was created by Félix Dorn under the MIT License.

twitter-stream-api's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

twitter-stream-api's Issues

->to is ->from

Hi, thanks VERY much for picking this project up and moving it on.

I have found a little bug within RuleBuilder function to() on line 45 which should be setting $this->to but is setting $this->from = $users instead.

fread(): SSL: Connection reset by peer

Hi,

When running Twitter Stream API v2, sometimes I get the following error:

PHP Warning:  fread(): SSL: Connection reset by peer in /(...)/guzzlehttp/psr7/src/Stream.php on line 231

After that, it was able to reconnect after 15-20 minutes.
I'm trying to reconnect by doubling up to 320 seconds with reference to the document below, but it does not work.
https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/handling-disconnections

When I was using Stream API v1.1 with phirehose, the program would automatically try to reconnect, but this library terminates the program when the connection is lost.
Is there an efficient way to reconnect?

And is this a problem on Twitter's side?
Is there anything I can do to prevent this problem?

Please understand that the wording is strange because English is not my native language 🙇‍♂️

Question: Request a thread

Hello,

is it possible to get a whole thread? I don't see any possibility of just catching the thread through the tweet replies alone. Unless there are no answers on the side.
It seems to me that the API doesn't offer any functions.

Regards

Unable to get twitter user public tweets using Twitter streaming API V2

I want to get real time user tweets from twitter using twitter streaming api v2, I use https://api.twitter.com/2/tweets/search/stream/rules endpoint to set rules with rule like "value": "from:GulzadeNaser" first according to twitter developer api doc, then by calling https://api.twitter.com/2/tweets/search/stream end point to get the real time tweets Unfortunately I could not get the user tweets, but when I add the rule beside above mentioned rule like #demetakalin hashtag, then the mentioned user tweets and other tweets which have hashtag #demetakalin will be on on the result response.

I could get connected with stream, and the connection lives for just 20 minutes or or but when I set the rule to just "from:twitteruseraccount" there is no response, but If I add "#trendinghash" I get the response tweets of hashtag plus "from:twitteruseraccount" tweets too.

I can receive real time tweets using https://npm.runkit.com/twitter-v2. with the same rule set for my account.

This is the rule
"data": [
{
"id": "1555098086317473794",
"value": "from:1549244593832697856",
"tag": "all tweets from the user"
},

Implemented code:
$twitterStream
->expansions('author_id')
->fields('media.duration_ms', 'media.height')
->fields('place.full_name', 'place.geo', 'place.id','place.country','place.name','place.place_type')
->fields('user.name','user.id','user.created_at')
->listen($connection, function (object $tweet) {
print_r($tweet);
//echo $tweet->data->text;

    if ($this->received >= 5) {
        $this->stopListening();
    }

});
Thanks,

Example from README fails

hi,

the ->listen($connection, function (array $tweet) { bails out as $tweet is a stdClass while array is expected

Rulebuilder::raw quotes input

hi,

when passing a string #hashtag to raw it generates #hashtag as expected.
passing ->raw('#hashtag1 #another #andso #on') generates
"#hashtag1 #another #andso #on" which only accepts exact matches. I haven't found out where the logic for that lives and how i can pass in a string e.g. from a form field without the quotes beeing added, any hint would be nice!

Getting error `429 Too Many Requests`

I'm listening to a filtered stream and most of the time its working just fine but every now and again the process falls over reporting a 429 Too Many Requests error with the message:-
Fatal error: Uncaught Felix\TwitterStream\Exceptions\TwitterException: Should not happen: please open an issue at https://github.com/felixdorn/twitter-stream-api/issues

I'm not sure what additional data (error say (include the payload) but I' not sure what you mean by payload - the header does imply a rate limit issue (see below)

I may well be hitting a twitter rate limit - please recommend how to monitor this and back off (pause) from my code when the rate limit is near zero.

Thanks, Andy

Header from request....

-response: GuzzleHttp\Psr7\Response^ {fennb#29
-reasonPhrase: "Too Many Requests"
-statusCode: 429
-headers: array:20 [
"date" => array:1 [
0 => "Mon, 27 Mar 2023 12:05:34 UTC"
]
"perf" => array:1 [
0 => "7626143928"
]
"server" => array:1 [
0 => "tsa_f"
]
"set-cookie" => array:1 [
0 => "guest_id=v1%3A167991873459084802; Max-Age=34214400; Expires=Fri, 26 Apr 2024 12:05:34 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None"
]
"content-type" => array:1 [
0 => "application/json; charset=utf-8"
]
"cache-control" => array:1 [
0 => "no-cache, no-store, max-age=0"
]
"content-length" => array:1 [
0 => "213"
]
"x-access-level" => array:1 [
0 => "read"
]
"x-frame-options" => array:1 [
0 => "SAMEORIGIN"
]
"x-transaction-id" => array:1 [
0 => "c12bee64dd78178f"
]
"x-xss-protection" => array:1 [
0 => "0"
]
"x-rate-limit-limit" => array:1 [
0 => "40000"
]
"x-rate-limit-reset" => array:1 [
0 => "1679919634"
]
"content-disposition" => array:1 [
0 => "attachment; filename=json.json"
]
"x-content-type-options" => array:1 [
0 => "nosniff"
]
"x-rate-limit-remaining" => array:1 [
0 => "39999"
]
"strict-transport-security" => array:1 [
0 => "max-age=631138519"
]
"x-response-time" => array:1 [
0 => "119"
]
"x-connection-hash" => array:1 [
0 => "58b90e5982e8edcc2f168c96018fcd8402b71aee4ab376bc85b09d912d816014"
]
"connection" => array:1 [
0 => "close"
]
]

best practice advice to run in prod

hi,

great package, now for production i plan to run it as an artisan command with supervisord. When rules are changed in the laravel app, is the a way to stop the stream ( and end the command ) so supervisord would restart it with the fresh rules?

Missed tweets from filteredStream

hi,

i have some problems where some tweets don't get fetched when using the FilteredStream api, i have no exceptions but it feels like a conenction / delay issue as also sometimes a whole chunk of feeds comes in but delayed. Using a elevated token this shouldn't happen, any hint how to debug this?

struggling to understand rules like t`o:`

Hi Felix,
I'm sorry but I am really struggling to understand the mechanics of Rules.

I am trying to catch only those tweets: to:myhandle
*I don't think I need any word to listen for as to is standalone.

I have a working command that listenes for a given string but when I change the rule from whenHears to whenTo (method below) so I only get tweets that are to a given handle ( using @myhandle at very start of the tweet ) the "to:myhandle" rule doesn't catch any tweets.

Should to: work as follows?...

public function whenTo(string | array $twitterUserIds, callable $whenTo): self
    {
        $this->rule->to($twitterUserIds);
        $this->onTweet = $whenTo;

        return $this;
    }

My startListening() method...

public function startListening(Sets $sets = null): void
    {
        // Delete old rules
        Rule::deleteBulk(...Rule::all());
        $this->rule->save();

        foreach ($this->stream->filteredTweets($sets) as $tweet) {
            if (is_null($tweet)) {
                continue;
            }

            call_user_func($this->onTweet, $tweet);
        }
    }

I call it from my command...

 TwitterStreamingApi::publicStream()
            ->whenTo($handle, function (array $tweet) {
                // log $tweet here;
            })
            ->startListening($sets);

would anyome have any pointers on helping me understand this part please?

RuntimeException('Unable to read from stream') followed by 429

In 2.2.1 from time-to-time I will get a Guzzle 7 RuntimeException('Unable to read from stream') error thrown then if I attempt to reconnect I immediately get a 429 from twitter with a ~900s timeout.

Is there any retry logic in this or the Guzzle that might be causing the 429 on twitter before the error get's thrown?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.