In light of changes to Twitter's API coming Feb 9

[robertoszek]

In light of changes to Twitter's API coming Feb 9

[edel79]

This one, very simple, is working, too : https://gitlab.com/jeancf/twoot
It is using random nitter instances to extract tweets.

[us3r1d]

After last week's API changes breaking nitter, I'm now using https://github.com/12joan/twitter-client to generate RSS for stork.

Just so you know stork is still working and still useful. :-)

It'd be nice if I could find some way to get profile updates happening while still getting the tweets from RSS; I'll post here again if I figure out a way to do that.

[edel79]

Hello, I'm using your script for some days and I do agree your statement.
I was wonderring about the support of the Twint python library (https://github.com/twintproject/twint), wich is capable to scrap Twitter content. Could be a good start to add this support.

[tomakun]

Saw that earlier, it sucks...

Just to confirm, if you get a paid access to the Twitter API, you theoretically still can use it as is right @robertoszek ? Providing you use a valid Twitter token of course.

[robertoszek]

Just to confirm, if you get a paid access to the Twitter API, you theoretically still can use it as is right @robertoszek ? Providing you use a valid Twitter token of course.

Potentially, yes.
Assuming they don't change the baseline API endpoints behavior or add additional steps to authenticate with a paid token, the bot would theoretically continue to work.

The thing is nobody really knows how's it going to change or be implemented.
We'll have to wait until the 9th and see once the dust settles what are our options going forward.

[edel79]

As a potential replacement, this scrapper seems good, to, and quite light : https://github.com/JustAnotherArchivist/snscrape
It's working great, today.

[robertoszek]

As a potential replacement, this scrapper seems good, to, and quite light : https://github.com/JustAnotherArchivist/snscrape It's working great, today.

It seems to use the unofficial GraphQL endpoint for scraping data:
https://github.com/JustAnotherArchivist/snscrape/blob/23ebdd2a3ce6c3e93012e2b5bc7c2b02c749aaf2/snscrape/modules/twitter.py#L1704

In addition to https://api.twitter.com/2/search/adaptive.json:
https://github.com/JustAnotherArchivist/snscrape/blob/23ebdd2a3ce6c3e93012e2b5bc7c2b02c749aaf2/snscrape/modules/twitter.py#L1549

We already use https://api.twitter.com/2/search/adaptive.json with guest tokens on the bot currently:

pleroma-bot/pleroma_bot/_twitter.py

Line 565 in 9a64891

"https://twitter.com/i/api/2/search/adaptive.json"

However the adaptive.json endpoint was severely limited recently (to only top results for non logged in users, removing any option to scrape by latest).

I'll look into how feasible would be to use the GraphQL endpoint for our own scraping too.

[edel79]

Using snscrape, I just did a request to get last 100 tweets for a specific Twitter user (@transportsidf), it worked well. So I don't know what are the limits, but if we can get at least 100 tweets at time, it seems enough for a bot, I think.
But, using Plroma in guest mode, gives me this error (same Twitter account) :

Gathering tweets... 0
✖ 2023-02-04 21:17:59,995 - pleroma_bot - ERROR - Unable to retrieve tweets. Is the account protected? If so, you need to provide the following OAuth 1.0a fields in the user config:

• consumer_key
• consumer_secret
• access_token_key
• access_token_secret (cli.py:645)

Should I use my API token and it's working fine. I don't know if I do something wrong or if it is a limitation/change in how guest mode works.

[nemobis]

I guess adding scraping capabilities to the bot has become a priority.

As a bridge solution, maybe pleroma-bot could scrape a Nitter instance? I'd be happy to set up a Nitter instance for my own pleroma-bot to scrape.

Then there's zedeus/nitter#389

[dawnerd]

Looks like it's finally here https://tapbots.social/@paul/110109551743991074

[dawnerd]

We just saw our access revoked overnight :/

[gigantuar]

Same here, it finally stopped working yesterday. I’ll need to start experimenting with using RSS via Nitter.

Edit: https://github.com/mahrtayyab/tweety looks like a great alternative to use instead of polling RSS.

[edel79]

My API key switched back to free plan so I can't extract tweets anymore, too.
As I previoulsy mentionned, snscrape is also still working to retrieve tweets.

[dawnerd]

I switched to using rsshub, tried nitter but that was very buggy. I think adopting the full graph endpoints would be the best path forward.

[edel79]

@robertoszek any chance of future developpments to handle the end of the free API using one of the above solutions ?

[dawnerd]

rsshub isn't perfect either, html ends up being embedded:

[Vardor]

I'm also having problems with twitter api. My bots are no longer working and I can't make it work with RSS source.
I've found a python scrapper for nitter called pnyter and I'm starting to explore it to see what I can do.
I've created a matrix channel in case anyone wants to join and exchange ideas #pletomabot:matrix.org https://matrix.to/#/!DmKYBjBcZXoeKlRmMU:matrix.org?via=matrix.org

[edel79]

Hello @AltGrCarlos the main problem here is that the creator of this bot is not active in the current time to make the necessary fixes. I would say 75% of the code is still working, and this bot is doing more than a simple scraper : it also updates the user profile, wich is great, and post tweets to mastodon.
So the part needing a fix is the scrape from Twitter part. Everything else can be kept as-is.
If you could create a fork with an upadated and fonctionnal scrapper, that would be great.

PS : I don't know about Matrix, in term of live chatting Discord must be more used.

[Vardor]

Hello @AltGrCarlos the main problem here is that the creator of this bot is not active in the current time to make the necessary fixes. I would say 75% of the code is still working, and this bot is doing more than a simple scraper : it also updates the user profile, wich is great, and post tweets to mastodon. So the part needing a fix is the scrape from Twitter part. Everything else can be kept as-is. If you could create a fork with an upadated and fonctionnal scrapper, that would be great.

PS : I don't know about Matrix, in term of live chatting Discord must be more used.

Hi. I'm not a really good programmer, but I'm trying to understand the code before to make any modification. I'm also trying to develop my own nitter scrapper in order to get the specific information i need from twitter.

[edel79]

Waiting for a fix to make Pleroma work again, I have set Twoot (previously mentionned) as replacement. It's working fine without API key.