Add support for GoToSocial

[ynakao]

Add support for GoToSocial

[robertoszek]

Hi!

I'll look into it by finding a GoToSocial instance where I can test the bot with, I've been following the development on and off and waiting for the beta before trying it out.

So far, I see that these endpoints are the ones missing that could cause issues with the bot:

/api/v1/statuses/:id/pin POST (Pin a status to profile)
/api/v1/statuses/:id/unpin POST (Unpin a status from profile)

(and /api/v1/statuses POST (Create a new status) is missing the poll parameter)

Regarding the first config you shared:

pleroma_base_url: https://gts.example.com
software: mastodon
users:
- twitter_username: nakaoyuji
  # I got an account ID by following the document.
  # curl 'https://gts.example.com/api/v2/search?q=<username>&resolve=true&limit=5' -H 'Authorization: Bearer xxxxx'
  # ID on GoTosocial seems to be 26 alphanumeric charactors.
  pleroma_username: 02G7NUT8E2RBPBPZS4OXQ89T1T
  pleroma_token: xxxxx
  archive: /path/to/twitter-archive.zip

I'm curious about the error, perhaps it is unrelated to the target instance being GoToSocial.

Could you set up a virtual environment, install 1.2.1rc9 on it and run the config with it?:

$ python3 -m venv testgts
$ source testgts/bin/activate
(testgts) $ pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.2.1rc9
(testgts) $ pleroma-bot --config /path/to/config

I'm curious if it would fail in some other unexpected way.

[ynakao]

Thank you for your reply.

I installed v1.2.1rc9 and ran pleroma-bot as you described. Now the error message changed to AttributeError: 'User' object has no attribute 'pinned_tweet_id'. So, is this because GoToSocial hasn't implement relevant API yet?

2023-01-23 20:40:15,025 pleroma_bot ERROR: Exception occurred for user, skipping...
Traceback (most recent call last):
  File "/home/ynakao/test/pleroma-bot/pleroma-bot-v1.2.1-rc9/lib/python3.10/site-packages/pleroma_bot/cli.py", line 709, in main
    user.check_pinned(posted)
  File "/home/ynakao/test/pleroma-bot/pleroma-bot-v1.2.1-rc9/lib/python3.10/site-packages/pleroma_bot/_utils.py", line 340, in check_pinned
    ).format(str(self.pinned_tweet_id)))
AttributeError: 'User' object has no attribute 'pinned_tweet_id'

[robertoszek]

So, is this because GoToSocial hasn't implement relevant API yet?

Most likely, perhaps we can simply skip the pinning functionality completely if the target is GoToSocial for now.

Any luck with 1.2.1rc10?:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.2.1rc10

[ynakao]

Yes! It looks working well! I didn't check whether all imports are properly done though, since it will take over 14 hours to finish.

[robertoszek]

@ynakao did you run into any new issues during your import?

I spun up a local GoToSocial instance to test with and I found that the response code for attachments being too big is different than on Mastodon/Pleroma (HTTP 400 instead of 413).
So I made some changes to take that into account on 1.2.1rc16

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.2.1rc16

Everything else seemed to work as I expected.
(except pinning posts and polls, which are not implemented in GoToSocial yet, so we skip them)

• Posting images:

• Posting videos:

• Updating the profile:

I'm going to keep trying to find edgecases but let me know if you ran into anything else weird.

[ynakao]

I encountered the following TypeError on a newly created GoToSocial test account. Is there a way to spot which tweet causes this issue easily?

(pleroma-bot-v1.2.1-rc16) [hetzarch@hetzarch][~/test/pleroma-bot/pleroma-bot-v1.2.1-rc16]% ./bin/pleroma-bot --archive twitter-archive-2022-12-27.zip
...
⚠ 2023-01-28 13:36:58,595 - pleroma_bot - WARNING - Raising max_tweets to the maximum allowed value (_utils.py:610)
ℹ 2023-01-28 13:37:05,061 - pleroma_bot - INFO - tweets gathered:        23497
Processing tweets... : 100%|█████████████████████████████████████████████████████████████| 23497/23497 [04:53<00:00, 80.19it/s]
ℹ 2023-01-28 13:41:58,113 - pleroma_bot - INFO - tweets to post:         23497
Posting tweets... :  82%|██████████████████████████████████████████████████▋           | 19205/23497 [2:54:53<39:28,  1.81it/s]✖ 2023-01-28 16:36:52,759 - pleroma_bot - ERROR - Exception occurred for user, skipping... (cli.py:721)
Traceback (most recent call last):
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc16/lib/python3.10/site-packages/pleroma_bot/cli.py", line 688, in main
    post_id = user.post(
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc16/lib/python3.10/site-packages/pleroma_bot/_utils.py", line 794, in post
    post_id = self.post_pleroma(tweet, poll, sensitive, media, cw=cw)
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc16/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 310, in post_pleroma
    media_id = json.loads(response.text)["id"]
TypeError: 'NoneType' object is not subscriptable

[robertoszek]

It looks like it failed to upload an attachment.
I've added a handler for the error that should display the Tweet ID, the response from the GoToSocial instance and which file was trying to upload on 1.2.1rc18:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.2.1rc18

Hopefully that would give us enough info to identify what's the issue there.

[ynakao]

It seems error occurs when uploading mp4 files. I guess this is because video support in GoToSocial will be landed on v0.7.0 and I'm running the latest release v0.6.0. Are you running GoToSocial latest git commit version as you can post videos in your test?

Except for this, all operation worked properly in my case.

Error log

(pleroma-bot-v1.2.1-rc18) [hetzarch@hetzarch][~/test/pleroma-bot/pleroma-bot-v1.2.1-rc18]% ./bin/pleroma-bot --archive twitter-archive-2022-12-27.zip
...
⚠ 2023-01-29 09:59:48,337 - pleroma_bot - WARNING - Raising max_tweets to the maximum allowed value (_utils.py:612)
ℹ 2023-01-29 09:59:54,767 - pleroma_bot - INFO - tweets gathered:        23497
Processing tweets... : 100%|█████████████████████████████████████████████████████████████| 23497/23497 [04:37<00:00, 84.69it/s]
ℹ 2023-01-29 10:04:32,289 - pleroma_bot - INFO - tweets to post:         23497
Posting tweets... :  82%|██████████████████████████████████████████████████▋           | 19205/23497 [2:54:55<39:33,  1.81it/s]✖ 2023-01-29 12:59:28,928 - pleroma_bot - ERROR - Error uploading media:
Status code:    200
Tweet ID:       773124013677654016
Filename:       /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/773124013677654016/773124013677654016-PTZmtVxlXHRDGMcg.mp4
Size:   0.09MB
Response:       null (_pleroma.py:317)
✖ 2023-01-29 12:59:28,929 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333)
Traceback (most recent call last):
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma
    media_id = json.loads(response.text)["id"]
TypeError: 'NoneType' object is not subscriptable
Posting tweets... :  82%|██████████████████████████████████████████████████▋           | 19209/23497 [2:54:58<41:13,  1.73it/s]✖ 2023-01-29 12:59:31,233 - pleroma_bot - ERROR - Error uploading media:
Status code:    200
Tweet ID:       773142132597125120
Filename:       /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/773142132597125120/773142132597125120-CrrAqk5VYAAWl88.mp4
Size:   0.1MB
Response:       null (_pleroma.py:317)
✖ 2023-01-29 12:59:31,234 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333)
Traceback (most recent call last):
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma
    media_id = json.loads(response.text)["id"]
TypeError: 'NoneType' object is not subscriptable
Posting tweets... :  84%|███████████████████████████████████████████████████▉          | 19699/23497 [2:59:30<36:35,  1.73it/s]✖ 2023-01-29 13:04:03,915 - pleroma_bot - ERROR - Error uploading media:
Status code:    200
Tweet ID:       811535032577732609
Filename:       /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/811535032577732609/811535032577732609-5xBU-2VYTfLvI_FV.mp4
Size:   0.54MB
Response:       null (_pleroma.py:317)
✖ 2023-01-29 13:04:03,921 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333)
Traceback (most recent call last):
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma
    media_id = json.loads(response.text)["id"]
TypeError: 'NoneType' object is not subscriptable
Posting tweets... :  84%|███████████████████████████████████████████████████▉          | 19701/23497 [2:59:31<36:10,  1.75it/s]✖ 2023-01-29 13:04:05,060 - pleroma_bot - ERROR - Error uploading media:
Status code:    200
Tweet ID:       811538743119052800
Filename:       /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/811538743119052800/811538743119052800-C0MqLTtVQAAwNfx.mp4
Size:   0.22MB
Response:       null (_pleroma.py:317)
✖ 2023-01-29 13:04:05,061 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333)
Traceback (most recent call last):
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma
    media_id = json.loads(response.text)["id"]
TypeError: 'NoneType' object is not subscriptable
Posting tweets... :  94%|██████████████████████████████████████████████████████████▍   | 22140/23497 [3:22:16<14:28,  1.56it/s]✖ 2023-01-29 13:26:49,244 - pleroma_bot - ERROR - Error uploading media:
Status code:    200
Tweet ID:       1114169013477629952
Filename:       /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/1114169013477629952/1114169013477629952-D3ZSzfbU0AAxZg3.mp4
Size:   0.12MB
Response:       null (_pleroma.py:317)
✖ 2023-01-29 13:26:49,245 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333)
Traceback (most recent call last):
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma
    media_id = json.loads(response.text)["id"]
TypeError: 'NoneType' object is not subscriptable
Posting tweets... :  99%|█████████████████████████████████████████████████████████████▎| 23253/23497 [3:32:43<02:33,  1.59it/s]✖ 2023-01-29 13:37:16,749 - pleroma_bot - ERROR - Error uploading media:
Status code:    200
Tweet ID:       1505537146530430977
Filename:       /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/1505537146530430977/1505537146530430977-FOS96kZaMAEFNmy.mp4
Size:   0.11MB
Response:       null (_pleroma.py:317)
✖ 2023-01-29 13:37:16,749 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333)
Traceback (most recent call last):
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma
    media_id = json.loads(response.text)["id"]
TypeError: 'NoneType' object is not subscriptable
Posting tweets... : 100%|██████████████████████████████████████████████████████████████| 23497/23497 [3:35:02<00:00,  1.82it/s]
ℹ 2023-01-29 13:39:34,789 - pleroma_bot - INFO - Updating profile:       <Response [200]>
(pleroma-bot-v1.2.1-rc18) [hetzarch@hetzarch][~/test/pleroma-bot/pleroma-bot-v1.2.1-rc18]%

[robertoszek]

Are you running GoToSocial latest git commit version as you can post videos in your test?

Yep, I just pulled the latest snapshot:
https://hub.docker.com/r/superseriousbusiness/gotosocial/tags

https://docs.gotosocial.org/en/latest/installation_guide/docker/#version