holoarchivists/tslazer

Simple Twitter Space CLI downloader

tslazer's Issues

Twitter's new API changes causes tool to malfunction; JSONDecodeError When Attempting to Download a Twitter Space

Hello,

I've been trying to use the tslazer program to download a Twitter Space using the following command:

python3 tslazer.py --space_id 1ynJOaBDloEKR --fileformat %St_%Ud

However, I'm encountering an issue where I receive a json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) error. This error suggests that the program is receiving an empty response when trying to fetch metadata for the Twitter Space.

Here is the full traceback for reference:

Traceback (most recent call last):
File "/home/kasita/Downloads/tslazer/tslazer.py", line 39, in <module>
TwitterSpace.TwitterSpace(space_id=args.space_id, filenameformat=args.fileformat, path=args.path, withChat=args.withchat)
File "/home/kasita/Downloads/tslazer/TwitterSpace.py", line 251, in init
self.metadata = TwitterSpace.getMetadata(self.space_id, guest_token)
File "/home/kasita/Downloads/tslazer/TwitterSpace.py", line 128, in getMetadata
metadataResponse = metadataRequest.json()
File "/usr/lib/python3/dist-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.10/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

No such file or directory

Space ID is 1kvJpmLywvXxE
In the TwitterSpace class I'm failing in the condition if metadata != None on 219
The try fails so I receive the exception
Through debugging I've narrowed it down to subprocess.run(command, cwd=path) in the else of the try
I looked what's being passed into the function:

command: ffmpeg -f concat -safe 0 -i chunkindex.txt -c copy -metadata title="AMA w/Kelsey Hightower - The Distinguished Gentleman" -metadata artist="escobyte" AMA_w_Kelsey_Hightower_The_Distinguished_Gentleman.m4a -loglevel fatal

path: /Users/{user}/Downloads/tslazer-master

Here's the traceback I got when moving the same command into the exception so it runs regardless. chunkindex.txt exists. In fact, I can run this same command manually in the terminal but it results in a file that is mix and mashed from the chunks concatenating out of order.

Traceback (most recent call last):
  File "/Users/{user}/Downloads/tslazer-master/TwitterSpace.py", line 228, in downloadChunks
    subprocess.run(command, cwd=path)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1823, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg -f concat -safe 0 -i chunkindex.txt -c copy -metadata title="AMA w/Kelsey Hightower - The Distinguished Gentleman" -metadata artist="escobyte" AMA_w_Kelsey_Hightower_The_Distinguished_Gentleman.m4a -loglevel fatal'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/{user}/Downloads/tslazer-master/./tslazer.py", line 39, in <module>
    TwitterSpace.TwitterSpace(space_id=args.space_id, filenameformat=args.fileformat, path=args.path, withChat=args.withchat)
  File "/Users/{user}/Downloads/tslazer-master/TwitterSpace.py", line 349, in __init__
    TwitterSpace.downloadChunks(livechunks, self.filenameformat, self.path, m4aMetadata)
  File "/Users/{user}/Downloads/tslazer-master/TwitterSpace.py", line 235, in downloadChunks
    subprocess.run(command, cwd=path)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1823, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg -f concat -safe 0 -i chunkindex.txt -c copy -metadata title="AMA w/Kelsey Hightower - The Distinguished Gentleman" -metadata artist="escobyte" AMA_w_Kelsey_Hightower_The_Distinguished_Gentleman.m4a -loglevel fatal'

Okay so trying to debug a bit in subprocess.py self._execute_child seems to fail and it's being passed the arguments:
None None 0 False -1 -1 -1 -1 -1 -1 True None None None -1 False

Not really sure how to get anything useful from line 1823 here

Twitter Space audio does not download if in a linux environment

When the Space is finished, nothing is downloaded. The setting is ' --path $HOME/tslazer_download --withchat --fileformat %Ud_%Un_%Ui_%St_%Si_%Dy_%Dm_%Dt' and there is no tmp folder existing during the space downloading.

unexpected keyword argument 'allow_unicode'

Is it possible to run the tslazer in Windows?

Traceback (most recent call last):
  File "C:\Users\USER\Desktop\twitterspace\tslazer\tslazer.py", line 43, in <module>
    TwitterSpace.TwitterSpace(space_id=args.space_id, filenameformat=args.fileformat, path=args.path, withChat=args.withchat, cookiesPath=args.cookies)
  File "C:\Users\USER\Desktop\twitterspace\tslazer\TwitterSpace.py", line 524, in __init__
    self.filenameformat = slugify(self.filenameformat, allow_unicode=True, lowercase=False, separator='_')
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: slugify() got an unexpected keyword argument 'allow_unicode'

Audio chunks in downloaded m4a file are out of order

It seems like the order of the audio chunks is out of sequence when playing the downloaded m4a file in various media players, including Audacity. It results in audio jumping around inconsistently and lacking coherence.

I used the following command to download the Twitter Space audio:
python3 tslazer.py --space_id 1lDxLnLkLXaGm --fileformat %St_%Ud --cookies=cookies.txt.

The output file generated is Bitcoinizing_Honduras_with_AmityAge_Adopting_Bitcoin.m4a.

As I understand, Twitter Spaces sends the audio in separate chunks, which can sometimes arrive in a non-sequential manner. The script appears to be writing the chunks as they arrive, leading to the final file being out of order.

A potential solution could involve modifying the script to store the chunks in memory or temporary files, then write them out in the correct order once all chunks have been downloaded.

I appreciate any guidance or updates to address this issue. Thank you for your assistance in advance.

playlist m3u8 sometimes isn't immediately available (HTTP 404) after a long space just ended

This is what I tried to solve with #4 but I noticed the cause is actually this:

When a long Space just ended, the playlist_xxx given in master_playlist sometimes won't actually be available (HTTP 404) for quite awhile.

Here are the logs from a monkey-patched version (based on V1) I have, just to demonstrate the issue:

D:\temp\tslazer>tslazer.py -s 1kvKpmXgMbMGE
[DEBUG] Update playlist URLs...
Space Found!
Space Title: 絵を描くスペース
Space Host Username: hoshiume0404
Space Host Display Name: 干し梅
Space Master URL:
https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/pxBnReI2LcTHLle-LVyb2qcazCAsPsAz1RjTGdm1WjIP3-dduSt965Bh7gp2JvjfTpheVRcPOFOaZeDnsKEXTw/transcode/ap-northeast-1/periscope-replay-direct-prod-ap-northeast-1-public/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsInZlcnNpb24iOiIyIn0.eyJIZWlnaHQiOjEwMCwiS2JwcyI6MzUwLCJUcmFuc2NvZGVBdWRpbyI6dHJ1ZSwiV2lkdGgiOjEwMH0.xKaxg744Z6K1TwhoGXRWJ3A3FlSeNBKxxUEyorWq8qY/audio-space/playlist_16760973110719389214.m3u8
Space Dynamic URL:
https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/pxBnReI2LcTHLle-LVyb2qcazCAsPsAz1RjTGdm1WjIP3-dduSt965Bh7gp2JvjfTpheVRcPOFOaZeDnsKEXTw/non_transcode/ap-northeast-1/periscope-replay-direct-prod-ap-northeast-1-public/audio-space/master_playlist.m3u8
Chat Token:
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2ODU4NTk3NjQsImFtYmlndW91c191c2VyX2lkIjoiZXlKVWQybDBkR1Z5VlhObGNrbGtJam94TmpnMU56Y3pNelkwTmpNeE9EWXlOelo5IiwiYnJvYWRjYXN0X2lkIjoiMWt2S3BtWGdNYk1HRSIsImxvd19sYXRlbmN5Ijp0cnVlLCJyZWFkX29ubHkiOmZhbHNlLCJwYXJ0aWNpcGFudF9pbmRleCI6NjgzOTg0ODQwfQ.fCzL8M6_X_1C9IeUV4jaKr9BHqpcoFSzWPbMo0PfElc
Downloading to 230603 @hoshiume0404 絵を描くスペース-twitter-space-1kvKpmXgMbMGE.m4a
Waiting for space to end...
Space Ended. Wait 1 minute for the recording to be processed.
[DEBUG] current master_url filename: playlist_16760973110719389214.m3u8
[DEBUG] Update playlist URLs...
[DEBUG] current master_url filename: playlist_16760972807730874215.m3u8
[DEBUG] current master_url:
https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/pxBnReI2LcTHLle-LVyb2qcazCAsPsAz1RjTGdm1WjIP3-dduSt965Bh7gp2JvjfTpheVRcPOFOaZeDnsKEXTw/transcode/ap-northeast-1/periscope-replay-direct-prod-ap-northeast-1-public/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsInZlcnNpb24iOiIyIn0.eyJIZWlnaHQiOjEwMCwiS2JwcyI6MzUwLCJUcmFuc2NvZGVBdWRpbyI6dHJ1ZSwiV2lkdGgiOjEwMH0.xKaxg744Z6K1TwhoGXRWJ3A3FlSeNBKxxUEyorWq8qY/audio-space/playlist_16760972807730874215.m3u8
[DEBUG] request status code: 404
[DEBUG] failed to get playlist m3u8, retry after 10 seconds...
[DEBUG] Update playlist URLs...
[DEBUG] current master_url filename: playlist_16760970710781794128.m3u8
[DEBUG] current master_url:
https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/pxBnReI2LcTHLle-LVyb2qcazCAsPsAz1RjTGdm1WjIP3-dduSt965Bh7gp2JvjfTpheVRcPOFOaZeDnsKEXTw/transcode/ap-northeast-1/periscope-replay-direct-prod-ap-northeast-1-public/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsInZlcnNpb24iOiIyIn0.eyJIZWlnaHQiOjEwMCwiS2JwcyI6MzUwLCJUcmFuc2NvZGVBdWRpbyI6dHJ1ZSwiV2lkdGgiOjEwMH0.xKaxg744Z6K1TwhoGXRWJ3A3FlSeNBKxxUEyorWq8qY/audio-space/playlist_16760970710781794128.m3u8
[DEBUG] request status code: 404
[DEBUG] failed to get playlist m3u8, retry after 10 seconds...
[DEBUG] Update playlist URLs...
[DEBUG] current master_url filename: playlist_16760971421756524434.m3u8
[DEBUG] current master_url:
https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/pxBnReI2LcTHLle-LVyb2qcazCAsPsAz1RjTGdm1WjIP3-dduSt965Bh7gp2JvjfTpheVRcPOFOaZeDnsKEXTw/transcode/ap-northeast-1/periscope-replay-direct-prod-ap-northeast-1-public/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsInZlcnNpb24iOiIyIn0.eyJIZWlnaHQiOjEwMCwiS2JwcyI6MzUwLCJUcmFuc2NvZGVBdWRpbyI6dHJ1ZSwiV2lkdGgiOjEwMH0.xKaxg744Z6K1TwhoGXRWJ3A3FlSeNBKxxUEyorWq8qY/audio-space/playlist_16760971421756524434.m3u8
[DEBUG] request status code: 404
[DEBUG] failed to get playlist m3u8, retry after 10 seconds...
[DEBUG] Update playlist URLs...
[DEBUG] current master_url filename: playlist_16760968411484182689.m3u8
[DEBUG] current master_url:
https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/pxBnReI2LcTHLle-LVyb2qcazCAsPsAz1RjTGdm1WjIP3-dduSt965Bh7gp2JvjfTpheVRcPOFOaZeDnsKEXTw/transcode/ap-northeast-1/periscope-replay-direct-prod-ap-northeast-1-public/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsInZlcnNpb24iOiIyIn0.eyJIZWlnaHQiOjEwMCwiS2JwcyI6MzUwLCJUcmFuc2NvZGVBdWRpbyI6dHJ1ZSwiV2lkdGgiOjEwMH0.xKaxg744Z6K1TwhoGXRWJ3A3FlSeNBKxxUEyorWq8qY/audio-space/playlist_16760968411484182689.m3u8
[DEBUG] request status code: 200
[DEBUG] get 5225 chunks.
5225/5225 chunks downloaded.
Finished Downloading Chunks.
[DEBUG] command is ffmpeg -loglevel error -stats -i "230603 @hoshiume0404 絵を描くスペース-twitter-space-1kvKpmXgMbMGE_merged.aac" -c copy -metadata title="絵を描くスペース" -metadata artist="hoshiume0404" "230603 @hoshiume0404 絵を描くスペース-twitter-space-1kvKpmXgMbMGE.m4a"
size=  181885kB time=04:21:16.35 bitrate=  95.0kbits/s speed=2.55e+03x
Successfully Downloaded Twitter Space 230603 @hoshiume0404 絵を描くスペース-twitter-space-1kvKpmXgMbMGE.m4a

I checked the twitter-space-overhaul branch, and I think the issue would be the same there.

I don't have time to write a proper PR for overhaul version at the moment, but I think you get the idea about the nature of this bug. Maybe you or someone else can write a patch for this.