holoarchivists / tslazer Goto Github PK
View Code? Open in Web Editor NEWSimple Twitter Space CLI downloader
Simple Twitter Space CLI downloader
Space ID is 1kvJpmLywvXxE
In the TwitterSpace class I'm failing in the condition if metadata != None on 219
The try fails so I receive the exception
Through debugging I've narrowed it down to subprocess.run(command, cwd=path)
in the else of the try
I looked what's being passed into the function:
command: ffmpeg -f concat -safe 0 -i chunkindex.txt -c copy -metadata title="AMA w/Kelsey Hightower - The Distinguished Gentleman" -metadata artist="escobyte" AMA_w_Kelsey_Hightower_The_Distinguished_Gentleman.m4a -loglevel fatal
path: /Users/{user}/Downloads/tslazer-master
Here's the traceback I got when moving the same command into the exception so it runs regardless. chunkindex.txt exists. In fact, I can run this same command manually in the terminal but it results in a file that is mix and mashed from the chunks concatenating out of order.
Traceback (most recent call last):
File "/Users/{user}/Downloads/tslazer-master/TwitterSpace.py", line 228, in downloadChunks
subprocess.run(command, cwd=path)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 505, in run
with Popen(*popenargs, **kwargs) as process:
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1823, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg -f concat -safe 0 -i chunkindex.txt -c copy -metadata title="AMA w/Kelsey Hightower - The Distinguished Gentleman" -metadata artist="escobyte" AMA_w_Kelsey_Hightower_The_Distinguished_Gentleman.m4a -loglevel fatal'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/{user}/Downloads/tslazer-master/./tslazer.py", line 39, in <module>
TwitterSpace.TwitterSpace(space_id=args.space_id, filenameformat=args.fileformat, path=args.path, withChat=args.withchat)
File "/Users/{user}/Downloads/tslazer-master/TwitterSpace.py", line 349, in __init__
TwitterSpace.downloadChunks(livechunks, self.filenameformat, self.path, m4aMetadata)
File "/Users/{user}/Downloads/tslazer-master/TwitterSpace.py", line 235, in downloadChunks
subprocess.run(command, cwd=path)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 505, in run
with Popen(*popenargs, **kwargs) as process:
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1823, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg -f concat -safe 0 -i chunkindex.txt -c copy -metadata title="AMA w/Kelsey Hightower - The Distinguished Gentleman" -metadata artist="escobyte" AMA_w_Kelsey_Hightower_The_Distinguished_Gentleman.m4a -loglevel fatal'
Okay so trying to debug a bit in subprocess.py self._execute_child seems to fail and it's being passed the arguments:
None None 0 False -1 -1 -1 -1 -1 -1 True None None None -1 False
Not really sure how to get anything useful from line 1823 here
It seems like the order of the audio chunks is out of sequence when playing the downloaded m4a file in various media players, including Audacity. It results in audio jumping around inconsistently and lacking coherence.
I used the following command to download the Twitter Space audio:
python3 tslazer.py --space_id 1lDxLnLkLXaGm --fileformat %St_%Ud --cookies=cookies.txt
.
The output file generated is Bitcoinizing_Honduras_with_AmityAge_Adopting_Bitcoin.m4a.
As I understand, Twitter Spaces sends the audio in separate chunks, which can sometimes arrive in a non-sequential manner. The script appears to be writing the chunks as they arrive, leading to the final file being out of order.
A potential solution could involve modifying the script to store the chunks in memory or temporary files, then write them out in the correct order once all chunks have been downloaded.
I appreciate any guidance or updates to address this issue. Thank you for your assistance in advance.
This is what I tried to solve with #4 but I noticed the cause is actually this:
When a long Space just ended, the playlist_xxx given in master_playlist sometimes won't actually be available (HTTP 404) for quite awhile.
Here are the logs from a monkey-patched version (based on V1) I have, just to demonstrate the issue:
D:\temp\tslazer>tslazer.py -s 1kvKpmXgMbMGE
[DEBUG] Update playlist URLs...
Space Found!
Space Title: 絵を描くスペース
Space Host Username: hoshiume0404
Space Host Display Name: 干し梅
Space Master URL:
https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/pxBnReI2LcTHLle-LVyb2qcazCAsPsAz1RjTGdm1WjIP3-dduSt965Bh7gp2JvjfTpheVRcPOFOaZeDnsKEXTw/transcode/ap-northeast-1/periscope-replay-direct-prod-ap-northeast-1-public/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsInZlcnNpb24iOiIyIn0.eyJIZWlnaHQiOjEwMCwiS2JwcyI6MzUwLCJUcmFuc2NvZGVBdWRpbyI6dHJ1ZSwiV2lkdGgiOjEwMH0.xKaxg744Z6K1TwhoGXRWJ3A3FlSeNBKxxUEyorWq8qY/audio-space/playlist_16760973110719389214.m3u8
Space Dynamic URL:
https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/pxBnReI2LcTHLle-LVyb2qcazCAsPsAz1RjTGdm1WjIP3-dduSt965Bh7gp2JvjfTpheVRcPOFOaZeDnsKEXTw/non_transcode/ap-northeast-1/periscope-replay-direct-prod-ap-northeast-1-public/audio-space/master_playlist.m3u8
Chat Token:
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2ODU4NTk3NjQsImFtYmlndW91c191c2VyX2lkIjoiZXlKVWQybDBkR1Z5VlhObGNrbGtJam94TmpnMU56Y3pNelkwTmpNeE9EWXlOelo5IiwiYnJvYWRjYXN0X2lkIjoiMWt2S3BtWGdNYk1HRSIsImxvd19sYXRlbmN5Ijp0cnVlLCJyZWFkX29ubHkiOmZhbHNlLCJwYXJ0aWNpcGFudF9pbmRleCI6NjgzOTg0ODQwfQ.fCzL8M6_X_1C9IeUV4jaKr9BHqpcoFSzWPbMo0PfElc
Downloading to 230603 @hoshiume0404 絵を描くスペース-twitter-space-1kvKpmXgMbMGE.m4a
Waiting for space to end...
Space Ended. Wait 1 minute for the recording to be processed.
[DEBUG] current master_url filename: playlist_16760973110719389214.m3u8
[DEBUG] Update playlist URLs...
[DEBUG] current master_url filename: playlist_16760972807730874215.m3u8
[DEBUG] current master_url:
https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/pxBnReI2LcTHLle-LVyb2qcazCAsPsAz1RjTGdm1WjIP3-dduSt965Bh7gp2JvjfTpheVRcPOFOaZeDnsKEXTw/transcode/ap-northeast-1/periscope-replay-direct-prod-ap-northeast-1-public/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsInZlcnNpb24iOiIyIn0.eyJIZWlnaHQiOjEwMCwiS2JwcyI6MzUwLCJUcmFuc2NvZGVBdWRpbyI6dHJ1ZSwiV2lkdGgiOjEwMH0.xKaxg744Z6K1TwhoGXRWJ3A3FlSeNBKxxUEyorWq8qY/audio-space/playlist_16760972807730874215.m3u8
[DEBUG] request status code: 404
[DEBUG] failed to get playlist m3u8, retry after 10 seconds...
[DEBUG] Update playlist URLs...
[DEBUG] current master_url filename: playlist_16760970710781794128.m3u8
[DEBUG] current master_url:
https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/pxBnReI2LcTHLle-LVyb2qcazCAsPsAz1RjTGdm1WjIP3-dduSt965Bh7gp2JvjfTpheVRcPOFOaZeDnsKEXTw/transcode/ap-northeast-1/periscope-replay-direct-prod-ap-northeast-1-public/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsInZlcnNpb24iOiIyIn0.eyJIZWlnaHQiOjEwMCwiS2JwcyI6MzUwLCJUcmFuc2NvZGVBdWRpbyI6dHJ1ZSwiV2lkdGgiOjEwMH0.xKaxg744Z6K1TwhoGXRWJ3A3FlSeNBKxxUEyorWq8qY/audio-space/playlist_16760970710781794128.m3u8
[DEBUG] request status code: 404
[DEBUG] failed to get playlist m3u8, retry after 10 seconds...
[DEBUG] Update playlist URLs...
[DEBUG] current master_url filename: playlist_16760971421756524434.m3u8
[DEBUG] current master_url:
https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/pxBnReI2LcTHLle-LVyb2qcazCAsPsAz1RjTGdm1WjIP3-dduSt965Bh7gp2JvjfTpheVRcPOFOaZeDnsKEXTw/transcode/ap-northeast-1/periscope-replay-direct-prod-ap-northeast-1-public/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsInZlcnNpb24iOiIyIn0.eyJIZWlnaHQiOjEwMCwiS2JwcyI6MzUwLCJUcmFuc2NvZGVBdWRpbyI6dHJ1ZSwiV2lkdGgiOjEwMH0.xKaxg744Z6K1TwhoGXRWJ3A3FlSeNBKxxUEyorWq8qY/audio-space/playlist_16760971421756524434.m3u8
[DEBUG] request status code: 404
[DEBUG] failed to get playlist m3u8, retry after 10 seconds...
[DEBUG] Update playlist URLs...
[DEBUG] current master_url filename: playlist_16760968411484182689.m3u8
[DEBUG] current master_url:
https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/pxBnReI2LcTHLle-LVyb2qcazCAsPsAz1RjTGdm1WjIP3-dduSt965Bh7gp2JvjfTpheVRcPOFOaZeDnsKEXTw/transcode/ap-northeast-1/periscope-replay-direct-prod-ap-northeast-1-public/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsInZlcnNpb24iOiIyIn0.eyJIZWlnaHQiOjEwMCwiS2JwcyI6MzUwLCJUcmFuc2NvZGVBdWRpbyI6dHJ1ZSwiV2lkdGgiOjEwMH0.xKaxg744Z6K1TwhoGXRWJ3A3FlSeNBKxxUEyorWq8qY/audio-space/playlist_16760968411484182689.m3u8
[DEBUG] request status code: 200
[DEBUG] get 5225 chunks.
5225/5225 chunks downloaded.
Finished Downloading Chunks.
[DEBUG] command is ffmpeg -loglevel error -stats -i "230603 @hoshiume0404 絵を描くスペース-twitter-space-1kvKpmXgMbMGE_merged.aac" -c copy -metadata title="絵を描くスペース" -metadata artist="hoshiume0404" "230603 @hoshiume0404 絵を描くスペース-twitter-space-1kvKpmXgMbMGE.m4a"
size= 181885kB time=04:21:16.35 bitrate= 95.0kbits/s speed=2.55e+03x
Successfully Downloaded Twitter Space 230603 @hoshiume0404 絵を描くスペース-twitter-space-1kvKpmXgMbMGE.m4a
I checked the twitter-space-overhaul
branch, and I think the issue would be the same there.
I don't have time to write a proper PR for overhaul version at the moment, but I think you get the idea about the nature of this bug. Maybe you or someone else can write a patch for this.
Hello,
I've been trying to use the tslazer program to download a Twitter Space using the following command:
python3 tslazer.py --space_id 1ynJOaBDloEKR --fileformat %St_%Ud
However, I'm encountering an issue where I receive a json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
error. This error suggests that the program is receiving an empty response when trying to fetch metadata for the Twitter Space.
Here is the full traceback for reference:
Traceback (most recent call last):
File "/home/kasita/Downloads/tslazer/tslazer.py", line 39, in <module>
TwitterSpace.TwitterSpace(space_id=args.space_id, filenameformat=args.fileformat, path=args.path, withChat=args.withchat)
File "/home/kasita/Downloads/tslazer/TwitterSpace.py", line 251, in init
self.metadata = TwitterSpace.getMetadata(self.space_id, guest_token)
File "/home/kasita/Downloads/tslazer/TwitterSpace.py", line 128, in getMetadata
metadataResponse = metadataRequest.json()
File "/usr/lib/python3/dist-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.10/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Is it possible to run the tslazer in Windows?
Traceback (most recent call last):
File "C:\Users\USER\Desktop\twitterspace\tslazer\tslazer.py", line 43, in <module>
TwitterSpace.TwitterSpace(space_id=args.space_id, filenameformat=args.fileformat, path=args.path, withChat=args.withchat, cookiesPath=args.cookies)
File "C:\Users\USER\Desktop\twitterspace\tslazer\TwitterSpace.py", line 524, in __init__
self.filenameformat = slugify(self.filenameformat, allow_unicode=True, lowercase=False, separator='_')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: slugify() got an unexpected keyword argument 'allow_unicode'
When the Space is finished, nothing is downloaded. The setting is ' --path $HOME/tslazer_download --withchat --fileformat %Ud_%Un_%Ui_%St_%Si_%Dy_%Dm_%Dt' and there is no tmp folder existing during the space downloading.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.