[twitter] x.com links give Unsupported URL error

[InklingGirl]

[twitter] x.com links give Unsupported URL error

[hsn10]

its high priority bugfix because x.com is very popular site, probably second only to youtube

new yt-dlp version should be published.

[bashonly]

Internally, the site still uses twitter domains. All we need to do is add x.com to the _VALID_URLs

[jshir]

I installed the nightly and public videos work fine, however, I cannot get it to download a protected video by passing a cookies.txt file from x.com (this is after a very recent change to their login system). I bet I'm not the only one affected and I don't want to give out someone's private link so here is just the error message without the verbose. Let me know if a verbose is truly necessary and I would have to set up a dummy protected acct to get one.

ERROR: [twitter] _________________: You are not authorized to view this protected tweet. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials

[bashonly]

@InklingGirl already has: python -m pip install -U --pre "yt-dlp[default]"

[bashonly]

@ggous

you need to have updated to nightly

yt-dlp --update-to nightly

If it still doesn't work, then show a verbose log: add -v to your download command and copy/paste the full output

[InklingGirl]

& it seems to be broken again.

[joe-oli]

I confirm yt-dlp no longer works on X/Twitter; just started happening very recently (approximately 14/May/2024), which is most likely related to the url for X is now x.com instead of twitter.com

[InklingGirl]

Will this update be pushed to pip soon?

[ggous]

I have updated : sudo yt-dlp -U

and I am receiving:

ERROR: Unsupported URL: https://x.com/i/status/1792192732015800675

[ggous]

@ggous

you need to have updated to nightly

yt-dlp --update-to nightly

If it still doesn't work, then show a verbose log: add -v to your download command and copy/paste the full output

Thanks! It works now!

[Grub4K]

its high priority bugfix because x.com is very popular site, probably second only to youtube

new yt-dlp version should be published.

Use yt-dlp --update-to nightly, download the nightly build manually or use pip install --pre "yt-dlp[default]".

[InklingGirl]

I installed the nightly and public videos work fine, however, I cannot get it to download a protected video by passing a cookies.txt file from x.com (this is after a very recent change to their login system). I bet I'm not the only one affected and I don't want to give out someone's private link so here is just the error message without the verbose. Let me know if a verbose is truly necessary and I would have to set up a dummy protected acct to get one.

ERROR: [twitter] _________________: You are not authorized to view this protected tweet. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials

same here @jshir

[bashonly]

show verbose log please @InklingGirl

[InklingGirl]

show verbose log please @InklingGirl

@bashonly this happens w/ any video that previously required & worked w/ cookies:

[debug] Command-line config: ['--cookies', 'Documents/cookies.txt', '-vU', '<MANUALLY_REDACTED_URL>']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version [email protected] from yt-dlp/yt-dlp-nightly-builds [6d8a53d87] (pip)
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-6.5.0-35-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.47.0, pyxattr-0.7.2, requests-2.31.0, secretstorage-3.3.1, sqlite3-3.37.2, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1803 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
[debug] Downloading _update_spec from https://github.com/yt-dlp/yt-dlp-nightly-builds/releases/latest/download/_update_spec
Current version: [email protected] from yt-dlp/yt-dlp-nightly-builds
Latest version: [email protected] from yt-dlp/yt-dlp-nightly-builds
ERROR: You installed yt-dlp with pip or using the wheel from PyPi; Use that to update
[twitter] Extracting URL: <MANUALLY_REDACTED_URL>
[twitter] <MANUALLY_REDACTED_URL>: Downloading guest token
[twitter] <MANUALLY_REDACTED_URL>: Downloading GraphQL JSON
ERROR: [twitter] <MANUALLY_REDACTED_URL>: NSFW tweet requires authentication. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 734, in extract
    ie_result = self._real_extract(url)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1428, in _real_extract
    status = self._extract_status(twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1406, in _extract_status
    status = self._graphql_to_legacy(self._call_graphql_api(self._GRAPHQL_ENDPOINT, twid), twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1283, in _graphql_to_legacy
    self.raise_login_required('NSFW tweet requires authentication')
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 1233, in raise_login_required
    raise ExtractorError(msg, expected=True)

[bashonly]

@InklingGirl update. Your (old) nightly version does not have 3e35aa3, which fixed that issue

python3 -m pip install --pre -U "yt-dlp[default]"

[InklingGirl]

I upgraded to it & get the exact same result.

EDIT: I generated a new cookies file & that solved the issue. The old cookies must have been for twitter.com instead of x.com & that was what caused the problem.

[bashonly]

@InklingGirl please share a verbose log

[InklingGirl]

[debug] Command-line config: ['--cookies', '/home/christian/Documents/cookies.txt', '-vU', '<MANUALLY_REDACTED_URL>']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version [email protected] from yt-dlp/yt-dlp-nightly-builds [111b61dde] (pip)
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-6.5.0-35-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.47.0, pyxattr-0.7.2, requests-2.31.0, secretstorage-3.3.1, sqlite3-3.37.2, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1820 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: [email protected] from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date ([email protected] from yt-dlp/yt-dlp-nightly-builds)
[twitter] Extracting URL: <MANUALLY_REDACTED_URL>
[twitter] <MANUALLY_REDACTED_URL>: Downloading guest token
[twitter] <MANUALLY_REDACTED_URL>: Downloading GraphQL JSON
ERROR: [twitter] <MANUALLY_REDACTED_URL>: NSFW tweet requires authentication. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 734, in extract
    ie_result = self._real_extract(url)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1434, in _real_extract
    status = self._extract_status(twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1412, in _extract_status
    status = self._graphql_to_legacy(self._call_graphql_api(self._GRAPHQL_ENDPOINT, twid), twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1289, in _graphql_to_legacy
    self.raise_login_required('NSFW tweet requires authentication')
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 1239, in raise_login_required
    raise ExtractorError(msg, expected=True)

[bashonly]

What happens when you try with --cookies-from-browser?

[InklingGirl]

Is that supposed to work w/ an apt-get installation of brave-browser on Ubuntu 22?

[bashonly]

it should yeah

[InklingGirl]

yt-dlp: error: unsupported browser specified for cookies: "https". Supported browsers are: brave, chrome, chromium, edge, firefox, opera, safari, vivaldi, whale

[InklingGirl]

I'm assuming I had to specify the browser, here's the verbose log for that:

[debug] Command-line config: ['--cookies-from-browser=brave', '-vU', '<MANUALLY_REDACTED_URL>']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version [email protected] from yt-dlp/yt-dlp-nightly-builds [111b61dde] (pip)
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-6.5.0-35-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.47.0, pyxattr-0.7.2, requests-2.31.0, secretstorage-3.3.1, sqlite3-3.37.2, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
Extracting cookies from brave
[debug] Extracting cookies from: "/home/christian/.config/BraveSoftware/Brave-Browser/Default/Cookies"
[Cookies] Loading cookie      0/  2019XDG_CURRENT_DESKTOP is set to an unknown value: "ubuntu"
[debug] detected desktop environment: OTHER
[debug] Chosen keyring: BASICTEXT
WARNING: cannot decrypt v11 cookies: no key found
Extracted 44 cookies from brave (1975 could not be decrypted)
[debug] cookie version breakdown: {'v10': 0, 'v11': 1975, 'other': 0, 'unencrypted': 44}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1820 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: [email protected] from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date ([email protected] from yt-dlp/yt-dlp-nightly-builds)
[twitter] Extracting URL: <MANUALLY_REDACTED_URL>
[twitter] <MANUALLY_REDACTED_URL>: Downloading guest token
[twitter] <MANUALLY_REDACTED_URL>: Downloading GraphQL JSON
ERROR: [twitter] <MANUALLY_REDACTED_URL>: NSFW tweet requires authentication. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 734, in extract
    ie_result = self._real_extract(url)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1434, in _real_extract
    status = self._extract_status(twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1412, in _extract_status
    status = self._graphql_to_legacy(self._call_graphql_api(self._GRAPHQL_ENDPOINT, twid), twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1289, in _graphql_to_legacy
    self.raise_login_required('NSFW tweet requires authentication')
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 1239, in raise_login_required
    raise ExtractorError(msg, expected=True)

[bashonly]

[debug] Extracting cookies from: "/home/christian/.config/BraveSoftware/Brave-Browser/Default/Cookies"
[Cookies] Loading cookie 0/ 2019XDG_CURRENT_DESKTOP is set to an unknown value: "ubuntu"

@InklingGirl Something is going wrong with --cookies-from-browser here. It might be worth opening a separate issue about this failure to extract cookies. So for this log, it's not working because yt-dlp isn't getting the cookies it needs.

As for your attempt with the cookies txt file, would you be willing to share a redacted copy of the cookiefile? There's no other way for me to see what's going on with it.

[InklingGirl]

I don't know how exactly to go about doing that at the moment, I wouldn't necessarily know what to redact, but what I can tell you is I am using an apt installation of brave-browser & the file was produced by the F/LOSS Chromium extension Get cookies.txt LOCALLY. @bashonly

[InklingGirl]

I do notice the second-top line of the cookies file is:

# This file is generated by yt-dlp.  Do not edit.

I'm not sure if that's because the aforementioned extension is really using yt-dlp code in some way or if yt-dlp's --get-cookies argument is tampering w/ the cookies file's contents somehow.

[InklingGirl]

Interestingly, I went ahead & regenerated the cookies.txt file again, & when I looked inside it now has many more contents & doesn't mention yt-dlp, & now I can use yt-dlp to DL age-restricted content again. Doing so also seems to have resolved my other issue (w/ mpv streaming) here: mpv-player/mpv#14233. I have no idea how my cookies.txt got messed up.