Add an alternative way to pass in login cookies (fully headless support)

[alexcsdev]

Add an alternative way to pass in login cookies (fully headless support)

[BobVul]

Thanks for the response! And the program really works quite well, locally :)

Unfortunately, it looks like simply copying existing cookies over isn't quite enough; they get cleared by Patreon. I can only assume Patreon IP-restricts existing sessions.

Perhaps the only alternative is a Puppeteer-controlled login with credentials passed via CLI or file, though of course that has the downside of needing to handle credentials. If you're willing to take such an implementation as a PR, I might look at doing one later, as time permits.

I think I may try X-forwarding for the initial login.

---

FWIW, if anyone else is seeing Chrome fail to launch early in the process, the following libraries were missing on a clean-ish Ubuntu container:

libnss3
libcups2
libxss1
libatk1.0-0
libatk-bridge2.0-0
libpangocairo-1.0-0
libgtk-3-0

Running .local-chromium/Linux-*/chrome-linux/chrome was a good way to figure out why Chrome wasn't launching.

[BobVul]

Just did a test run by copying a local logged-in chromedata over, but it looks like there are issues deeper than that:

2020-10-16 15:35:56.9801 INFO Retrieving cookies...
2020-10-16 15:35:57.0220 DEBUG Calling login check
2020-10-16 15:35:57.0327 DEBUG Retrieving browser
2020-10-16 15:35:57.0348 DEBUG Downloading browser
2020-10-16 15:35:57.0442 DEBUG Launching browser
2020-10-16 15:35:57.0896 FATAL Browser communication error. Exception: PuppeteerSharp.ChromiumProcessException: Failed to launch Chromium!
   at PuppeteerSharp.ChromiumProcess.State.StartingState.StartCoreAsync(ChromiumProcess p)
   at PuppeteerSharp.ChromiumProcess.State.StartingState.StartCoreAsync(ChromiumProcess p)
   at PuppeteerSharp.Launcher.LaunchAsync(LaunchOptions options)
   at PuppeteerSharp.Launcher.LaunchAsync(LaunchOptions options)
   at PatreonDownloader.PuppeteerEngine.PuppeteerEngine.GetBrowser() in E:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.PuppeteerEngine\PuppeteerEngine.cs:line 92
2020-10-16 15:35:57.0981 FATAL Fatal error, application will be closed: System.NullReferenceException: Object reference not set to an instance of an object.
   at PatreonDownloader.PuppeteerEngine.PuppeteerCookieRetriever.Login() in E:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.PuppeteerEngine\PuppeteerCookieRetriever.cs:line 49
   at PatreonDownloader.PuppeteerEngine.PuppeteerCookieRetriever.RetrieveCookies() in E:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.PuppeteerEngine\PuppeteerCookieRetriever.cs:line 93
   at PatreonDownloader.App.Program.RunPatreonDownloader(String creatorName, Boolean headlessBrowser, PatreonDownloaderSettings settings) in E:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.App\Program.cs:line 118
   at PatreonDownloader.App.Program.Main(String[] args) in E:\Sources\BigProjects\PatreonDownloader\PatreonDownloader.App\Program.cs:line 59

e: missing shared libs, investigating

[AlexCSDev]

Unfortunately I don't think I will have any free time in the near future to add this feature into the app itself. But you should be able to transfer cookies to your server by running app on your computer, logging in and then copying chromedata/Default/Cookies file to your server.

[AlexCSDev]

I don't think the issue is with patreon cookies. Their cookies are not ip-restricted unless they have changed something within last few months. But the api has some really annoying cloudflare bot protection and the problem might be that your server's ip and/or user agent triggers it, in that case you need to solve their challenge page somehow before you can do anything.

CLI-based login procedure doesn't sound like something that is going to be easy to implement as it should be able to support recaptcha interaction, but if you are willing to work on it I will be more than happy to accept PR as long as there are no issues with code quality.

[BobVul]

Ah, that's quite likely too. In that case maybe nothing for it but X-forwarding to manually solve the captcha. Obviously trying to automate through a captcha isn't going to be easy or robust.

[bobobo1618]

FWIW, I handle this with a full-fledged headless Chrome instance and remote debugging. The process I'd suggest is:

• SSH to your host, forwarding port 9222: ssh -L 9222:127.0.0.1:9222 <host>
• Start Chrome with:

google-chrome-stable \
  --headless \
  --disable-gpu \
  --remote-debugging-port=9222 \
  --user-data-dir=(pwd)/chromedata \
  --user-agent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36'

• Open the Chrome remote debugger by opening Chrome on your local machine and navigating to http://127.0.0.1:9222
• Click the about:blank link you see
• You'll be shown a page that looks like Chrome's debug tools, but with an address bar at the top and a large display of the browser's screen. You can interact with the address bar, click things on the screen and type things with your keyboard.
• Enter https://www.patreon.com in the debugger's address bar and hit enter
• Use the keyboard and mouse to log in

You'll then have a fully logged-in chromedata, created headlessly, so with the same IP and whatnot.

However I haven't found PuppeteerSharp (upon which PatreonDownloader is built) to be very reliable running headless, it always has issues, so I made a modification that connects it to the headless Chrome browser. I've found this much more reliable, flawless even.

[AlexCSDev]

@bobobo1618 That is a really interesting way of getting around this issue. Thank you for information about this, I will consider integrating the ability to connect to remote browser in the next version.

[AlexCSDev]

0.9.4 now has support for remote browser. Refer to docs/REMOTEBROWSER.md for additional information.