headless mode fails at hcaptcha challenge

[vogler]

headless mode fails at hcaptcha challenge

[vogler]

Proxies to bypass Cloudflare protection also have problems with hcaptcha:

• FlareSolverr/FlareSolverr#128
• https://github.com/NoahCardoza/CloudProxy#captcha-solvers
  • CaptchaHarvester collects for manual self-solving instead of paying for 2captcha etc.
  • hcaptcha-solver randomly selects images until solved

There are several browser extensions but extensions can not be used in headless mode: https://playwright.dev/docs/chrome-extensions

Free captcha solvers:

• https://github.com/dessant/buster browser extension that uses speech recognition to solve reCAPTCHA audio challenges
  • no support for hcaptcha: dessant/buster#319
• https://www.npmjs.com/package/puppeteer-hcaptcha uses TensorFlow's image recognition (unreliable?)
• https://www.nopecha.com/ - chrome extension seems to be free and work well
  • https://github.com/Sqaaakoi/noptcha-patch - above, but without affiliate tracking code injection
• https://github.com/shimuldn/hCaptchaSolverApi - says 1000/month for free?
• https://github.com/privacypass/challenge-bypass-extension - allows to solve challenges in advance; "Privacy Pass is currently supported by Cloudflare to allow users to redeem validly signed tokens instead of completing CAPTCHA solutions."

General info, detecting headless mode, anti-scraping:

• https://stackoverflow.com/questions/55493536/how-to-deal-with-the-captcha-when-doing-web-scraping-in-puppeteer/55500565#55500565
• https://intoli.com/blog/making-chrome-headless-undetectable/ - example for chrome-remote-interface
• https://help.apify.com/en/articles/1961361-several-tips-on-how-to-bypass-website-anti-scraping-protections
• https://incolumitas.com/2021/05/20/avoid-puppeteer-and-playwright-for-scraping/ - frameworks will always be detected, use normal browser; also has examples for simulating human mouse movement and typing
• https://stackoverflow.com/questions/62751377/bypass-cloudflare-with-puppeteer#comment117761360_62751377
  • "The only workaround that I could make work involved using a non-headless browser over the same IP address and snapshotting the cookie info from the site (which included the CloudFlare cookies) and then using those in the cookie jar for my Puppeteer and Perl scripts. It's less than ideal because it's not fully-automated, but it did work for me. Those cookies appear to be good for at least 3 months, so every 3 months I have to just manually get/set them again."
• berstend/puppeteer-extra#614 - Non-headless via docker + VNC
• berstend/puppeteer-extra#609 - control Chrome through debug port browserWSEndpoint

[vogler]

Tried https://www.hcaptcha.com/accessibility, got a link via E-Mail:

This will let you set a cookie that enables you to skip requests for hCaptcha challenges.

Save this email or bookmark the link above, then open it and click the link again in order to refresh the cookie if you start to see challenges again.

Please note that the number of requests per day is limited, and your IP and account will be banned in the event of abuse.

Visited link, but after clicking 'Set Cookie' button, it just says 'Could not issue a cookie'.

[vogler]

Assumption is that the captcha is shown in headless mode, despite the stealth plugin.

Confirmed by saving screenshots on failure: 3766754.

[vogler]

Fixed by #11 (run non-headless in docker).

[vogler]

Tried https://www.hcaptcha.com/accessibility, got a link via E-Mail:

Visited link, but after clicking 'Set Cookie' button, it just says 'Could not issue a cookie'.

Maybe that was because I didn't read the Browser Instructions for Cross-Site Cookies:

Go to Privacy and Security, then Cookies and other site data.
Make sure "Block third-party cookies" is not enabled.

Edit: no, the setting already was 'Block third-party cookies in Incognito'.
However, now it seems to work. Requested a new link and got 'Cookie set'.