Timeout error when waiting for browser response

[zfcsoftware]

Timeout error when waiting for browser response

[atalantus]

I had a very similar problem due to the targetFilter argument set by puppeteer-real-browser, when launching puppeteer. While (target) => !!target.url() avoids attaching to turnstile iframes it also avoids attaching to a lot of other possible targets. So some puppeteer code will lead to unexpected behavior, like for example timing out on a goto even though the page loaded.

There is no real and easy way to solve this in the general case and my recommendation is to set targetFilter to a user-controlled function like for example:

targetFilter: (target) => avoidTurnstileTarget() ? !!target.url() : true

I didn't test it for @MiguelRomaniw case however.

Just confirmed this to be the case.

Try:

const {browser, page} = await puppeteerRealBrowser({headless: false});
    
await page.goto('https://www.betano.de/');
    
await browser.close();

With the puppeteer-real-browser target filter targetFilter: (target) => !!target.url(), this will give a timeout error while with the default target filter (target) => true the code works as expected.

[zfcsoftware]

@MiguelRomaniw

This does not seem to be a library problem. If the site has cloudflare, the goto function does not complete and waits. After 30 seconds you get an error. For this
await page.goto(`https://br.betano.com/live/`, { waitUntil: 'domcontentloaded' }).catch(e => { })
you need to use your code in this way. Since the page load cannot be completed, the code does not continue and seems to give an error and stop working.
Can you add waitUntil and .catch(e => { }) to the redirects in the codes you use and test them? This should solve your problem.

I don't recall any code in puppeteer-real-browser that could cause this kind of error. This error is usually caused by incomplete loading.

await page.goto(`https://br.betano.com/live/`, { waitUntil: 'domcontentloaded' }).catch(e => { })

await page.reload({ waitUntil: 'domcontentloaded' }).catch(e => { })

Please change the redirection parts in your code as above. I believe the problem will be solved.

[atalantus]

I had a very similar problem due to the targetFilter argument set by puppeteer-real-browser, when launching puppeteer. While (target) => !!target.url() avoids attaching to turnstile iframes it also avoids attaching to a lot of other possible targets. So some puppeteer code will lead to unexpected behavior, like for example timing out on a goto even though the page loaded.

There is no real and easy way to solve this in the general case and my recommendation is to set targetFilter to a user-controlled function like for example:

targetFilter: (target) => avoidTurnstileTarget() ? !!target.url() : true

I didn't test it for @MiguelRomaniw case however.

[MiguelRomaniw]

Another detail that I just noticed, the browser is not going through cloudflare if you have the
'handless: true'

[zfcsoftware]

@MiguelRomaniw

First of all, the code you provided is very different from the event you conveyed. I tried to edit your code and add it below. I could not see an error in this code as you stated.

If you are having a problem like this when trying to switch to cloudflare, the solution is very easy.

await page.goto(`https://br.betano.com/live/`, { waitUntil: 'domcontentloaded' })

In this way you should issue a command to wait until the dom is loaded. Otherwise puppeteer will not accept the page as loaded and will give a timeout error.

If the problem persists, I can help if you provide a working code. I think the code below is not related to this issue.

async function login(credentials) {
    try {
        
        var { puppeteerRealBrowser } = await import('puppeteer-real-browser')
        const { page, browser } = await puppeteerRealBrowser({})
        setInterval(() => {
            page.screenshot({ path: 'example.png' });
        }, 200);
        await page.goto(`https://br.betano.com/live/`)

        await page.locator('[name = "Username"]').fill(credentials.user)
        await page.locator('[name = "Password"]').fill(credentials.password)
        await page.click('[data - qa="submit"]')
        await page.waitForTimeout(10000)
        await page.waitForNavigation({ waitUntil: 'domcontentloaded' });
        await page.evaluate(() => {
            window.location.href = 'https://br.betano.com/live/'
        })
        await page.waitForTimeout(10000)
        await page.reload()
        await page.waitForTimeout(10000)
        
    } catch (error) {
        console.log(error)
    }
}

login({ user: 'user', password: 'password' })

[zfcsoftware]

Another detail that I just noticed, the browser is not going through cloudflare if you have the 'handless: true'

Can you try Headless as 'new'?
Ubuntu headles true
windows headless false
It should also work when used with 'new'.

[zfcsoftware]

Basically, during the execution of the code, after I give a new page.goto (new url), the puppeteer loses the browser reference and does not return anything, just the TimeoutError error

export async function login(page, credentials){ try { await page.locator([name="Username"]).fill(credentials.user) await page.locator([name="Password"]).fill(credentials.password) await page.click([data-qa="submit"]) await page.waitForTimeout(10000) await page.goto([https://br.betano.com/live/](https://br.betano.com/live/%60)) // await page.waitForNavigation({ waitUntil: 'domcontentloaded' }); // await page.evaluate(() =>{ // window.location.href = https://br.betano.com/live/ // }) await page.waitForTimeout(10000) await page.reload() await page.waitForTimeout(10000) } catch (error) { console.log(error) } }

If your problem is that the browser connection is lost after redirects, I did not have such a problem with the code in the image below. If you can provide more details about the error, I can help.

[MiguelRomaniw]

I ended up sending only part of the code, as it is modularized, I will put them all in the same file to show you

Functions responsible for creating the browser and logging in
The error occurs either in await page.reload() or in await page.goto() both at login

export async function configBrowser(url) {
  const {browser, page} = await puppeteerRealBrowser({
    headless: false,
    action:'default',
    executablePath:'default',
})
  await page.setViewport({
    width: 1920,
    height: 10080,
  });
  await page.goto(url);
  await page.waitForTimeout(10000);

  return { browser, page };
}


export async function login(page, credentials){
  try {
      await page.locator(`[name="Username"]`).fill(credentials.user)
      await page.locator(`[name="Password"]`).fill(credentials.password)
      await page.click(`[data-qa="submit"]`)
      await page.waitForTimeout(10000)
      await page.goto(`https://br.betano.com/live/`)
      // await page.waitForNavigation({ waitUntil: 'domcontentloaded' });
      // await page.evaluate(() =>{
      //     window.location.href = `https://br.betano.com/live/`
      // })
      await page.waitForTimeout(10000)
      await page.reload()
      await page.waitForTimeout(10000) 
  } catch (error) {
      await page.screenshot({fullPage: true, path: "capturaDeTela.png"})
      console.log(error)
  }
}

And this is the part of the code where I import and run configBrowser and login

export async function run(config) {
  const executionId = +`${new Date().getTime()}${config.id}`;
  const store = new Store();
  const { browser, page } = await configBrowser("https://br.betano.com/myaccount/login");
  console.log("Browser Configurado betano");
  await login(page, { user: config.usuario, password: config.senha });
  console.log("login efetuado com sucesso betano");
}

Please note that this error does not occur on the first run, however after it occurs once, I need to end all processes related to it through the task manager, or restart the PC and then it will behave normally again.

[MiguelRomaniw]

@MiguelRomaniw

This does not seem to be a library problem. If the site has cloudflare, the goto function does not complete and waits. After 30 seconds you get an error. For this await page.goto(`https://br.betano.com/live/`, { waitUntil: 'domcontentloaded' }).catch(e => { }) you need to use your code in this way. Since the page load cannot be completed, the code does not continue and seems to give an error and stop working. Can you add waitUntil and .catch(e => { }) to the redirects in the codes you use and test them? This should solve your problem.

I don't recall any code in puppeteer-real-browser that could cause this kind of error. This error is usually caused by incomplete loading.

await page.goto(`https://br.betano.com/live/`, { waitUntil: 'domcontentloaded' }).catch(e => { })

await page.reload({ waitUntil: 'domcontentloaded' }).catch(e => { })

Please change the redirection parts in your code as above. I believe the problem will be solved.

When I put handless: true it does not pass the cloudflare check

And when I put handless: 'new'

It displays the browser normally

export async function configBrowser(url) {
  const {browser, page} = await puppeteerRealBrowser({
    headless: 'new',
    action:'default',
    executablePath:'default',
})
  await page.setViewport({
    width: 1920,
    height: 10080,
  });
  await page.goto(url);
  await page.waitForTimeout(10000);
  return { browser, page };
}

Now, regarding the page.goto() error and the oage.reload(), with the code you provided so far there were no problems

export async function login(page, credentials){
    try {
        await page.locator(`[name="Username"]`).fill(credentials.user)
        await page.locator(`[name="Password"]`).fill(credentials.password)
        await page.click(`[data-qa="submit"]`)
        await page.waitForTimeout(10000)
        await page.goto(`https://br.betano.com/live/`, { waitUntil: 'domcontentloaded' }).catch(e => { })
        // await page.waitForNavigation({ waitUntil: 'domcontentloaded' });
        // await page.evaluate(() =>{
        //     window.location.href = `https://br.betano.com/live/`
        // })
        await page.waitForTimeout(10000)
        await page.reload({ waitUntil: 'domcontentloaded' }).catch(e => { })
        await page.waitForTimeout(10000) 
    } catch (error) {
        await page.screenshot({fullPage: true, path: "capturaDeTela.png"})
        console.log(error)
    }
}

[zfcsoftware]

I'm glad you didn't encounter an error during your trial. Please add to this discussion if you get an error.
@atalantus I will look into the targetfilter issue he mentioned, and if there is a development, I will share it through this issue.
I tried Puppeteer-real-browser library with an account opening size by opening more than 1000 browsers. I did not have this kind of problem. However, since 2 people experienced it, it is a problem that needs to be solved. You will definitely make an update about this.

Headless false and New were not tried in Windows environment. This issue will be fixed in the next update.
Thank you for your contribution.

[GuiKlesseMiranda]

if we add = (target) => true cloudflare captcha enters in a loop

[zfcsoftware]

This issue has been resolved please try the latest version of the library. If the problem persists you can start a new discussion.