Unable to get content for some urls

[ecarter202]

Unable to get content for some urls

[ecarter202]

I continually get the following error(s) when trying to visit Newegg.com, wikipedia.org, etc...:

ERROR: Browser failed to start before browser startup time cutoff
panic: Chrome console failed to init in the alotted time

[integrii]

Interesting. Try upping the BrowserStartupTime value... By default it is 20 seconds. It could be that you are unable to startup a chrome process fast enough for some reason...

headlessCrhome.BrowserStartupTime = time.Minute * 5

also, make sure you are running with the right OS. By default we assume MacOS. Set the chrome path with headlessChrome.ChromePath = "somePath". By default, the path is /Applications/Google Chrome.app/Contents/MacOS/Google Chrome.

[ecarter202]

I've made your recommended changes, and it is still not getting any content with https://newegg.com.
`package main

import (
"fmt"
"time"

"github.com/integrii/headlessChrome"

)

func main() {
headlessChrome.BrowserStartupTime = time.Minute * 5

// make a new session
browser, err := headlessChrome.NewBrowser(`http://newegg.com`)
if err != nil {
	panic(err)
}
// Close the browser process when this func returns
defer browser.Exit()

// sleep while content is rendered.  You could replace this
// with some javascript that only returns when the
// content exists to be safer.
time.Sleep(time.Second * 5)

// Query all the HTML from the web site
browser.Write(`document.documentElement.outerHTML`)
time.Sleep(time.Second)

// loop over all the output that came from the output channel
// and print it to the console
for len(browser.Output) > 0 {
	fmt.Println(<-browser.Output)
}

}
`

Have you tried this on your end by chance?

Thanks!

[integrii]

I just tested your code snippet and didnt get any output. I turned on debugging with headlessChrome.Debug = true and still didn't get much. I then tried a variety of other javascript commands that use console.log() and window.location.href and was able to see those commands being entered into the console. I never got page content to load, though. Running as root didn't help.

I am not sure what changed here, but it does look like headless chrome isn't taking commands anymore. It would take a lot of rebuilding and testing to figure out what exactly has changed with Chrome. For now, I think this may be busted...

[sio2boss]

Confirmed, busted...

[integrii]

I tested this from the docker container example and it works. Check this: https://github.com/integrii/headlessChrome/tree/master/examples/docker

Something about the MacOS version broke... I will leave this issue open as a reminder to look into why the Mac version isn't working, but honestly Chrome changes things all the time and the constant chase may not be worth it.

I did hear that Chrome has introduced some new API for doing headless operations. Check this out: https://developers.google.com/web/tools/puppeteer/