lawzava/scrape

CLI utility to scrape emails from websites

scrape's Issues

error while visiting: Forbidden domain

Hi,

When I pass the domain https://php.com then it returns error while visiting: Forbidden domain this is because the line scrape.go#L90 call the function trimProtocol scrape.go#L107 and it returns .com .

Basically Trim returns a slice of the string s with all leading and trailing Unicode code points contained in cutset removed.

Instead of Trim if we use TrimPrefix then it returns the correct domain php.com not .com. i.e.

return strings.TrimPrefix(strings.TrimPrefix(requestURL, "http://"), "https://")

If I'm right then can you please update these chages.

FYR: https://play.golang.org/p/vjrqxFPsj_U

My code:

url := "https://php.com"
params := scraper.Parameters{
		Async:               true,
		Emails:              true,
		PrintLogs:           isDevelopment(),
		Recursively:         false,
		Website:             url,
	}
s := scraper.New(params)
err := s.Scrape(&emails)

Error: can't scrape sites with http://

Tried to scrape site:
./scrape -w "http://sneakycorp.htb" --debug

Got result:

[000001] 1 [ 1 - request] map["url":"https://sneakycorp.htb"] (50.331µs)
[000002] 1 [ 1 - error] map["status":"" "url":"https://sneakycorp.htb"] (53.622255ms)

There is no HTTPS server.

ERROR: encountered exception 'Uncaught' (122:21)

In the below code when I set JSWait true then it returns the output with the error in the console:

ERROR: encountered exception 'Uncaught' (122:21)

If I set JSWait to false then It will not show any error.

em := []string{}
p := scraper.Parameters{
    Async:               true,
    Emails:              true,
    FollowJSWaitExternalLinks: false,
    JSWait:              true,
    MaxDepth:            9,
    PrintLogs:           isDevelopment(),
    Recursively:         false,
    Website:             website,
}
s := scraper.New(p)
err := s.Scrape(&em)

Why I am getting this error?

Is it possible to suppress the mentioned error with JSWait true?

Wait indefinitely with --js option

The app sticks on c.Wait() https://github.com/lawzava/scrape/blob/master/scraper/scrape.go#L70 when I try the --js option.
Any idea ?

I use chromium under Windows 10 and the following command :

scrape --js --debug #default website

In debug logs I have a kind of js error, but it doesn't help me :

[000039] 1 [     5 - scraped] map["url":"https://lawzava.com/contact/"] (6.8101774s)
2020/10/01 16:58:47 executing chromedp: could not retrieve attribute "innerHTML": exception "Uncaught" (1:10): TypeError: Cannot read property 'innerHTML' of undefined
    at <anonymous>:2:11
    at <anonymous>:3:4

The mail is well collected in scrapedEmails variable...

Homebrew package

We already have a nice linux installation with snap, I am missing a mac installation with Homebrew.

Is it possible?