lawzava/scrape

CLI utility to scrape emails from websites

Scrape

CLI utility to scrape emails from websites

Features

• Asynchronous scraping
• Recursive link follow
• External link follow
• Cloudflare email obfuscation decoding
• Client side rendered pages support through headless chromium load awaits
• Simple, grepable output

Install

• MacOS:

  brew tap lawzava/scrape https://github.com/lawzava/scrape
  brew install scrape

• Linux:

  sudo snap install scrape

Usage

Sample call:

scrape -w https://lawzava.com

Depends on chromium or google-chrome being available in path if --js is used

Parameters:

      --async             Scrape website pages asynchronously (default true)
      --debug             Print debug logs
  -d, --depth int         Max depth to follow when scraping recursively (default 3)
      --follow-external   Follow external 3rd party links within website
  -h, --help              help for scrape
      --js                Enables EnableJavascript execution await
      --output string     Output type to use (default 'plain', supported: 'csv', 'json') (default "plain")
      --output-with-url   Adds URL to output with each email
      --recursively       Scrape website recursively (default true)
      --timeout int       If > 0, specify a timeout (seconds) for js execution await
  -w, --website string    Website to scrape (default "https://lawzava.com")

Note about scraper package

For those that are looking for scraper package - this repository was intended as a cli-use only thus the scraper package was moved to lawzava/emailscraper. The scrape utility will be maintained as a CLI implementation of emailscraper package.

scrape's Issues

Wait indefinitely with --js option

The app sticks on c.Wait() https://github.com/lawzava/scrape/blob/master/scraper/scrape.go#L70 when I try the --js option.
Any idea ?

I use chromium under Windows 10 and the following command :

scrape --js --debug #default website

In debug logs I have a kind of js error, but it doesn't help me :

[000039] 1 [     5 - scraped] map["url":"https://lawzava.com/contact/"] (6.8101774s)
2020/10/01 16:58:47 executing chromedp: could not retrieve attribute "innerHTML": exception "Uncaught" (1:10): TypeError: Cannot read property 'innerHTML' of undefined
    at <anonymous>:2:11
    at <anonymous>:3:4

The mail is well collected in scrapedEmails variable...

Error: can't scrape sites with http://

Tried to scrape site:
./scrape -w "http://sneakycorp.htb" --debug

Got result:

[000001] 1 [ 1 - request] map["url":"https://sneakycorp.htb"] (50.331µs)
[000002] 1 [ 1 - error] map["status":"" "url":"https://sneakycorp.htb"] (53.622255ms)

There is no HTTPS server.

ERROR: encountered exception 'Uncaught' (122:21)

In the below code when I set JSWait true then it returns the output with the error in the console:

ERROR: encountered exception 'Uncaught' (122:21)

If I set JSWait to false then It will not show any error.

em := []string{}
p := scraper.Parameters{
    Async:               true,
    Emails:              true,
    FollowJSWaitExternalLinks: false,
    JSWait:              true,
    MaxDepth:            9,
    PrintLogs:           isDevelopment(),
    Recursively:         false,
    Website:             website,
}
s := scraper.New(p)
err := s.Scrape(&em)

Why I am getting this error?

Is it possible to suppress the mentioned error with JSWait true?

Homebrew package

We already have a nice linux installation with snap, I am missing a mac installation with Homebrew.

Is it possible?

error while visiting: Forbidden domain

Hi,

When I pass the domain https://php.com then it returns error while visiting: Forbidden domain this is because the line scrape.go#L90 call the function trimProtocol scrape.go#L107 and it returns .com .

Basically Trim returns a slice of the string s with all leading and trailing Unicode code points contained in cutset removed.

Instead of Trim if we use TrimPrefix then it returns the correct domain php.com not .com. i.e.

return strings.TrimPrefix(strings.TrimPrefix(requestURL, "http://"), "https://")

If I'm right then can you please update these chages.

FYR: https://play.golang.org/p/vjrqxFPsj_U

My code:

url := "https://php.com"
params := scraper.Parameters{
		Async:               true,
		Emails:              true,
		PrintLogs:           isDevelopment(),
		Recursively:         false,
		Website:             url,
	}
s := scraper.New(params)
err := s.Scrape(&emails)