andripwn/crawler-python

email scraper/crawls using python (Google/Bing)

Python Email Crawler

This python script search/google certain keywords, crawls the webpages from the results, and return all emails found.

Requirements

• sqlalchemy
• urllib2

If you don't have, simply sudo pip install sqlalchemy.

Usage

Start the search with a keyword. We use "iphone developers" as an example.

python email_crawler.py "iphone developers"

The search and crawling process will take quite a while, as it retrieve up to 500 search results (from Google), and crawl up to 2 level deep. It should crawl around 10,000 webpages :)

After the process finished, run this command to get the list of emails

python email_crawler.py --emails

The emails will be saved in ./data/emails.csv

crawler-python's Issues

email_crawler.py does not even exist

When exporting the email list using the email export command, it simply searches for more emails.

It never exports anything, and behaves as if you just typed in another keyword(s) to scrape for more emails.