skindud / python_crawler Goto Github PK
View Code? Open in Web Editor NEWThis project forked from arbuckle/python_crawler
Python-driven web crawler and scraper. Uses BeautifulSoup to gather all URLs from a target page, and initiates a crawl from a start URL, considering Whitelist/Blacklist criteria that are populated in crawl.py