tigertron/parvist

Python web spider, web crawler, email harvester, and possible LFI/RFI vulnerability finder.

Parvist

Parvist is a web-crawler bot that can be used to harvest emails and many other bits of information on the web.

A new feature in version 0.5 is link saving for possible LFI/RFI vulnerabilities.

License

GPLv3

Author

Josten Moore

Features

1. Single domain scrape mode.
2. Link builder mode.
3. Time delays and "lag" to mitigate red-flags on the server.
4. Email harvesting mode.
5. Able to spoof your user-agent (3 supported, default is Googlebot).
6. Output is now threaded in a timer to make it easier to read
7. Link scanning for possible LFI/RFI vulnerabilites.

Getting Started

Clone this repo by doing:

git clone [email protected]:josten/parvist.git

Make sure that you have python 2.7.x installed.

Switches

• -s Scrape only the specified domain (no third party domains).
• -t Specify the connection timeout (default is 3 seconds).
• -b Specify the user-agent bot (default is google).
• -d Specify the depth to scrape (default is 3).
• -l Specify the lag between connection attempts (default is 100ms).
• -e If specified the scraper will harvest emails.
• -r If specified the scraper will look for possible LFI/RFI vulnerabilities.
• -h Prints help.

Usage

Using parvist is very simple; here are some examples:

• python parvist.py -h will print the help information.
• python parvist.py www.google.com -s -e will start scraping www.google.com for emails and ignore all third party domains.
• python parvist.py www.google.com -s -e -l 1 -t 5 will start scraping www.google.com for emails while ignoring third party domains and setting a 'lag' delay of 1 second and a connection timeout of 5 seconds.
• python parvist.py www.google.com -s -e -r will start scraping for possible LFI/RFI vulnerabilies while harvesting emails using default settings.

RFI/LFI Sample

The new -r switch has proven to be quite useful; here is a sample of some finds (real domains ommitted):

http://www.example.com/template.php?page=ref.php http://www.example.com/template.php?page=products.php http://www.example.com/template.php?page=history.php http://www.example.com/template.php?page=impressum.php http://www.example.com/template.php?page=finance.php http://www.example.com/template.php?page=customer.php http://www.example.com/template.php?page=sales.php http://www.example.com/template.php?page=contact.php http://www.example.com/template.php?page=index2.php http://www.example.com/template.php?page=intro.php