devanshbatham/paramspider

Mining URLs from dark corners of Web Archives for bug hunting/fuzzing/further probing

paramspider

Mining URLs from dark corners of Web Archives for bug hunting/fuzzing/further probing

📖 About • 🏗️ Installation • ⛏️ Usage • 🚀 Examples • 🤝 Contributing •

About

paramspider allows you to fetch URLs related to any domain or a list of domains from Wayback achives. It filters out "boring" URLs, allowing you to focus on the ones that matter the most.

Installation

To install paramspider, follow these steps:

git clone https://github.com/devanshbatham/paramspider
cd paramspider
pip install .

Usage

To use paramspider, follow these steps:

paramspider -d example.com

Examples

Here are a few examples of how to use paramspider:

• Discover URLs for a single domain:

  paramspider -d example.com

• Discover URLs for multiple domains from a file:

  paramspider -l domains.txt

• Stream URLs on the termial:

  paramspider -d example.com -s

• Set up web request proxy:

  paramspider -d example.com --proxy '127.0.0.1:7890'

• Adding a placeholder for URL parameter values (default: "FUZZ"):

   paramspider -d example.com -p '"><h1>reflection</h1>'

Contributing

Contributions are welcome! If you'd like to contribute to paramspider, please follow these steps:

1. Fork the repository.
2. Create a new branch.
3. Make your changes and commit them.
4. Submit a pull request.

Star History

paramspider's Issues

Tool not working on any of the domain

python3 paramspider.py --domain bugcrowd.com --exclude woff,css,js,png,svg,php,jpg --output bugcrowd.txt

Can not get target information If you think this is a bug or unintentional behaviour. Report here : https://github.com/devanshbatham/ParamSpider/issues

For any of the domain it throws this error please help

TypeError: argument of type 'bool' is not iterable

    
Can not get target information
If you think this is a bug or unintentional behaviour. Report here : https://github.com/devanshbatham/ParamSpider/issues
Traceback (most recent call last):
  File "paramspider.py", line 73, in <module>
    main()
  File "paramspider.py", line 37, in main
    response = unquote(response)
  File "/usr/lib/python3.7/urllib/parse.py", line 609, in unquote
    if '%' not in string:
TypeError: argument of type 'bool' is not iterable

problem

gf redirect domain.txt
As you mentioned i follwed but its not working

no such pattern

Can not get target information If you think this is a bug or unintentional behaviour. Report here : https://github.com/devanshbatham/ParamSpider/issues

error

Can not get target information
If you think this is a bug or unintentional behaviour. Report here : https://github.com/devanshbatham/ParamSpider/issues
Traceback (most recent call last):
File "paramspider.py", line 73, in
main()
File "paramspider.py", line 37, in main
response = unquote(response)
File "/usr/lib/python3.7/urllib/parse.py", line 629, in unquote
if '%' not in string:
TypeError: argument of type 'bool' is not iterable

not work with subdomains

when i'm trying to do with subdomain max time it says
Can not get target information
If you think this is a bug or unintentional behaviour. Report here : https://github.com/devanshbatham/ParamSpider/issues

[+] Total unique urls found : 0
[+] Output is saved here : output/b.txt

[!] Total execution time : 2.0016s

query used :-

python paramspider.py --domain engage.abc.com -o b.txt

Cannot get target information

root@vivo1606:~/Downloads/ParamSpider-master# python3 paramspider.py --domain bugcrowd.com --exclude woff,css,js,png,svg,php,jpg

     ___                               _    __       
    / _ \___ ________ ___ _  ___ ___  (_)__/ /__ ____
   / ___/ _ `/ __/ _ `/  ' \(_-</ _ \/ / _  / -_) __/
  /_/   \_,_/_/  \_,_/_/_/_/___/ .__/_/\_,_/\__/_/   
                              /_/                    
                        
                        - coded with <3 by Devansh Batham 

Can not get target information
If you think this is a bug or unintentional behaviour. Report here : https://github.com/devanshbatham/ParamSpider/issues
root@vivo1606:~/Downloads/ParamSpider-master#

not work

OOPS!! Timeout Error. Retrying in 2 seconds.

does not find parameters

I have problem becouse paramspider not fint paramaters:( why?

I have website:

└─# cat /var/www/html/index.php

<form action="index.php" method="get">
<input type="text" name="name" value="name">
<input type="submit" name="send" value="send">
</form>

<a href="index.php?name=michas">aaa</a>
Look this shit
<?php
echo $_GET['name'];
?>

i run paramspider:

┌──(root💀za)-[/opt/ParamSpider]
└─# python3 /opt/ParamSpider/paramspider.py --domain  http://mydomian/index.php


         ___                               _    __
        / _ \___ ________ ___ _  ___ ___  (_)__/ /__ ____
       / ___/ _ `/ __/ _ `/  ' \(_-</ _ \/ / _  / -_) __/
      /_/   \_,_/_/  \_,_/_/_/_/___/ .__/_/\_,_/\__/_/
                                  /_/

                            - coded with <3 by Devansh Batham

Can not get target information
If you think this is a bug or unintentional behaviour. Report here : https://github.com/devanshbatham/ParamSpider/issues




[+] Total unique urls found : 0
[+] Output is saved here   : output/http://maindomain/index.php.txt

[!] Total execution time      : 0.48279s

┌──(root💀za)-[/opt/ParamSpider]

CURL:

┌──(root💀za)-[/opt/ParamSpider]
└─# curl http://mydomain/index.php
<form action="index.php" method="get">
<input type="text" name="name" value="name">
<input type="submit" name="send" value="send">
</form>

<a href="index.php?name=michas">aaa</a>
Look this shit

┌──(root💀za)-[/opt/ParamSpider]
└─# **curl http://mydomainl/index.php?name=XSS**
<form action="index.php" method="get">
<input type="text" name="name" value="name">
<input type="submit" name="send" value="send">
</form>

<a href="index.php?name=michas">aaa</a>
Look this shit
**XSS**                                                                                                                                                   
┌──(root💀za

Python Syntax Error

File "paramspider.py", line 40
url = f"http://web.archive.org/cdx/search/cdx?url=*.{args.domain}/*&output=txt&fl=original&collapse=urlkey&page=/"
^
SyntaxError: invalid syntax

It doesn't work, man

why do not you answer? It does not work!!!!!
error: OOPS!! Timeout Error. Retrying in 2 seconds.

please , help me

erro: {e} Can not get target information
If you think this is a bug or unintentional behaviour. Report here : https://github.com/devanshbatham/ParamSpider/issues

Not able to featch info

when i am giving command like python3 paramspider.py --domain hackerone.com, always showing target info not showing.

Service Temporary Unavailable

503 Server Error: Service Temporarily Unavailable for url:

not work

can not target get information error message !!

cli tool

create setup.py and publish to pypi

coded with <3 by Devansh Batham

I can’t run this scrip with a 'coded with <3 by Devansh Batham' error.
My env:Python3.7

Update Request

Hello friend, I just uploaded an update on Paramspider just updating the request

if you can check thank you

Request 2.27.0

@devanshbatham @vzamanillo @rotemreiss @RoninNakomoto @storenth @CowlingBanana @rotemplay

Possibility to save in any other directory the results?

What about changing in this file core/save_it.py:

This:

if outfile:
        filename = f'output/{outfile}'

to this:

if outfile:
        filename = f'{outfile}'

So this way the output file can be saved in other root directory.

Tool runnning but showing no results

Everytime i run the tool its not working , using ubuntu wsl
there are no errors or warning in runtime but still its not giving any results. Count is always 0 for every domain i use it for .

[+] Total unique urls found : 0
[+] Output is saved here : output/google.com.txt

command : python3 paramspider.py --domain google.com

please resolve this issue.

Not run paramspider

![Capture](https://user-images.githubusercontent.com/37695035/83180599-7c648800-a145-11ea-8395-ed1842299d9e.JPG

OOPS!! Timeout Error. Retrying in 2 seconds.

Hello
Every time I check the site with "paramspider", this message will be displayed to me!
Is there a problem or are those sites not vulnerable?

something wrong

can't use it ,always say Can not get target information. maybe i use wrong ...

Can not connect to server. Check your internet connection

I have installed the tool and found the following prompt while running it:

"Cannot connect to server. Check your internet connection"

Please find below the snapshot:

pip3 install -r requirements.txt
Collecting certifi==2020.4.5.1
Using cached certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)
Requirement already satisfied: chardet==3.0.4 in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 2)) (3.0.4)
Collecting idna==2.9
Using cached idna-2.9-py2.py3-none-any.whl (58 kB)
Collecting requests==2.23.0
Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Collecting urllib3==1.25.8
Using cached urllib3-1.25.8-py2.py3-none-any.whl (125 kB)
Installing collected packages: urllib3, idna, certifi, requests
Attempting uninstall: urllib3
Found existing installation: urllib3 1.24.3
Uninstalling urllib3-1.24.3:
Successfully uninstalled urllib3-1.24.3
Attempting uninstall: idna
Found existing installation: idna 2.8
Uninstalling idna-2.8:
Successfully uninstalled idna-2.8
Attempting uninstall: certifi
Found existing installation: certifi 2021.5.30
Uninstalling certifi-2021.5.30:
Successfully uninstalled certifi-2021.5.30
Attempting uninstall: requests
Found existing installation: requests 2.21.0
Uninstalling requests-2.21.0:
Successfully uninstalled requests-2.21.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
search-that-hash 0.2.8 requires click<8.0.0,>=7.1.2, but you have click 8.0.1 which is incompatible.
search-that-hash 0.2.8 requires requests<3.0.0,>=2.25.1, but you have requests 2.23.0 which is incompatible.
metafinder 1.1 requires beautifulsoup4>=4.9.3, but you have beautifulsoup4 4.6.3 which is incompatible.
metafinder 1.1 requires requests>=2.25.1, but you have requests 2.23.0 which is incompatible.
metafinder 1.1 requires urllib3>=1.26.4, but you have urllib3 1.25.8 which is incompatible.
emailfinder 0.2.3b0 requires beautifulsoup4>=4.9.3, but you have beautifulsoup4 4.6.3 which is incompatible.
emailfinder 0.2.3b0 requires requests>=2.25.1, but you have requests 2.23.0 which is incompatible.
emailfinder 0.2.3b0 requires urllib3>=1.26.4, but you have urllib3 1.25.8 which is incompatible.
censys 2.0.0 requires requests>=2.25.1, but you have requests 2.23.0 which is incompatible.
Successfully installed certifi-2020.4.5.1 idna-2.9 requests-2.23.0 urllib3-1.25.8
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv

┌──(root💀kali)-[~/ParamSpider]
└─# python3 paramspider.py --domain target.com --exclude woff,png,svg,php,jpg --output ~/target.com/DOMAINS/params_result.txt

     ___                               _    __                                                                                                                                                                                           
    / _ \___ ________ ___ _  ___ ___  (_)__/ /__ ____                                                                                                                                                                                    
   / ___/ _ `/ __/ _ `/  ' \(_-</ _ \/ / _  / -_) __/                                                                                                                                                                                    
  /_/   \_,_/_/  \_,_/_/_/_/___/ .__/_/\_,_/\__/_/                                                                                                                                                                                       
                              /_/                                                                                                                                                                                                        
                        
                        - coded with <3 by Devansh Batham 

Can not connect to server. Check your internet connection.

q:

Can not get target information
If you think this is a bug or unintentional behaviour. Report here : https://github.com/devanshbatham/ParamSpider/issues
Traceback (most recent call last):
File "paramspider.py", line 73, in
main()
File "paramspider.py", line 37, in main
response = unquote(response)
File "D:\Python38\lib\urllib\parse.py", line 614, in unquote
if '%' not in string:
TypeError: argument of type 'bool' is not iterable

To run an error

Traceback (most recent call last):
File "paramspider.py", line 73, in
main()
File "paramspider.py", line 37, in main
response = unquote(response)
File "C:\python3\lib\urllib\parse.py", line 634, in unquote
if '%' not in string:
TypeError: argument of type 'bool' is not iterable

Help

How can to use gf in windows im struggling following your steps i already install go located in C:\Users\juan\go\bin\gf. Im confused in this part Note : Replace '/User/levi/go/bin/gf' with the path where gf binary is located in your system.

Error with domain name

Hi,

Each time i try a domain name that contain a "-" i have this error:
Can not get target information

invalid syntax

on line 35 using Python 3.5

ParamSpider/paramspider.py

Line 35 in 3e1c5b1

url = f"http://web.archive.org/cdx/search/cdx?url=*.{args.domain}/*&output=txt&fl=original&collapse=urlkey&page=/"

Import Request Error

HI I`m using Python 3.7.4 v

when i tried to run this script I'm getting this below mentioned error

Traceback (most recent call last):
File "/root/XSS/ParamSpider/core/requester.py", line 1, in
import requests
ModuleNotFoundError: No module named 'requests'

Question?

This only supports .com?

RAM memory saturation

Some domains, if they are large, saturate the RAM memory and linux kills the process

For exaxmple: python3 paramspider.py -d crd.auctions.yahoo.co.jp

OS: Linux Manjaro 5.4.62-1-MANJARO

PLEASE ADD file option in Param Spider

Means instead of add -d example.com
we can add test file of url in paramaspider please add in next update

it doesn't work

python paramspider.py -d xxx.com
DOS print shbatham/ParamSpider//issues

version:python3.8

what happened? how can I do for it？
Longing for your valuable suggestions

tools is not running

The tools throws the following error
Can not get target information
If you think this is a bug or unintentional behaviour.

Steps to reproduce:

1. clone the repo
2. install all requirements using pip3
3. then ran python3 paramspider.py -d www.tesla.com

my system specs are in attachment

gf+paramspider on window

i installed gf and paramspider on windows machine, can you explain how can use in windows env?
gf is in C:\Users\nath\go\src\github.com\tomnomnom\gf
ps is in D:\Dropbox\bug\Asus\Web\Tools\OSINT\ParamSpider

exclude subdomains using -s is not working

unintentional behaviour

Can not get target information while scan the sites at sometimes

Path Related Issue

How can I find path of GF in my System?

'/User/levi/go/bin/gf'

i have problem

when i run gf i have error

SyntaxError: line 40

Hi, I have just noticed that you updated your script 5 days ago, so I tested out your script but somehow there's an error in line 40.

I get a bug

it's show "Can not get target information If you think this is a bug or unintentional behaviour."
and i change other domain it also bug。thanks

Not working on mac os and ubuntu ( all the requirement is instaled)

python paramspider.py --domain yahoo.com
File "paramspider.py", line 35
url = f"http://web.archive.org/cdx/search/cdx?url=*.{args.domain}/*&output=txt&fl=original&collapse=urlkey&page=/"
^
SyntaxError: invalid syntax

do not work correctly

Whatever target you enter, the prompt is shown in the figure below:

hi

Excuse me, this script not support the format of IP? such as 192.168.1.1:8080/admin/

Encoding issue with non-unicode characters

In this case the letter 'ć' or '\u0107'.

Traceback (most recent call last): File "paramspider.py", line 67, in <module> main() File "paramspider.py", line 52, in main print('\n'.join(final_uris)) File "AppData\Local\Programs\Python\Python37\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\u0107' in position 2259: character maps to <undefined>

error!! ##Traceback

Traceback (most recent call last):
File "paramspider.py", line 1, in
from core import requester
File "/home/user/Downloads/ParamSpider/core/requester.py", line 1, in
import requests
ModuleNotFoundError: No module named 'requests'
how to fix this bug can you please help me out

Unable to enumerate parameters

PC freeze when using paramspider

It does not show any response when scanning for domain. It freezes everything.
Please help

tool not working

There is no output from the tool after scanning any domain