nsonaniya2010/subdomainizer

A tool to find subdomains and interesting things hidden inside, external Javascript files of page, folder, and Github.

Buy Me A Coffee

SubDomainizer

SubDomainizer is a tool designed to find hidden subdomains and secrets present is either webpage, Github, and external javascripts present in the given URL. This tool also finds S3 buckets, cloudfront URL's and more from those JS files which could be interesting like S3 bucket is open to read/write, or subdomain takeover and similar case for cloudfront. It also scans inside given folder which contains your files.

Cloud Storage Services Supported:

SubDomainizer can find URL's for following cloud storage services:

1. Amazon AWS services (cloudfront and S3 buckets)
2. Digitalocean spaces 
3. Microsoft Azure 
4. Google Cloud Services 
5. Dreamhost 
6. RackCDN. 

Secret Key's Searching: (beta)

SubDomainizer will also find secrets present in content of the page and javascripts files. Those secret finding depends on some specific keywords and Shannon Entropy formula. It might be possible that some secrets which searched by tool will be false positive. This secret key searching is in beta and later version might have increased accuracy for search results.

Screenshots:

Installation Steps

1. Clone SubDomainzer from git:

git clone https://github.com/nsonaniya2010/SubDomainizer.git

2. Change the directory:

cd SubDomainizer

3. Install the requirements:

pip3 install -r requirements.txt

4. Enjoy the Tool.

Update to latest version:

Use following command to update to latest version:

git pull

Usage

Short Form	Long Form	Description
-u	--url	URL in which you want to find (sub)domains.
-l	--listfile	File which contain list of URL's needs to be scanned.
-o	--output	Output file name in which you need to save the results.
-c	--cookie	Cookies which needs to be sent with request.
-h	--help	show the help message and exit.
-cop	--cloudop	Give file name in which you need to store cloud services results.
-d	--domains	Give TLD (eg. for www.example.com you have to give example.com) to find subdomain for given TLD seperated by comma (no spaces b/w comma).
-g	--gitscan	Needed if you want to get things via Github too.
-gt	--gittoken	Github API token is needed, if want to scan (also needed -g also).
-gop	--gitsecretop	Saving secrets to a file found in github.
-k	--nossl	Use this to bypass the verification of SSL certificate.
-f	--folder	Root folder which contains files/folder.
-san	--subject_alt_name	Find Subject Alternative Names for all found subdomains, Options: 'all', 'same'.

SAN options description:

• all - This option will find all domains and subdomains.
• same - This will only find subdomains for specific subdomains.

Examples

• To list help about the tool:

python3 SubDomainizer.py -h

• To find subdomains, s3 buckets, and cloudfront URL's for given single URL:

python3 SubDomainizer.py -u http://www.example.com

• To find subdomains from given list of URL (file given):

python3 SubDomainizer.py -l list.txt

• To save the results in (output.txt) file:

python3 SubDomainizer.py -u https://www.example.com -o output.txt

• To give cookies:

python3 SubDomainizer.py -u https://www.example.com -c "test=1; test=2"

• To scan via github:

python3 SubDomainizer.py -u https://www.example.com -o output.txt -gt <github_token> -g 

• No SSL Certificate Verification:

python3 SubDomainizer.py -u https://www.example.com -o output.txt -gt <github_token> -g  -k

• Folder Scanning:

python3 SubDomainizer.py -f /path/to/root/folder/having/files/and/folders/  -d example.com  -gt <github_token> -g  -k

• Subject Alternative Names:

python3 SubDomainizer.py -u https://www.example -san all

• Saving secrets to a file scan found in github:

python3 SubDomainizer.py -u https://www.example.com -o output.txt -gt <github_token> -g -gop filename_to_save

Difference in results (with cookies and without cookies on facebook.com):

Results before using facebook cookies in SubDomainizer:

Results after using facebook cookies in SubDomainizer:

Changes:

In the latest version (2.0) following important features are added:

1. Find Subject Alternative Names for the found subdomains.
2. Added where the secrets were found.

License

This tools is licensed under the MIT license. take a look at the LICENSE for information about it.

Want to Help?

Want to help if you like features and tools? or Liked this tool? Help Here

subdomainizer's Issues

Ip address can't be above 255

###My Result:

`Got Some IPv4 addresses:

2.343.543.543
1.031.844.254
28.22.5.5
7.2.2.5
22.5.5.5
05.7.18.2
5.24.7.06
2.5.5.5
02.02.04.04
24.33.02.04
2.343.516.516
16.53.05.7
394.394.909.59
2.2.5.2
45.01.81.39
3.996.89.89
382.167.814.178
2.18.52.15
2.343.544.544
5.22.5.5
`

savesecretsresults() doesn't get called

Hi,

Firstly, a great tool, so glad I found it!

I have noticed that savesecretsresults() isn't begin called in the SubDomainizer.py. As a result, nothing is getting saved to -sop --SECRETOP.

SubDomainizer/SubDomainizer.py

Lines 699 to 705 in ea9a166

	def savesecretsresults():
	"""
	This function will save secret data into the given file.
	"""
	with open(secretop, 'w+') as f:
	for item in secretList:
	f.write(item + '\n')

I don't like to raise an issue without giving a possible solution but unfortunately, my coding isn't up to scratch.

Increased parallelism

As you mentioned in #13 you'll be re-writing the code to support showing the URL's that secrets are found from. Is it possible to also increase the parallelism of this script?

When running this to scan 100~ url's it takes hours. I created a quick wrapper in golang to have an instance of the script run on all of my cpu's cores and it finished in 15 minutes so there are definitely some bottlenecks in the code slowing it down that could likely be threaded.

Thanks for making this amazing tool!

func SubDomainizer(dir string) {
	println("starting SubDomainizer")

	if _, err := os.Stat(dir + "/" + date + "/" + "subdomainizer/domains"); os.IsNotExist(err) {
		os.MkdirAll(dir+"/"+date+"/"+"subdomainizer/domains", os.ModePerm)
	}
	if _, err := os.Stat(dir + "/" + date + "/" + "subdomainizer/cloud"); os.IsNotExist(err) {
		os.MkdirAll(dir+"/"+date+"/"+"subdomainizer/cloud", os.ModePerm)
	}
	if _, err := os.Stat(dir + "/" + date + "/" + "subdomainizer/secrets"); os.IsNotExist(err) {
		os.MkdirAll(dir+"/"+date+"/"+"subdomainizer/secrets", os.ModePerm)
	}

	var wg = sync.WaitGroup{}
	maxGoroutines := 10
	guard := make(chan struct{}, maxGoroutines)

	domains := ReadFile(dir + "[redacted]")
	for _, domain := range domains {
		guard <- struct{}{}
		wg.Add(1)
		go func(dir string, date string, domain string) {
			hash := GenerateRandomString()

			cmd := exec.Command("python3", "[redacted]]tools/SubDomainizer/SubDomainizer.py", "-u", domain,
				"-o", dir+"/"+date+"/subdomainizer/domains/"+hash+"_domains.txt", "-cop", dir+"/"+date+"/subdomainizer/cloud/"+hash+"_cloud.txt", "-sop", dir+"/"+date+"/subdomainizer/secrets/"+hash+"_secrets.txt",
				"-g", "-gt", "[redacted]")

			println(cmd.String())
			cmd.Start()
			cmd.Wait()
			<-guard
			wg.Done()
		}(dir, date, domain)

	}
	wg.Wait()

}

Add flag for ignoring SSL errors

Describe the bug
When testing against an internal site with invalid SSL I get this error:
An error occured while fetching URL, Might be server is down, or domain does not exist, Please check!

Suggested Fix
Add a flag which ignores all ssl errors (-k in this example)

1. Have site with invalid ssl
2. Run python3 SubDomainizer.py -u -k https://mysite.com
3. ???
4. Profit

Mtn.sd

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

bug

i use mitm for check requests, so sometimes script make request with joined strings.
i try using sript for github.com , and i see this:

172.18.0.11:37246: GET https://github.com/
200 OK 39.23k
172.18.0.11:37250: GET https://github.com/
200 OK 39.25k
GET https://github.githubassets.com/assets/compat-bootstrap-edde9aec.jshttps://github.githubassets.com/assets/compat-bootstrap-edde9aec.js
404 Not Found 30b
GET https://github.githubassets.com/assets/github-bootstrap-e9911286.js
200 OK 122.5k
GET https://github.githubassets.com/assets/compat-bootstrap-edde9aec.js
200 OK 9.42k
GET https://github.githubassets.com/assets/frameworks-564f5a6e.jshttps://github.githubassets.com/assets/frameworks-564f5a6e.js
404 Not Found 30b
GET https://github.githubassets.com/assets/frameworks-564f5a6e.js
200 OK 48.33k
GET https://github.githubassets.com/assets/github-bootstrap-e9911286.jshttps://github.githubassets.com/assets/github-bootstrap-e9911286.js
404 Not Found 30b

So few request contains of 2 urls

SubDomainizer error on RPi3

I am getting this error on Raspberry Pi 3. Any ideas on how to fix it?

Some urls cause Domainizer to totally freeze

This is a really annoying bug, after I gather subdomains and put the URLS in a list to scan them, subdomainizer completely freezes at specific urls for whatever reasons despite being alive hosts, I delete the URL from the txt file that's causing this and rescan, another URL that causes the tool to completely freeze and I had to remove that too..etc until the output finally shows when there are no more URLs causing a permanent freeze

other than that the tool is an exception and is very useful despite simplicity, thank you very much for publicly posting the project

Associated Js File

Hi @nsonaniya2010
It would be much more useful if you display associated js file from which the secret is exposed.

Error on Launch

The following error is thrown:

`Traceback (most recent call last):
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "", line 2, in raise_from
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "C:\Python310\lib\http\client.py", line 1374, in getresponse
response.begin()
File "C:\Python310\lib\http\client.py", line 318, in begin
version, status, reason = self._read_status()
File "C:\Python310\lib\http\client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Python310\lib\socket.py", line 705, in readinto
return self._sock.recv_into(b)
File "C:\Python310\lib\site-packages\urllib3\contrib\pyopenssl.py", line 312, in recv_into
return self.recv_into(*args, **kwargs)
File "C:\Python310\lib\site-packages\urllib3\contrib\pyopenssl.py", line 310, in recv_into
raise timeout('The read operation timed out')
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Python310\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 637, in urlopen
retries = retries.increment(method, url, error=e, _pool=self,
File "C:\Python310\lib\site-packages\urllib3\util\retry.py", line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Python310\lib\site-packages\urllib3\packages\six.py", line 686, in reraise
raise value
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 597, in urlopen
httplib_response = self._make_request(conn, method, url,
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 386, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 306, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='www.fisglobal.com', port=443): Read timed out. (read timeout=20)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\ricky\Tools\SubDomainizer\SubDomainizer.py", line 785, in
subextractor(compiledRegexCloud, compiledRegexSecretList,
File "C:\Users\ricky\Tools\SubDomainizer\SubDomainizer.py", line 618, in subextractor
jsfile.IntJsExtract(url, heads)
File "C:\Users\ricky\Tools\SubDomainizer\SubDomainizer.py", line 201, in IntJsExtract
req = requests.get(url, headers=heads, verify=False, timeout=(20, 20))
File "C:\Python310\lib\site-packages\requests\api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "C:\Python310\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Python310\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python310\lib\site-packages\requests\sessions.py", line 668, in send
history = [resp for resp in gen] if allow_redirects else []
File "C:\Python310\lib\site-packages\requests\sessions.py", line 668, in
history = [resp for resp in gen] if allow_redirects else []
File "C:\Python310\lib\site-packages\requests\sessions.py", line 239, in resolve_redirects
resp = self.send(
File "C:\Python310\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\Python310\lib\site-packages\requests\adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='www.example.com', port=443): Read timeout.`

Please help!

Cloud Storage

It would be good if you add some more cloud storage services like digitaloceanspaces.com,windows.net,storage.googleapis.com etc.

S3 Bucket Issue

Hello @nsonaniya2010
SubDomainizer is not parsing amazon bucket from url as the source is like

<a href="https://example.com" target="NEW"><img src="https://s3-us-west-2.amazonaws.com/domain/image.jpg"></a>

JS Scan

Hello @nsonaniya2010 ,
Can you please enhance SubDomainizer by adding one more switch to scan already downloaded js files from a folder.

[Add headless support]

Hi, this is awesome project for recon, May I ask, you have any plan add headless support? Headless mode can detect more domain and js files.

आ

Append http/https to subdomain list

Would be great if when reading from the subdomain list http or https is pre-appended, so it's easier to import from other tools.

Will be happy to send a PR later if needed.

Suggesting optional print of where the secretList items have been found

Is your feature request related to a problem? Please describe.
I know what the secrets are but I need to know where were they found so I can remove the data leak from my server

Describe the solution you'd like
I'd like to see where the items in secretList are found, to make it easier to understand which files/data is their source

TypeError: zip argument #1 must support iteration

By switched on search in GitHub such error appears here

facing issue with the tool when combined with gnu parallel

Timeout option

Option to have a timeout, some websites keep parsing for 30 min.. Timeout option would be a solution to skip such sites

https://github.com/nsonaniya2010/SubDomainizer.git

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Log results in different files

Currently the output is logged into one file which we can specify with the "-o" flag. Ideally, we could use that same flag and two different files would be generated: one with the results from subdomains another with cloud storage results. The names could be pre-appended with the parameter from the output flag.

This would make much easier to parse results from the tool and integration into different workflows.

Cheres

Js url print + threaded ?

Hi ,

Thanks for the great tool .

• Given the tool fetches many js files , regex prints the secrets on hits but does not print the URL where it was found . Printing js url wouldn't be a good enhancement ? it can save extra effort back-tracing the url .

• It is slow even when given list of 10-20 domains . Can multiprocessing be applied here to make it faster ?

Thanks

Issue when running a gitscan

I'm running linux Mint tina

uname:
Linux d0urd3n 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

This is the error I'm getting:

Traceback (most recent call last):
File "/home/n0w0nd3r/tools/SubDomainizer/SubDomainizer.py", line 869, in
gitThread.map(getGithubData, contentApiURLs)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/home/n0w0nd3r/tools/SubDomainizer/SubDomainizer.py", line 617, in getGithubData
_data = base64.b64decode(jsonData['content'])
KeyError: 'content'

Help

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Support for Json and subdomain IP Query

I love this tools, it's quite faster, and it has a features particularly i love that is

• Github data checking scan

• External Javascript examination

But particulary, Github scan, so It is possible to add just this few features

• Json output

• Query subdomain IP

And also add support for module import, that way it will support this tool from being used in
other python scripts. I wanted to intergrate this tool at https://www.nmmapper.com and particulary at it's collection of subdomain tools

So to be something like this

import subdomainizer
scan_result = subdomainizer.subdomain_scan("example.com")

git_scan_result = subdomainizer.git_scan("whatever")

Thank you.

no module named termcolor

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

y run the tool sudo python3 SubDomainizer.py -u target.com

I get

Traceback (most recent call last):
File "/home/bug/Desktop/tool2/SubDomainizer/SubDomainizer.py", line 13, in
import termcolor
ModuleNotFoundError: No module named 'termcolor'

the module is already installed in my linux mint.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

envalid escape sequence error

Hello @nsonaniya2010 ,
Getting this after scanning 10+ subdomains using list function:

Searching for External Javascript links in page..... Successfully got all the external js links SubDomainizer.py:132: DeprecationWarning: invalid escape sequence '\|' content = unquote(requests.get(js).content.decode('unicode-escape'))

Traceback (most recent call last): File "SubDomainizer.py", line 360, in <module> subextractor(compiledRegexCloud, compiledRegexSecretList, compiledRegexDomain,compiledRegexIP, i) File "SubDomainizer.py", line 290, in subextractor threads = ThreadPool(300) File "/usr/lib/python3.7/multiprocessing/dummy/__init__.py", line 124, in Pool return ThreadPool(processes, initializer, initargs) File "/usr/lib/python3.7/multiprocessing/pool.py", line 802, in __init__ Pool.__init__(self, processes, initializer, initargs) File "/usr/lib/python3.7/multiprocessing/pool.py", line 176, in __init__ self._repopulate_pool() File "/usr/lib/python3.7/multiprocessing/pool.py", line 241, in _repopulate_pool w.start() File "/usr/lib/python3.7/multiprocessing/dummy/__init__.py", line 51, in start threading.Thread.start(self) File "/usr/lib/python3.7/threading.py", line 847, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can't start new thread

Issue with termcolor

Describe the bug
When I run python3 SubDomainizer.py -h, I get the following error message:

Traceback (most recent call last):
File "SubDomainizer.py", line 13, in
import termcolor
ImportError: No module named 'termcolor'

I have verified that the module 'termcolor' is installed