dedsecinside/torbot

Dark Web OSINT Tool

torbot's Issues

Release torBot V 1.2 (Stable release)

Things to be done before migrating to v1.2(stable)

• Update README.MD
• Update CHANGELOG.MD
• FIX BUGS
• CODE Standardization
• A minimal To-Do List

Insallation issue on Kali

Describe the bug
when i install torBot whith requirements.txt have this problem:

Collecting PyQt5==5.12.1 (from -r requirements.txt (line 9))
Could not find a version that satisfies the requirement PyQt5==5.12.1 (from -r requirements.txt (line 9)) (from versions: )
No matching distribution found for PyQt5==5.12.1 (from -r requirements.txt (line 9))

I try install PyQt manually and don't work anyway...

Appreciate your help

How do I fix this?

Traceback (most recent call last):
File "torBot.py", line 10, in
from modules.analyzer import LinkTree
ImportError: No module named analyzer
[4542] Failed to execute script torBot

What have I done wrong in the process of installing?

Design GUI for TorBot

I think we should create a GUI for TorBot to make it more user-friendly using something like Electron (JavaScript) possibly? I think it'd be a good framework because of the pure popularity so it'll be easy to find contributors and it's supposedly very productive so we'll be able to get out a version very soon.

Format .gitignore

Add some since of formatting to .gitignore and also add any useful patterns. A pattern that I was thinking of is

# Modules
modules/pattern1
modules/pattern2
modules/pattern3

# Tests
tests/pattern1
tests/pattern2
tests/pattern3

Set metadata for LinkNode

Describe the solution you'd like
Extract metadata from HTTP response and set it to LinkNode properties

ImportError: No module named dis3

Hi. When I try to install TorBot on Kali linux(in Vmware) error occurs.

root@kali:~/Downloads/TorBot-dev# ./install.sh Collecting pyinstaller Requirement already satisfied: pefile>=2017.8.1 in /usr/lib/python2.7/dist-packages (from pyinstaller) (2018.8.8) Collecting macholib>=1.8 (from pyinstaller) Using cached https://files.pythonhosted.org/packages/41/f1/6d23e1c79d68e41eb592338d90a33af813f98f2b04458aaf0b86908da2d8/macholib-1.11-py2.py3-none-any.whl Collecting altgraph (from pyinstaller) Using cached https://files.pythonhosted.org/packages/0a/cc/646187eac4b797069e2e6b736f14cdef85dbe405c9bfc7803ef36e4f62ef/altgraph-0.16.1-py2.py3-none-any.whl Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages (from pyinstaller) (40.2.0) Installing collected packages: altgraph, macholib, pyinstaller Successfully installed altgraph-0.16.1 macholib-1.11 pyinstaller-3.4 Traceback (most recent call last): File "/usr/local/bin/pyinstaller", line 10, in <module> sys.exit(run()) File "/usr/local/lib/python2.7/dist-packages/PyInstaller/__main__.py", line 79, in run import PyInstaller.building.build_main File "/usr/local/lib/python2.7/dist-packages/PyInstaller/building/build_main.py", line 33, in <module> from ..depend import bindepend File "/usr/local/lib/python2.7/dist-packages/PyInstaller/depend/bindepend.py", line 26, in <module> from . import dylib, utils File "/usr/local/lib/python2.7/dist-packages/PyInstaller/depend/utils.py", line 25, in <module> from ..lib.modulegraph import util, modulegraph File "/usr/local/lib/python2.7/dist-packages/PyInstaller/lib/modulegraph/util.py", line 16, in <module> from ._compat import StringIO, BytesIO, get_instructions, _READ_MODE File "/usr/local/lib/python2.7/dist-packages/PyInstaller/lib/modulegraph/_compat.py", line 29, in <module> from dis3 import get_instructions ImportError: No module named dis3 mv: cannot stat 'tmp_dist/torBot': No such file or directory rm: cannot remove 'torBot.spec': No such file or directory

What am I missing? I try to find out but can't.

Build visualizer module

• A visualizer module
• See other OSINT tools for reference

Error using -i flag

How does this flag work?

I have tried: python3 torBot.py -i and the results are equal to python3 torBot.py

However when I run it with an url, I get this error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/socks.py", line 813, in connect
    negotiate(self, dest_addr, dest_port)
  File "/usr/local/lib/python3.5/dist-packages/socks.py", line 477, in _negotiate_SOCKS5
    CONNECT, dest_addr)
  File "/usr/local/lib/python3.5/dist-packages/socks.py", line 552, in _SOCKS5_request
    raise SOCKS5Error("{0:#04x}: {1}".format(status, error))
socks.SOCKS5Error: 0x06: TTL expired

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/urllib/request.py", line 1254, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib/python3.5/http/client.py", line 1107, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/usr/lib/python3.5/http/client.py", line 849, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib/python3.5/socket.py", line 712, in create_connection
    raise err
  File "/usr/lib/python3.5/socket.py", line 703, in create_connection
    sock.connect(sa)
  File "/usr/local/lib/python3.5/dist-packages/socks.py", line 96, in wrapper
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/socks.py", line 817, in connect
    raise GeneralProxyError("Socket error", error)
socks.GeneralProxyError: Socket error: 0x06: TTL expired

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "torBot.py", line 130, in <module>
    main()
  File "torBot.py", line 116, in main
    b = readPage(link)
  File "/root/torbot/TorBoT/modules/pagereader.py", line 11, in readPage
    response = urllib.request.urlopen(req)
  File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 466, in open
    response = self._open(req, data)
  File "/usr/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 1282, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib/python3.5/urllib/request.py", line 1256, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error Socket error: 0x06: TTL expired>

[FEATURE_REQUEST] Save to File Feature

Hello, I would like to add save to JSON feature in TorBoT.
Should I directly make changes in core modules or I should create a new Module?

Thanks.

Use BFS for traversing links

Implement BFS for web crawling URLs, we currently only search the URL that is passed to TorBot for links on that page. We'd like to search them as a graph using BFS.

Bad configuration - Help needed

Hi,
I'm new to GitHub and I found this project interesting and I would like to contribute, but after setting the tor service and running the python script I get 0 results all the time.

Can someone help me, please?

Sorry for opening an issue for this but I didn't know what else to do.

Thanks

Invalid Connection header

Invalid Connection header: ['keep-alive'] when using --server flag [PR #152]

Save to DB

Hey, Anyone working on this feature?

Add Documentation to Functions

The code is already decently documented but we don't have full coverage yet and it would be extremely helpful since it would help us to build an API that can then easily generate docs. If you need any instructions or directions, I'd be glad to help you. We use Google Style Python Docstrings, here's a reference if needed: http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html

Torbot breaks for non onion site

usage:
python torBot.py -u www.google.com

Error:
Traceback (most recent call last):
File "torBot.py", line 188, in
main()
File "torBot.py", line 162, in main
html_content = pagereader.readPage(link)
File "/root/tor/TorEnv/TorBoT/modules/pagereader.py", line 9, in readPage
req = urllib.request.Request(site, None, headers)
File "/usr/lib/python3.5/urllib/request.py", line 269, in init
self.full_url = url
File "/usr/lib/python3.5/urllib/request.py", line 295, in full_url
self._parse()
File "/usr/lib/python3.5/urllib/request.py", line 324, in _parse
raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: 'www.google.com'

Stop making actual network connections when running tests

By either creating mock objects or simply downloading the test data locally and keeping it in a separate test_data folder. This will help with increasing the speed of test and not relying on a network connection to run tests. We don't want to touch actual servers while running tests and the tests currently take an unnecessary long amount of time.

Data_Collector [Feature Request]

Is your feature request related to a problem? Please describe.
Data collector for future prediction and analysis.

More details soon...

Feature/Idea Suggestions!!

All suggestions are welcome. Let's talk about what you'd like to see be added to TorBot and we can work on incorporating it into v2. This will be used to track your suggestions.

Does it store to a DB>

Does it store the results to a db>

Turn link tree into JSON

Is your feature request related to a problem? Please describe.
We need to be able to insert our trees into a DB and this will be easily done as JSON if we can figure out an algorithm/library/etc that can convert the link tree into a JSON object that can be stored.

Describe the solution you'd like
Not sure yet, open to suggestions.

Switch to Twisted from requests

We're currently using requests, which is a synchronous HTTP library. One of our features, which retrieves links and tests if the link is reachable via HTTP/HTTPS is extremely slow due to this. Switching to Twisted, which is asynchronous should provide a speed up as well as give us the ability to take advantage of the async nature of Twisted.

-i flag ideas

The -i flag is still under development and I will be listing the features which are to be implemented.

• Webpage Title
• Description of the site (By analyzing the meta data content of the page and using some mining techniques)
• Basic Reconnaissance
• Your idea ...

PEP8ify code

I would like to be able to pep8ify the code to fit Python standards

No module named 'ete3'

I did a fresh install on latest Kali and get this error:
Traceback (most recent call last):
File "torBot.py", line 10, in
from modules.analyzer import LinkTree
File "/usr/local/lib/python3.7/dist-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.dict)
File "modules/analyzer.py", line 6, in
from ete3 import Tree, TreeStyle, TextFace, add_face_to_node
ModuleNotFoundError: No module named 'ete3'
[6494] Failed to execute script torBot

I checked and ete3 is already installed but still get this error

Add get_images to LinkNodes

Describe the solution you'd like
Add a method to LinkNode that retrieves all of the urls from a link and caches them, similarly to emails and other links

Other solutions are welcomed and will be considered.

Invalid Connection header when using --server

Invalid Connection header: ['keep-alive'] when using --server flag

Fix 'Duplicate Code' issue in modules\pagereader.py

CodeFactor found an issue: Duplicate Code

It's currently on:
modules\pagereader.py:38-50
modules\pagereader.py:81-93

Add Documentation

Use Google Style Docstrings for documentation.
Style Guide: http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html

Fix 'Unused argument 'live'' issue in modules\getweblinks.py

CodeFactor found an issue: Unused argument 'live'

It's currently on:
modules\getweblinks.py:88

Connection Refused

Hey guys! I really like your project and i was trying to run it on my Kali linux but even after configuring torrc file I'm getting "ConnectionRefusedError: [Errno 111] Connection refused".

There is no documentation. Can you please explain the configuration to run this project. Thank you!

This is the error thrown by running torBot.py -

Traceback (most recent call last):
File "/usr/lib/python3.5/site-packages/stem/socket.py", line 398, in _make_socket
control_socket.connect((self._control_addr, self._control_port))
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):`
File "torBot.py", line 12, in
with Controller.from_port(port = 9051) as controller:
File "/usr/lib/python3.5/site-packages/stem/control.py", line 998, in from_port
control_port = stem.socket.ControlPort(address, port)
File "/usr/lib/python3.5/site-packages/stem/socket.py", line 372, in init
self.connect()
File "/usr/lib/python3.5/site-packages/stem/socket.py", line 243, in connect
self._socket = self._make_socket()
File "/usr/lib/python3.5/site-packages/stem/socket.py", line 401, in _make_socket
raise stem.SocketError(exc)
stem.SocketError: [Errno 111] Connection refused

TypeError: hasattr expected 2 arguments, got 3

TorBot/modules/utils.py

Line 84 in fa8eb53

if not isinstance(nodes, list) and not hasattr(nodes, 'Visit', False):

>>> hasattr(str, 'replace')
True
>>> hasattr(str, 'replace', False)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
TypeError: hasattr expected 2 arguments, got 3

https://docs.python.org/3/library/functions.html#hasattr

Fix code style

Some file don't have correct code style.

Add more test coverage

We should add more checks to each test case to cover cases more thoroughly. We should add coverage for edge cases, unexpected input and etc.

This is being addressed with #95 but we still need more test for network utilities and we should still cover more test cases.

Exception handling

When I run this (with the Tor service being down) [DEV branch]
python torBot.py -u http://torlinkbgs6aabns.onion/
I get this error

Attempting to connect to https://check.torproject.org/
Attempting to connect to https://check.torproject.org/
Attempting to connect to https://check.torproject.org/
Traceback (most recent call last):
  File "torBot.py", line 186, in <module>
    main(conn=True)
  File "torBot.py", line 158, in main
    print("Tor IP Address :", pagereader.get_ip())
  File "/tmp/TorBoT/modules/pagereader.py", line 102, in get_ip
    pg = page.find('strong')
AttributeError: 'NoneType' object has no attribute 'find'

its got to do something to do with

    if err == HTTPError:
        raise("There has been an HTTP error after three attempts.")
    if err == ConnectionError:
        raise("There has been a connection error after three attempts.")

in pagereader.py
Can we fix this?

Dockerize TorBot

• Dockerize Torbot to avoid issues like #155 .
• Entry point should torbot binary.

Fix multi-threading issue

Describe the bug
We're using multi threading to display links and Python isn't great at handling threads due to the GIL so this causes stalling when trying to display several hundred (at minimum) links. I'm thinking that the solution in this case is to process the links asynchronously instead, it may not be as fast but it'll be stable.

To Reproduce

1. python3 torBot.py -u https://hiddenwikitor.org

Expected behavior
LInks are retrieved until user cancels operation or all links have been found.

Website link listing taking too long

Status code taking too long to print the results.

if (urllib.request.urlopen(web).getcode() == 200):
   print (web)

- e flag not working

@KingAkeem . See the previous version (current master) . We missed the ext flag. That is no longer in the dev branch. Please add the code for it and submit to dev before porting to 1.2 .

Error when trying to install requirements.txt

Requirements.txt file is missing an equal sign operator for termcolor so you get an invalid operator error.
Invalid requirement: 'termcolor=1.1.0' = is not a valid operator. Did you mean == ?

Allow user to configure IP Address and Port

So currently TorBot runs on localhost (127.0.0.1) and port 9050, we'd like to add --ip & --port flags to allow a user to optionally give a new ip address or port.

-L flag not showing any output

The -l flag is supposed to return live web links but its not doing any thing in the latest build.

Error in getemails module

python3 torBot.py -u https://www.rapidtables.com/web/html/mailto.html -m

Traceback (most recent call last):
  File "torBot.py", line 186, in <module>
    main()
  File "torBot.py", line 163, in main
    emails = getemails.getMails(html_content)
  File "/home/psn/Documents/TorBoT/modules/getemails.py", line 30, in getMails
    emails.append(email_addr[1])
IndexError: list index out of range

```

Problem when running torBot binary

Hi,
When I try to run TorBot binary, I obtain an error and the tool stop. Here is the traceback :

Traceback (most recent call last):
  File "torBot.py", line 5, in <module>
    import settings
  File "/tmp/pip-install-Xz8Ej3/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 396, in load_module
  File "settings.py", line 4, in <module>
    load_dotenv(dotenv_path=env_path)
  File "dotenv/main.py", line 255, in load_dotenv
  File "dotenv/main.py", line 95, in set_as_environment_variables
  File "dotenv/main.py", line 74, in dict
  File "collections.py", line 69, in __init__
  File "_abcoll.py", line 571, in update
  File "dotenv/main.py", line 79, in parse
  File "dotenv/main.py", line 60, in _get_stream
  File "genericpath.py", line 26, in exists
TypeError: coercing to Unicode: need string or buffer, PosixPath found
[2568] Failed to execute script torBot

Do you know how to solve this?

Thanks for your help!

Fix 'Using type() instead of isinstance() for a typecheck.' issue in modules\pagereader.py

CodeFactor found an issue: Using type() instead of isinstance() for a typecheck.

It's currently on:
modules\pagereader.py:90

Invalid Syntax in savedb.py

Hi guys,

First of all, thank you for handling this project which seems really interesting.

When I try to test your tool, I have an error concerning savedb.py. Here is the traceback I have :

Traceback (most recent call last):
File "torBot.py", line 6, in
from modules import (bcolors, getemails, pagereader, getweblinks, updater,
File "/usr/toolz/TorBot/modules/savedb.py", line 31
print "Error '{0}' occurred. Arguments {1}.".format(e.message, e.args):
^
SyntaxError: invalid syntax

Did the mistake came from my side and you can I solve that?

Thank you for your help!

EDIT : Just saw that this error will be corrected in TorBot 1.3, am I right?

Refactor analyzer.py

I've selected analyzer.py for refactoring, which is a module of 53 lines of code that couples the top-level components tightly. Addressing this will make our codebase more maintainable and improve Better Code Hub's Couple Architecture Components Loosely guideline rating! 👍

Here's the gist of this guideline:

• Definition 📖
  Minimize the amount of interface code; that is, code in modules that are both called from and call modules of other components ("throughput"), and code in modules that are called from modules of other components ("incoming").
• Why❓
  Having loose coupling between top-level components makes it easier to maintain them in isolation.
• How 🔧
  You can hide a component's implementation details through various means, e.g. using the "abstract factory" design pattern.

You can find more info about this guideline in Building Maintainable Software. 📖

---

ℹ️ To know how many other refactoring candidates need addressing to get a guideline compliant, select some by clicking on the 🔲 next to them. The risk profile below the candidates signals (✅) when it's enough! 🏁

---

Good luck and happy coding! ✨ 💯

cprint's broken?

Describe the bug

# termcolor.py

def cprint(text, color=None, on_color=None, attrs=None, **kwargs):
    """Print colorize text.

    It accepts arguments of print function.
    """

    print((colored(text, color, on_color, attrs)), **kwargs)

How can cprint output properly with color without a comma separating the text and color?

TorBot/modules/info.py

Line 12 in fa8eb53

cprint("No robots.txt file Found!""blue")

TorBot/modules/info.py

Line 16 in fa8eb53

cprint("Error !""red")

TorBot/modules/info.py

Line 20 in fa8eb53

cprint("Error""red")

TorBot/modules/info.py

Line 24 in fa8eb53

cprint("Error""red")

TorBot/modules/info.py

Line 28 in fa8eb53

cprint("Error""red")

TorBot/modules/info.py

Line 32 in fa8eb53

cprint("Error""red")

TorBot/modules/info.py

Line 45 in fa8eb53

cprint("[*]Checking for Robots.txt"'yellow')

TorBot/modules/info.py

Line 53 in fa8eb53

cprint("[*]Checking for .git folder"'yellow')

TorBot/modules/info.py

Lines 58 to 62 in fa8eb53

	if r == 200:
	cprint("Alert!"'red')
	cprint(".git folder exposed publicly"'red')
	else:
	print("NO .git folder found"'blue')

TorBot/modules/info.py

Line 66 in fa8eb53

cprint("[*]Checking for .svn folder"'yellow')

TorBot/modules/info.py

Lines 71 to 75 in fa8eb53

	if r == 200:
	cprint("Alert!"'red')
	cprint(".SVN folder exposed publicly"'red')
	else:
	cprint("NO .SVN folder found"'blue')

TorBot/modules/info.py

Line 79 in fa8eb53

cprint("[*]Checking for .htaccess"'yellow')

TorBot/modules/info.py

Lines 84 to 90 in fa8eb53

	if statcode == 403:
	cprint("403 Forbidden"'blue')
	elif statcode == 200:
	cprint("Alert!!"'blue')
	cprint(".htaccess file found!"'blue')
	else:
	cprint("Status code"'blue')

TorBot/modules/info.py

Line 95 in fa8eb53

cprint("[*]Checking for description meta tag"'yellow')

Fix 'Final newline missing' issue in tests\test_savetofile.py

CodeFactor found an issue: Final newline missing

It's currently on:
tests\test_savetofile.py:40

Create a Wiki Page

Hello,
You are welcome to create a new wiki page for our project. The page should include almost every details of our project including usage,contribution guidelines and CHANGELOG. Please read the wiki guide before making a wiki page. For more information comment on this issue.
Wiki Guide