Giter Site home page Giter Site logo

elceef / dnstwist Goto Github PK

View Code? Open in Web Editor NEW
4.5K 155.0 734.0 4.43 MB

Domain name permutation engine for detecting homograph phishing attacks, typo squatting, and brand impersonation

Home Page: https://dnstwist.it

License: Apache License 2.0

Python 83.16% Dockerfile 1.91% HTML 14.93%
phishing typosquatting domains dns osint idn fuzzing threat-hunting homograph-attack scanner

dnstwist's Introduction

dnstwist

See what sort of trouble users can get in trying to type your domain name. Find lookalike domains that adversaries can use to attack you. Can detect typosquatters, phishing attacks, fraud, and brand impersonation. Useful as an additional source of targeted threat intelligence.

Demo

DNS fuzzing is an automated workflow that aims to uncover potentially malicious domains that target your organization. This tool generates a comprehensive list of permutations based on a provided domain name, and subsequently verifies whether any of these permutations are in use. Additionally, it can generate fuzzy hashes of web pages to detect ongoing phishing attacks or brand impersonation, and much more!

In a hurry? Try it in your web browser: dnstwist.it

Key features

  • Variety of highly effective domain fuzzing algorithms
  • Unicode domain names (IDN)
  • Additional domain permutations from dictionary files
  • Efficient multithreaded task distribution
  • Live phishing webpage detection:
    • HTML similarity with fuzzy hashes (ssdeep/tlsh)
    • Screenshot visual similarity with perceptual hashes (pHash)
  • Rogue MX host detection (intercepting misdirected e-mails)
  • GeoIP location
  • Export to CSV and JSON

Installation

Python PIP

$ pip install dnstwist[full]

Alternatively install the bare minimum and add other requirements manually depending on your needs:

$ pip install dnstwist

Git

If you want to run the latest version of the code, you can install it from Git:

$ git clone https://github.com/elceef/dnstwist.git
$ cd dnstwist
$ pip install .

Debian/Ubuntu/Kali Linux

Invoke the following command to install the tool with all extra packages:

$ sudo apt install dnstwist

Fedora Linux

$ sudo dnf install dnstwist

Arch Linux User Repository (yay)

$ yay -S dnstwist

macOS

This will install dnstwist along with all dependencies, and the binary will be added to $PATH.

$ brew install dnstwist

Docker

Pull and run official image from the Docker Hub:

$ docker run -it elceef/dnstwist

Alternatively you can build your local images:

$ docker build -t dnstwist .
$ docker build -t dnstwist:phash --build-arg phash=1 .

Quick start guide

The tool will run the provided domain name through its fuzzing algorithms and generate a list of potential phishing domains along with DNS records.

Usually thousands of domain permutations are generated - especially for longer input domains. In such cases, it may be practical to display only the ones that are registered:

$ dnstwist --registered domain.name

Ensure your DNS server can handle thousands of requests within a short period of time. Otherwise, you can specify an external DNS or DNS-over-HTTPS server with --nameservers argument.

If domain permutations generated by the fuzzing algorithms are insufficient, please supply dnstwist with a dictionary file. Some dictionary samples with a list of the most common words used in phishing campaigns are included.

$ dnstwist --dictionary dictionaries/english.dict domain.name

If you need to check whether domains with different TLD exist, just supply a dictionary file with the list of TLD.

$ dnstwist --tld dictionaries/common_tlds.dict domain.name

On the other hand, if only selected algorithms need to be used, --fuzzers argument is available, which takes a comma-separated list.

$ dnstwist --fuzzers "homoglyph,hyphenation" domain.name

Apart from the colorful terminal output, the tool allows exporting results to CSV and JSON:

$ dnstwist --format csv domain.name | column -t -s,
$ dnstwist --format json domain.name | jq

In case you need just the bare permutations without making any DNS lookups, use --format list argument:

$ dnstwist --format list domain.name

The tool can perform real-time lookups to return geographical location (approximated to the country) of IPv4 addresses.

$ dnstwist --geoip domain.name

The GeoIP2 library is used by default. Country database location has to be specified with $GEOLITE2_MMDB environment variable. If the library or the database are not present, the tool will fall-back to the older GeoIP Legacy.

To display all available options with brief descriptions simply execute the tool without any arguments.

Phishing detection

Manually checking each domain name in terms of serving a phishing site might be time-consuming. To address this, dnstwist makes use of so-called fuzzy hashes (locality-sensitive hash, LSH) and perceptual hashes (pHash). Fuzzy hashing is a concept that involves the ability to compare two inputs (HTML code) and determine a fundamental level of similarity, while perceptual hash is a fingerprint derived from visual features of an image (web page screenshot).

Fuzzy hashing

The unique feature of detecting similar HTML source code can be enabled with --lsh argument. For each generated domain, dnstwist will fetch content from responding HTTP server (following possible redirects), normalize HTML code and compare its fuzzy hash with the one for the original (initial) domain. The level of similarity is expressed as a percentage.

In cases when the effective URL is the same as for the original domain, the fuzzy hash is not calculated at all in order to reject false positive indications.

Note: Keep in mind it's rather unlikely to get 100% match, even for MITM attack frameworks, and that a phishing site can have a completely different HTML source code.

$ dnstwist --lsh domain.name

In some cases, phishing sites are served from a specific URL. If you provide a full or partial URL address as an argument, dnstwist will parse it and apply for each generated domain name variant. Use --lsh-url to override URL to fetch the original web page from.

$ dnstwist --lsh https://domain.name/owa/
$ dnstwist --lsh --lsh-url https://different.domain/owa/ domain.name

By default, ssdeep is used as LSH algorithm, but TLSH is also available and can be enabled like so:

$ dnstwist --lsh tlsh domain.name

Perceptual hashing

If Chromium browser is installed, dnstwist can utilize its headless mode, which operates without a graphical user interface, to capture web page screenshots, render them, and calculate pHash values. These pHash values are then compared to evaluate the visual similarity, expressed as a percentage.

$ dnstwist --phash domain.name

Moreover, it is possible to save the captured screenshots in the PNG format to a location of choice:

$ dnstwist --phash --screenshots /tmp/domain domain.name

Note: Due to the multi-threaded use of a fully functional web browser, an appropriate amount of free resources (mainly memory) should be provided.

Proxy support

For all HTTP connections, proxies are automatically used when the presence of environment variables named $<scheme>_proxy, in a case insensitive approach, is detected. If both lowercase and uppercase environment variables exist, lowercase is preferred.

API

In case you need to consume the data produced by the tool within your code, the most convenient and efficient way is to pass the input as follows.

>>> import dnstwist
>>> data = dnstwist.run(domain='domain.name', registered=True, format='null')

To work in a completely passive operating mode and produce just domain permutations, it is required to combine the list format with output redirection to the null device.

>>> dnstwist.run(domain='domain.name', format='list', output=dnstwist.devnull)

The arguments for dnstwist.run() are translated internally, so the usage is very similar to the command line. The returned data structure is an easy-to-process list of dictionaries. Keep in mind that dnstwist.run() spawns a number of daemon threads.

Performance tuning

When it comes to testing thousands of domain permutations, speed and efficiency are obvious priorities. On the other hand the tool was designed to "work out of the box", refraining from overwhelming DNS resolvers and conserving precious resources. That said, the default settings strike a cautious balance, but there's always area for improvement.

It is recommended to experiment with the number of threads. Initially this number is computed based on the available CPU cores, but in most cases elevating this value gives a substantial performance boost. Another suggestion is to select fast DNS resolver(s) with the lowest network round-trip time (RTT). While a few miliseconds may not sound as a big difference, when multiplied across thousands of domain permutations, it translates to noticeable time savings.

Notes on coverage

As the length of the input domain increases, the number of variants generated by the algorithms increases significantly, resulting in a substantial increase in the time and resources required to verify them. Checking every possible domain permutation is impractical, especially for longer input domains, which would require millions of DNS lookups. Thus, this tool generates and checks domains that are very similar to the original one. Theoretically, these domains are the most appealing from an attacker's point of view. However, it's essential to note that attackers' imagination is unlimited.

Unicode tables comprise thousands of characters with many of them visually similar to one another. However, despite the fact certain characters are encodable using punycode, most TLD authorities will reject them during domain registration process. In general, TLD authorities disallow mixing of characters coming from different Unicode scripts or maintain their own sets of acceptable characters. With that being said, the homoglyph fuzzer was build on top of carefully researched range of Unicode characters (homoglyphs) to ensure that generated domains can be registered in practice.

Integrations

The scanner is utilized by tens of SOC and incident response teams around the globe, as well as independent information security analysts and researchers. On top of this, it's integrated into products and services of many security providers, in particular but not only:

Splunk add-on, RecordedFuture, SpiderFoot, DigitalShadows, SecurityRisk, SmartFense, ThreatPipes, PaloAlto Cortex XSOAR, Rapid7 InsightConnect SOAR, Mimecast, Watcher, Intel Owl, PatrOwl, VDA Labs, Appsecco, Maltego, Conscia ThreatInsights, Fortinet FortiSOAR, ThreatConnect, CISA Crossfeed.

Contact

To send questions, thoughts or a bar of chocolate, just drop an e-mail at [email protected]. Any feedback is appreciated. If you have found some confirmed phishing domains or just like this tool, please don't hesitate and send a message. Thank you.

dnstwist's People

Contributors

alex-plutoflume avatar alphagolf33 avatar andrewjbennett avatar anroots avatar bayandin avatar c0rner avatar cireo avatar ctdhr avatar ekwen avatar elceef avatar emmiegit avatar eugk avatar fabaff avatar fproldan avatar hardwaterhacker avatar ikke avatar ipv4v6 avatar krmaxwell avatar lksnyder0 avatar lupo42 avatar makyotox avatar monoidic avatar morisson avatar nuno-andre avatar ream88 avatar silberschleier avatar thisismyrobot avatar wiene avatar ya0guang avatar yahiakandeel avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dnstwist's Issues

Release 20180528 runtime error: TypeError: cannot use a string pattern on a bytes-like object

When I'm using the latest (20180528) release I get this error:

Traceback (most recent call last):
  File "dnstwist.py", line 990, in <module>
    main()
  File "dnstwist.py", line 845, in main
    dfuzz.generate()
  File "dnstwist.py", line 483, in generate
    self.__filter_domains()
  File "dnstwist.py", line 275, in __filter_domains
    if self.__validate_domain(d['domain-name']) and d['domain-name'] not in seen:
  File "dnstwist.py", line 266, in __validate_domain
    return allowed.match(domain.encode('idna'))
TypeError: cannot use a string pattern on a bytes-like object

Executing: python dnstwist.py amazon.com

I am using a dedicated virtual environment with requirements.txt installed according to the README.
Python version is 3.6. I also tried in a venv with Python 3.4, it equally fails.

Some possible domain names are not found.

Some possible domain names are not found with this tool.
For example "mìcrosoft.net" is not in the result of: "./dnstwist.py microsoft.net" and is still available for registration at this moment. (I guess someone will register it after reading this issue)

kali linux installing not working

BUILD_LIB=1 pip install -r requirements.txt
Requirement already satisfied: GeoIP>=1.3.2 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 1)) (1.3.2)
Requirement already satisfied: dnspython>=1.14.0 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 2)) (1.16.0)
Requirement already satisfied: requests>=2.20.0 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 3)) (2.21.0)
Collecting ssdeep>=3.1.1 (from -r requirements.txt (line 4))
  Using cached https://files.pythonhosted.org/packages/e0/d3/f17602a7dde1231d332f4067fdd421057ffe335c3bbc295e7ccfab769d95/ssdeep-3.4.tar.gz
    Complete output from command python setup.py egg_info:
    warning: no previously-included files found matching 'setup.pyc'
    warning: no previously-included files matching 'yacctab.*' found under directory 'tests'
    warning: no previously-included files matching 'lextab.*' found under directory 'tests'
    warning: no previously-included files matching 'yacctab.*' found under directory 'examples'
    warning: no previously-included files matching 'lextab.*' found under directory 'examples'
    zip_safe flag not set; analyzing archive contents...
    pycparser.ply.lex: module references __file__
    pycparser.ply.lex: module MAY be using inspect.getsourcefile
    pycparser.ply.yacc: module references __file__
    pycparser.ply.yacc: module MAY be using inspect.getsourcefile
    pycparser.ply.yacc: module MAY be using inspect.stack
    pycparser.ply.ygen: module references __file__
    
    Installed /tmp/pip-install-67kgst/ssdeep/.eggs/pycparser-2.19-py2.7.egg
    running egg_info
    creating pip-egg-info/ssdeep.egg-info
    writing requirements to pip-egg-info/ssdeep.egg-info/requires.txt
    writing pip-egg-info/ssdeep.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/ssdeep.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/ssdeep.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/ssdeep.egg-info/SOURCES.txt'
    sh: 0: Can't open configure
    /bin/sh: 1: libtoolize: not found
    configure.ac: error: no proper invocation of AM_INIT_AUTOMAKE was found.
    configure.ac: You should verify that configure.ac invokes AM_INIT_AUTOMAKE,
    configure.ac: that aclocal.m4 is present in the top-level directory,
    configure.ac: and that aclocal.m4 was recently regenerated (using aclocal)
    configure.ac:7: installing './config.guess'
    configure.ac:7: installing './config.sub'
    configure.ac:13: installing './install-sh'
    Makefile.am:9: error: Libtool library used but 'LIBTOOL' is undefined
    Makefile.am:9:   The usual way to define 'LIBTOOL' is to add 'LT_INIT'
    Makefile.am:9:   to 'configure.ac' and run 'aclocal' and 'autoconf' again.
    Makefile.am:9:   If 'LT_INIT' is in 'configure.ac', make sure
    Makefile.am:9:   its definition is in aclocal's search path.
    Makefile.am: installing './depcomp'
    /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCC does not appear in AM_CONDITIONAL
    /usr/share/automake-1.16/am/depend2.am:   The usual way to define 'am__fastdepCC' is to add 'AC_PROG_CC'
    /usr/share/automake-1.16/am/depend2.am:   to 'configure.ac' and run 'aclocal' and 'autoconf' again
    /usr/share/automake-1.16/am/depend2.am: error: AMDEP does not appear in AM_CONDITIONAL
    /usr/share/automake-1.16/am/depend2.am:   The usual way to define 'AMDEP' is to add one of the compiler tests
    /usr/share/automake-1.16/am/depend2.am:     AC_PROG_CC, AC_PROG_CXX, AC_PROG_OBJC, AC_PROG_OBJCXX,
    /usr/share/automake-1.16/am/depend2.am:     AM_PROG_AS, AM_PROG_GCJ, AM_PROG_UPC
    /usr/share/automake-1.16/am/depend2.am:   to 'configure.ac' and run 'aclocal' and 'autoconf' again
    /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL
    /usr/share/automake-1.16/am/depend2.am:   The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX'
    /usr/share/automake-1.16/am/depend2.am:   to 'configure.ac' and run 'aclocal' and 'autoconf' again
    Failed while building ssdeep lib with configure and make.
    Retry with autoreconf ...
    Failed to reconfigure the project build.
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-67kgst/ssdeep/

using kali and need your help that i can test this tool?

Incomplete copyright information

In the context of packaging dnstwist for Debian I am trying to document the copyright information for the files included in your software. It seems to me that at least for the files inside the database directory proper copyright information is missing. Could you please add the following information for those files:

  • the copyright year(s)
  • the name of the author(s)/copyright holder(s)
  • the licence
  • the URL to the source of those files

For the file effective_tld_names.dat I've found information for the last two items but unless I overlooked something, the remaining information is missing.

Using two dictionaries while querying

I am trying to use two dictionaries, a TLD, and a word file dictionary, to detect typo squat domains. For example, If I had to catch googlefinance.live while querying google.com, I need "finance" in the word dictionary and "live" in the TLD. Is there an option to do this already? If so, could you please guide me through this?

SPYING-MX, mxcheck, missing from csv output

SPYING-MX, mxcheck, missing from csv output

$ ./dnstwist.py -f cli -rm example.com
_ _ _ _
| | __ | | ()| |
/ ` | ' / __| __\ \ /\ / / / __| __|
| (
| | | | __ \ |_ \ V V /| __ \ |_
_,|| ||/_| _/_/ ||_/_| {20200429}

Processing 1648 domain variants ..........15%.......30%......57%..81%......96%..... 92 hits (5%)
[...]
Addition exampled.com 91.195.240.126 NS:ns1.sedoparking.com SPYING-MX:mail.pickelhost.com


$ ./dnstwist.py -f csv -rm example.com
fuzzer,domain-name,dns-a,dns-aaaa,dns-mx,dns-ns,geoip-country,whois-created,whois-updated,ssdeep-score
[...],
Addition,exampled.com,91.195.240.126,,mail.pickelhost.com,ns1.sedoparking.com,,,,

GeoIP module no longer working

It's been a little while since I've used dnstwist so I updated my brew formulae and they attempted to run my script across multiple domains. I noticed that I was getting an error that stated the following:

Error Opening file /usr/local/var/GeoIP/GeoIP.dat

I then tried to run dnstwist directly and not via a script and I get the same error just running dnstwist with no arguments.

Current version of dnstwist installed via brew is 20200916.

The version of GeoIP installed is "stable 1.6.12" and from research conducted appears to be deprecated and no longer supported.

Will GeoIP support be moved to a more updated module or removed completely?

DNS over https/tls

First of all; thank you very much for this project!

I'm using it for my own pet project and had the requirement to, instead of making regular DNS requests, make DNS requests over https (DoH).

I was wondering if this functionality is something you would consider incorporating into dnstwist?

As DoH uses HTTP GET requests to; there might be some additional performance improvements as well if we leverage async coroutines (haven't fully tested that btw)

'AttributeError' when using -m argument

user@5a834e1e46b1:/# dnstwist -r -t 1 -m example.com
_ _ _ _
| | __ | | ()| |
/ ` | ' / __| __\ \ /\ / / / __| __|
| (
| | | | __ \ |_ \ V V /| __ \ |_
_,|| ||/_| _/_/ ||_/_| {20200521}

Processing 1648 permutations ·Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.8/dist-packages/dnstwist.py", line 579, in run
if domain['domain-name'] is not self.domain_orig:
AttributeError: 'DomainThread' object has no attribute 'domain_orig'


user@5a834e1e46b1:/# pip3 list
Package Version


certifi 2020.4.5.1
chardet 3.0.4
dnspython 1.16.0
dnstwist 20200521
idna 2.9
pip 20.0.2
ppdeep 20200505
requests 2.23.0
setuptools 45.2.0
tld 0.12.2
urllib3 1.25.9
wheel 0.34.2
whois 0.9.7

user@5a834e1e46b1:/# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04 LTS"

python whois package

From looking at the code and if I understand correctly the whois package being used is: https://pypi.org/project/whois . This is an old package and from the tests I did it is failing on many queries (mostly issues with parsing the response).

Are there any plans to move to the following whois package: https://pypi.org/project/python-whois/ . It seems to work better on most queries. Also, it does direct queries and not via the whois executable.

Note that there may be some confusion as the debian package python-whois is actually using the package https://pypi.org/project/whois .

The docker image is not working

Hello,

I have pulled the docker image from the docker hub. (https://hub.docker.com/r/elceef/dnstwist)
The image pulls successfully to my system, but when I go to run the container it instantly crashes.

I have tried a number of things, but no luck.
I do not see any older images besides latest, so I am not sure what else to try.

If you could look into this it would be greatly appreciated!

Image ID I pulled: f1f1e4857cf6

Also I am on the latest Docker, I am wondering if that could have something to do with it.
See Attached Image.

DockerVersion

Why does calling the script need to process so many domains, and the CLI is more terse/efficient?

$ dnstwist --registered maximumfun.org
     _           _            _     _
  __| |_ __  ___| |___      _(_)___| |_
 / _` | '_ \/ __| __\ \ /\ / / / __| __|
| (_| | | | \__ \ |_ \ V  V /| \__ \ |_
 \__,_|_| |_|___/\__| \_/\_/ |_|___/\__| {1.02}

Processing 284 domain variants ...

-vs.-

$ python dnstwist.py --registered maximumfun.org
     _           _            _     _
  __| |_ __  ___| |___      _(_)___| |_
 / _` | '_ \/ __| __\ \ /\ / / / __| __|
| (_| | | | \__ \ |_ \ V  V /| \__ \ |_
 \__,_|_| |_|___/\__| \_/\_/ |_|___/\__| {20180623}

Processing 5392 domain variants ...

Connection error using ssdeep

Can someone help me please? I'm getting a connection error when trying to use the ssdeep option. I'm running on Ubuntu 16.04 with python 2.7.

UnicodeEncodeError: 'latin-1' codec can't encode character

I am having the error "UnicodeEncodeError: 'latin-1' codec can't encode character" after several weeks of using dnstwist without error and just updating it to the latest release version.

Below is an example:

$ dnstwist -r -m example.com
_ _ _ _
| | __ | | ()| |
/ ` | ' / __| __\ \ /\ / / / __| __|
| (
| | | | __ \ |_ \ V V /| __ \ |_
_,|| ||/_| _/_/ ||_/_| {20201022}

Processing 1650 permutations ▒▒▒▒▒30%▒52%▒77%▒▒▒▒98%▒▒▒ 387 hits

Traceback (most recent call last):
File "/usr/bin/dnstwist", line 33, in
sys.exit(load_entry_point('dnstwist==20201022', 'console_scripts', 'dnstwist')())
File "/usr/lib/python3/dist-packages/dnstwist.py", line 956, in main
print(create_cli(domains))
UnicodeEncodeError: 'latin-1' codec can't encode character '\u0229' in position 3628: ordinal not in range(256)

I have the same problem if I go to the site-packages folder and execute the script explicitly with python 3.8.5.
I installed the latest version using apt.

What could be causing this error?

Chained arguements

I am attempting to retrieve a list of registered results off of a scan. I want to put it in idle format as I am using that list in another tool. However, if I want to do something such as:
$dnstwist -r domain.com --format idle
This returns just idle format of a non registered scan or so it seems. I have run a registered scan using:
$dnstwist -r domain.com
this returns some reachable websites.

All I am saying is that chaining arguments together doesn't seem to work.

install_requires in setup.py

It would be nice if you could provide install_requires in the setup.py file to specify the requirements. There are tools (like pip/pip3 and Debian's dh-python) that use the dependency information from this entry. For more information on this, have a look at the discussion section of the Python Packaging User Guide.

UnicodeEncodeError: 'charmap' codec can't encode character

I'm a Python novice, but running under Windows, I think I'm encountering a conflict between Windows-1252 and ISO-8859-1 character sets.

python.exe dnstwist.py c.com

dnstwist (20150616) by [email protected]
NOTICE: missing dnspython module - functionality is limited !
Processing 8 domains .....!!.

Bitsquatting b.com -
Bitsquatting a.com -
Bitsquatting g.com -
Bitsquatting k.com -
Bitsquatting s.com -
Traceback (most recent call last):
File "_Active\Security\DNS\dnstwist\dnstwist.py", line 208, in
sys.stdout.write('%-20s %-20s %s' % (i['type'], i['domain'], dns))
File "C:\Python34\lib\encodings\cp437.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\xe3' in position 21: character maps to

DomainFuzz generation for IDNA encoded domains fails

>>> from dnstwist import DomainFuzz
>>> domain = 'xn--9krx6rh9s.com'
>>> fuzz = DomainFuzz(domain)
>>> fuzz.generate()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "dnstwist.py", line 483, in generate
    self.__filter_domains()
  File "dnstwist.py", line 275, in __filter_domains
    if self.__validate_domain(d['domain-name']) and d['domain-name'] not in seen:
  File "dnstwist.py", line 263, in __validate_domain
    if len(domain) == len(domain.encode('idna')) and domain != domain.encode('idna'):
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/idna.py", line 164, in encode
    result.append(ToASCII(label))
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/idna.py", line 92, in ToASCII
    raise UnicodeError("Label starts with ACE prefix")
UnicodeError: Label starts with ACE prefix

License?

Currently, there's no license on the code. Do you intend to release it as GPL or BSD/MIT or something else, or just keep it as-is?

Change shebang to python2

As per PEP 394, the shebang python should only be used if the script is compatible with both python2 and python3.

For this program the shebang should be changed to:

#!/usr/bin/env python2

Python 3 incompatible

Currently the script is incompatible to python 3. It throws following error:

  File "dnstwist.py", line 97
    print 'dnstwist (' + __version__ + ') by [email protected]'
                     ^
SyntaxError: invalid syntax

I use

Python 3.4.2 (v3.4.2:ab2c023a9432, Oct  6 2014, 22:16:31) [MSC v.1600 64 bit (AMD64)] on win32

Adding parenthesis to all prints that do not have them solves the issue. I don't know if it's backwards compatible that way, but I whink python 3 should be preferred over python 2.

Missing module DNSPython

Hi,
I use
./dnstwist.py --registered eridon.ua --tld tld-list-full-2021.txt > /tmp/test.txt
or
./dnstwist.py --registered eridon.ua --tld tld-list-full-2021.txt | grep 'com.ua'
or
./dnstwist.py --registered eridon.ua --tld tld-list-full-2021.txt | wc -l

and I got error
Notice: Missing module DNSPython (DNS features limited)

dnstwist 20201022 test failure

👋 when I tried update the source code archive for homebrew formula to use pypi source, there is some issue with the test build as shown below:

==> /usr/local/Cellar/dnstwist/20201022/bin/dnstwist -rsw --thread=1 brew.sh
Traceback (most recent call last):
  File "/usr/local/Cellar/dnstwist/20201022/libexec/bin/dnstwist", line 962, in <module>
    main()
  File "/usr/local/Cellar/dnstwist/20201022/libexec/bin/dnstwist", line 936, in main
    if whoisq.creation_date:
AttributeError: 'NoneType' object has no attribute 'creation_date'

relates to Homebrew/homebrew-core#67654

Adding `--format json` causes the cli tool to error out

macOS 12 / brew install

$ dnstwist --format json maximumfun.org
usage: /usr/local/Cellar/dnstwist/1.02_2/libexec/bin/dnstwist [OPTION]... DOMAIN
dnstwist: error: unrecognized arguments: --format maximumfun.org
$ dnstwist maximumfun.org --format json
usage: /usr/local/Cellar/dnstwist/1.02_2/libexec/bin/dnstwist [OPTION]... DOMAIN
dnstwist: error: unrecognized arguments: --format json

Nope, these were wrong:
macOS 12 / brew install
ubuntu docker / cloned repo

Bad character encoding in non-cli output format modes

When exporting to formats other than cli - json, csv, or idle - non-latin characters are improperly encoded, leading to substitution.

For example the homoglyph fuzzer's first result for twisting 'abc.com' will be output as 'xn--ab-wvb81j.com' (versus the cli output 'ab̔с.com') when output format is set to anything other than cli. Not Python fluent otherwise would have tried to contribute a fix.

urlcrazy

is it based on urlcrazy or is it my idea?

Not all second level domains are detected.

While running dnstwist against alexa rank 155 domain "hatena.ne.jp" only the "ne" portion of the domain was modified in the output.
Behavior against "bbc.co.uk" was as expected, with the "bbc" portion being modified.
Similar to "co.uk", "ne.jp" is not available for use by users as explained here https://www.ne.jp/ .
"ne" is a second level domain for ".jp" and as such the "hatena" portion of the query should have been modified.
The list of all common second level domains can be found on wikipedia https://en.wikipedia.org/wiki/Second-level_domain .
I am not certain if this problem happens across other second level domains, but it would improve the tool if all common second level domains were detected.

Mixed parameter use, sometimes different results, and more questions.

Hi,

  1. Thank you for your really cool product.
  2. Please help me with some questions:
    a) How I can use options --tld and all mutations together.
    for example in result:
    domain.com
    domain.com.ua
    domain.org
    domain.org.ua
    domuin.com
    domuin.com.ua
    dimain.org.ua
    b) I use your product and count the numbers of strings in the result and compare the previous count of string and a new count of string.
    When changed I send the alarm. But sometimes I see flapping in the results, one or two domain disappears and appears in the results...
    Why? How can i avoid this?

Dict entries with numbers not tried

I am aware that no TLDs currently have numbers in them, however when using custom entries in a dictionary file containing digits they are not attempted when the file is called to make more permutations.

for example:

abc-123
abc123

UnicodeEncodeError

Running on Ubuntu 16.04, I was getting:

Traceback (most recent call last):
  File "dnstwist.py", line 1009, in <module>
    main()
  File "dnstwist.py", line 1003, in main
    p_cli(generate_cli(domains))
  File "dnstwist.py", line 120, in p_cli
    sys.stdout.write(data)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u1ecf' in position 2994: ordinal not in range(128)

I fixed it with the attached patch.

dnstwist.py.patch.txt

check from file

Hi, thanks for the tool

i would like ask if you can add the option for run the check from a file... in that way i can add some domains in one big check for domains and only one output.

regards

Typo in azerty keyboard layout

There is a slight typo in the azerty keyboard layout resulting in some of the keyboard permutations to be wrong/missing. PR #56 fixes this.

multi-letter homoglyph

Homoglyph faceb0ok.com 199.59.243.120
Homoglyph facebo0k.com 75.126.104.241

What about detecting faceb00k.com?

"--only-new" option?

I was wondering if somebody worked on a patch to implement a command-line argument like "--only-new" to report only newly discovered domains (between different runs of dnstwist.py). Or did you implement another technique?

Registered Domains without IN A record will be ignored

Due to the use of gethostbyname Domainnames that are registered/delegated but for which no IN A record has been set will get listed as "empty". Potentially those can sport the relevant IN A records at times when say their phishing website is temporarily up and appear "harmless" in dnstwist otherwise.

As http://www.heise.de/newsticker/meldung/Dnstwist-variiert-und-testet-Domainnamen-2690418.html [DE] has pitched this project a little bit of documentation pointing out the limits of dnstwist might be nice.

[Question] Running dnstwist inside another script

Hi,

I'd like to run dnstwist inside another script with some additional parameters.

import dnstwist
fuzz = dnstwist.DomainFuzz("google.com")
fuzz.generate()
fuzz.domains

[{'domain-name': 'google.com', 'fuzzer': 'Original*'}, {'domain-name': 'googlea.com', 'fuzzer': 'Addition'}, {'domain-name': 'googleb.com', 'fuzzer': 'Addition'}, {'domain-name': 'googlec.com', 'fuzzer': 'Addition'}, {'domain-name': 'googled.com', 'fuzzer': 'Addition'}, {'domain-name': 'googlee.com', 'fuzzer': 'Addition'}, {'domain-name': 'googlef.com', 'fuzzer': 'Addition'}, {'domain-name': 'googleg.com', 'fuzzer': 'Addition'}, 
..................
..................
 'fuzzer': 'Transposition'}, {'domain-name': 'googel.com', 'fuzzer': 'Transposition'}, {'domain-name': 'gaogle.com', 'fuzzer': 'Vowel swap'}, {'domain-name': 'geogle.com', 'fuzzer': 'Vowel swap'}, {'domain-name': 'googlo.com', 'fuzzer': 'Vowel swap'}, {'domain-name': 'googli.com', 'fuzzer': 'Vowel swap'}, {'domain-name': 'guogle.com', 'fuzzer': 'Vowel swap'}, {'domain-name': 'gougle.com', 'fuzzer': 'Vowel swap'}, {'domain-name': 'goegle.com', 'fuzzer': 'Vowel swap'}, {'domain-name': 'goagle.com', 'fuzzer': 'Vowel swap'}, {'domain-name': 'wwgoogle.com', 'fuzzer': 'Various'}, {'domain-name': 'wwwgoogle.com', 'fuzzer': 'Various'}, {'domain-name': 'www-google.com', 'fuzzer': 'Various'}, {'domain-name': 'googlecom.com', 'fuzzer': 'Various'}]

The above code works fine but how can I add the following parameters to the above code?

--debug
--registered
--format=json
--tld=./abused_tlds.txt

Thanks

TLD and domains second level

Hi!
I created TLD-list with domains second level but when I used it, I didn't get a result with domain from my tld-list.
Why?

ssdeep missing

I have ssdeep already installed, but dntwist gives me this error:

dnstwist.py: notice: missing module: ssdeep (fuzzy hashes not available)

I'm on latest Kali version. Any idea how to fix?

Homoglyph detection for '11' in place of 'll' not working

I've tried to figure out why this isn't working since there is already a glyph for '1' in place of an 'l', but I haven't come up with a solution. Dnstwist is properly detecting the other double homoglyph of 'ii' in place of 'll', but not '11'.
Homoglyph weiipoint.com -

For reference, we11point.com was used in the phishing attack against Anthem BCBS, announced earlier this year - http://www.threatconnect.com/the-anthem-hack-all-roads-lead-to-china/.

Replace GeoIP Legacy by GeoIP2

The GeoIP module, nowadays called "GeoIP Legacy", alongside the old-style databases (.dat files) are deprecated, and have been for some time. The free-as-in-beer databases in that format (GeoLite Legacy) were discontinued in January 2019, and the paid GeoIP Legacy versions are being retired in May 2022.

The replacement stack exists since 2014, and is by the same author (MaxMind), and fairly mature. These comprise the MMDB file format (an openly licensed, well-documented format), alongside the MaxMindDB libraries, in C, Python etc. In Python, the MaxMindDB library is an abstraction for reading the databases on disk (like the old GeoIP module was), while the GeoIP2 module is one level of abstraction further, allowing one to use either the on-disk databases or their web services instead.

I was wondering if dnstwist could be adapted to use GeoIP2. Thank you for your efforts!

WHOIS

WHOIS not working with command

Cyrillic domains support?

Hi!
Thanks for your project. Could you add support for сyrillic domains?

VALID_FQDN_REGEX = re.compile(r'(?=^.{4,253}$)(^((?!-)[a-zA-Z0-9-]{1,63}(?<!-)\.)+[a-zA-Z]{2,63}$)', re.IGNORECASE) doesn't match cyrillic domains, for example дом.рф or xn--d1aqf.xn--p1ai.
So i get output like this:

user@ubuntu:~$ dnstwist дом.рф
usage: /home/user/.local/bin/dnstwist [OPTION]... DOMAIN
dnstwist: error: invalid domain name: дом.рф
user@ubuntu:~$ dnstwist xn--d1aqf.xn--p1ai
usage: /home/user/.local/bin/dnstwist [OPTION]... DOMAIN
dnstwist: error: invalid domain name: xn--d1aqf.xn--p1ai

You have рф here https://github.com/elceef/dnstwist/blob/master/dictionaries/common_tlds.dict but it never gets into the results.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.