hhucn/webvulnscan

automated web application vulnerability scanner

webvulnscan

Quickstart

$ git clone https://github.com/hhucn/webvulnscan.git
$ cd webvulnscan
$ python -m webvulnscan http://example.target/

What is it?

As the name suggests, webvulnscan is (or wants to be someday) a security scanner for Web Applications with the intent of automatic testing, licensed under the MIT-License. It's written in Python(compatible with 2.7 and 3.3) and doesn't require any external libraries.

Features

• Link & Form Crawling
• Detection for XSS, CRSF, Breach, Clickjacking and cacheable Cookies
• White- and Blacklisting of Pages
• Authentification

Examples

vulnsrv

vulnsrv is sample exploitable website for educational purposes. We will use it here as an example:

$ wget https://raw.github.com/phihag/vulnsrv/master/vulnsrv.py
$ python vulnsrv.py

It's running now under http://localhost:8666/ on your computer. Open now a new console for running webvulnsrv. Assuming that you are in your home directory and already cloned webvulnscan...

$ cd webvulnscan
$ python -m webvulnscan http://localhost:8666/
Vulnerability: CSRF under http://localhost:8666/csrf/send
Vulnerability: XSS on http://localhost:8666/xss/?username=Benutzer%21 in parameter username

You may notice that this aren't all vulnerabilties, but webvulnsrv is still a work in progress.

Specific scanning

If you want to scan only for specific vulnerabilities(for example, only for BREACH), you simply try the following:

$ python -m webvulnscan --breach http://localhost:8666/

or you want to scan for XSS and CSRF vulnerabilities:

$ python -m webvulnscan --xss --csrf http://localhost:8666/

What if you want to be more specific? What if you want to test only one site? Use --no-crawl

$ python -m webvulnscan --no-crawl http://localhost:8666/

And the links will be ignored. However, Forms are not.

White- and Blacklisting

Sometimes, you have links on a site you that you want to test to. But the default whitelist points only on the host of the given link. Here's how you can add more:

$ python -m webvulnscan --whitelist http://ex.am.ple/ http://localhost/

However, what if you want to use Authentification and there's a /logout-Link? If the crawler hits it, the session is lost. Simply Blacklist it!

$ python -m webvulnscan --blacklist logout http://localhost/

And the site will be never visited. Please note that the blacklist Parameter accepts Regular Expressions, the python version.

Authentification

We have a login handler under /perform_login which wants the post-fields username and password, who can we log in? The account we want to use has the username "abc" and password "123456". The command would look like the following:

$ python -m webvulnscan --auth http://no.tld/perform_login --auth-data username=abc --auth-data password=123456 http://no.tld/

Yes, you have to use the --auth-data option for every field you want to send.

Configuration

As you see, there you end up with a lot of parameters in the end. To avoid typing so much, you can add the --write-out-Option and

$ python -m webvulnscan --write-out=example.conf http://localhost:8666/

save it to a file. If you want to rerun the test because you (think you) fixed it, simply run:

$ python -m webvulnscan -c example.conf

webvulnscan's Issues

Rename get_url_parameters

page.get_url_parameters sounds like a callable, but it isn't. Rename it to url_parameters.

Specific scanning doesn't work

Currently specific scanning for --exotic_characters is completly ignored. This has to be fixed.

Write a attack module which checks for timing attacks

Timing attacks occur when the web server responds in other time lengths when the user is accesing them, based on the input data, as example, a Password. They can be used for more efficient bruteforcing or to determine the length of data. As we also wanted to check that GET-Requests are idempotent, this can be done in one module.

Intresting Links:

• http://www.stanford.edu/~dabo/papers/webtiming.pd
  f
• http://pentestmonkey.net/tools/timing-attack-checker

Add CLRF-Attacker

CLRF-Injection occurs when a Attacker is able to split the HTTP-Response using in %0d%0a-Sequence in a reflected field.

Intresting links about it:

• http://cwe.mitre.org/data/definitions/93.html
• https://www.owasp.org/index.php/CRLF_Injection

drive_all should keep a list of already attacked Forms.

Currently, every Form is attacked as often as it occurs. This is simply unnecessary, the form-action should be logged in already_visited.

New attacker module which scans for HTTP Header Pollution

HTTP Header Pollution is a attack where a HTTP Parameter(Post and GET) is repeated again. As example, http://localhost/xss?username=nowhere&username=shown. The only valid response would be a error, however it isn't some pages only return the last/first occurence of this string. This should be checked.

Intresting Links:

• http://www.owasp.org/images/b/ba/AppsecEU09_CarettoniDiPaola_v0.8.pdf
• http://www.acunetix.com/blog/whitepaper-http-parameter-pollution/
• http://capec.mitre.org/data/definitions/460.html

Blacklisting is ignored by get_forms().

Handle KeyboardInterrupt

If the user uses ^C, we will get a long traceback. This isn't really necessary, the application should simply exit.

Rename clrf

It should be crlf (Carriage Return / Line Feed).

Write a attack module which checks session-lengths and tokens

It would be intresting, to do this because it is commonly handled by the various frameworks and not the "real" application. However it should be detected to do this. It is possible to determine the exact time, using a equation, avaible here. I think 100 for A is a reasonable amount of possible requests and we can assume that there is one user for S. B should be calculateable.

Intresting Links:

• https://www.owasp.org/index.php/Insufficient_Session-ID_Length
• http://de.wikipedia.org/wiki/Entropie_%28Informationstheorie%29
• Real world example: http://cwe.mitre.org/data/definitions/6.html
• http://www.cgisecurity.com/lib/sessionids.pdf
• http://cookies.lcs.mit.edu/pubs/webauth:tr.pdf

The length of the session expiration should be also checked.

Command-line options should override configuration file options

When using a configuration file and the command line (i.e. passing something like

python -m webvulnscan -c cfg.json --xss http://localhost:8666/

) the command line options should be applied after the configuration file (in this case, the user wants to use the auth data from the configuration file, but clearly only wants the XSS tests to run).

Filter output

Currently a scanning of a page can result in a lot of messages, as example if developer forget X-Frame-Options or Cache-Control headers. vulnsrv results in the following:

[theron@localhost webvulnscan]$ python -m webvulnscan --blacklist reset http://localhost:8666/
Vulnerability: Clickjacking under http://localhost:8666/ no X-Frame-Options!
Vulnerability: Implicit Cacheable Cookies under http://localhost:8666/
Vulnerability: Clickjacking under http://localhost:8666/mac/ no X-Frame-Options!
Vulnerability: Implicit Cacheable Cookies under http://localhost:8666/mac/
Vulnerability: Clickjacking under http://localhost:8666/pathtraversal/ no X-Frame-Options!
Vulnerability: Implicit Cacheable Cookies under http://localhost:8666/pathtraversal/
Vulnerability: Clickjacking under http://localhost:8666/sqlinjection/ no X-Frame-Options!
Vulnerability: Implicit Cacheable Cookies under http://localhost:8666/sqlinjection/
Vulnerability: Vulnerability: XSS under http://localhost:8666/xss/?username=%3Cscript%3Ealert%28%22XSS_STRING%22%29%3B%3C%2Fscript%3E in URL parameter username
Vulnerability: Clickjacking under http://localhost:8666/xss/?username=Benutzer%21 no X-Frame-Options!
Vulnerability: Implicit Cacheable Cookies under http://localhost:8666/xss/?username=Benutzer%21
Vulnerability: Clickjacking under http://localhost:8666/csrf/ no X-Frame-Options!
Vulnerability: Implicit Cacheable Cookies under http://localhost:8666/csrf/
Vulnerability: Clickjacking under http://localhost:8666/clientauth/ no X-Frame-Options!
Vulnerability: Implicit Cacheable Cookies under http://localhost:8666/clientauth/
Vulnerability: Clickjacking under http://localhost:8666/sqlinjection/msg?id=4 no X-Frame-Options!
Vulnerability: Implicit Cacheable Cookies under http://localhost:8666/sqlinjection/msg?id=4
Vulnerability: Clickjacking under http://localhost:8666/sqlinjection/msg?id=1 no X-Frame-Options!
Vulnerability: Implicit Cacheable Cookies under http://localhost:8666/sqlinjection/msg?id=1
Warning: Strange content type: image/png
Warning: Strange content type: image/svg+xml
Warning: Strange content type: image/png
Warning: Strange content type: text/x-python

Sitewide vulnerabilities should be printed once, to make the output still readable for humans. As URL should be for http://target/* these addresses.

Extend XSS tests to check for stored XSS

Sometimes, the XSS is not immediately reflected back, but instead only shown on a different page. For example, after creating an object, the user could be redirected to the created object, and only experience the XSS when editing the object again, or looking at the list of objects.

Extend travis-File to check for test coverage.

The test coverage is necessary, to ensure that the application is still maintainable and working. The test coverage should never falll below %70 percent. This should be checked automatically.

Add Ignoring Options

It is possible that the user is aware of a possible vulnerability, as example if he runs a javascript web IDE, we doesn't want to be notified of the XSS-Problems in the input field. However, we still wants to check for XSS-Attacks in other fields. There should be a option for this. The syntax could be:

--ignore xss=http://test.com/

Update README.rst

The readme should be partially rewritten to present the current state of the application.

Add a test to find non-idempotent GET targets

Sending a GET should never change the state of the application. Test that

• Links
• GET forms

do not change the application's state.

In case of Error, all results are lost.

When the site, as example, outputs invalid HTML, webvulnscan just exits and the logs are lost. It should be possible to print them, and after that the error.

More intellegent guess_value function

guess_value shouldn't ignore the minlength and default field. Add handling for them to the InputForm-Class.

Add testing for commandline options.

It should be possible, to test the command line interface, in order to avoid errors. This should be also done for configuration files.

Make webvulnscan an official HHU cn project

• Create repository (with access rights)
• Move issues
• Update local push targets on dev machines

-vv Option

The verbose Option does provide Information on what is currently tested, but @phihag has recommended that this could be extended. -vv should print every Request, simply through a modification in webvulnscan/client.py.

Add Except-Attack Option

Currently, the users is enabled to perform specific Attacks but he can't simply except one from the group. It should behave like this:

$ python -m webvulnscan --except-xss http://localhost:8666

Check for handling of exotic characters

Add a check that the application in question can handle exotic characters and does not break down. In particular, this includes

• Quotes (", ')
• Brackets (<{[()]}>)
• Ampersands and pipe symbols (&|)
• Characters outside of the BMP, such as this one
• The NUL character
• Byte sequences that are not valid UTF-8

WebTest Integration

WebTest is a Framework for testing WSGI-Application without actually performing HTTP-Communicating and has intresting features. It is intresting, because with it, it is possible to lookup the source of the application. One could traceback vulnerable sourcecode to specific functions or maybe even lines. It also offers better integration with forms, etc. However, this should be kept strictly a optional feature.

Links:

• Docs: http://webtest.readthedocs.org/
• Guide for pylons testing: http://docs.pylonsproject.org/projects/pylons-webframework/en/latest/testing.html
• Django Integration: https://pypi.python.org/pypi/django-webtest

HTML-Error messages are still not very usefull

As example, it tried to scan https://github.com/lefnire/habitrpg with webvulnscan and ended up with the following output:

Warning: No Content-Type header on http://localhost:3000/
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/theron/webvulnscan/webvulnscan/__main__.py", line 18, in <module>
   webvulnscan.main()
  File "webvulnscan/__init__.py", line 141, in main
        run(options, arguments)
  File "webvulnscan/__init__.py", line 48, in run
        for page in urls:
  File "webvulnscan/crawler.py", line 44, in __iter__
        page = self.client.download_page(link, blacklist=self.blacklist)
  File "webvulnscan/client.py", line 89, in download_page
       return Page(url, html, headers, status_code, blacklist)
  File "webvulnscan/page.py", line 18, in __init__
    self.document = self.generate_document()
  File "webvulnscan/page.py", line 26, in generate_document
    return ET.fromstring(self.html, parser)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1301, in XML
    return parser.close()
  File "webvulnscan/EtreeParser.py", line 49, in close
    return self.tb.close()
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1385, in close
    assert self._last is not None, "missing toplevel element"
AssertionError: missing toplevel element

Add an option to abort early

Sometimes, I just want to know whether my application is flawless or not. Add an option --abort-early that aborts webvulnscan after the first warning or vulnerability.

Selenium Integration

Next to supporting WebTest(see #37), Selenium would be a good oppertunity to attack JS-Based sites. Selenium seems easy to integrate, it supports xpath and other intresting plugins. It is simply necessary to write some new classes, as example SeleniumClient, SeleniumPage and add a option to use them instead of the default URLLib Classes. Like #37 it should be kept optional, the user should still only need a default python installation to run the application, but he should have the possibility to use selenium, when installed.

Links:

• Docs: http://selenium-python.readthedocs.org/en/latest/
• Pypi: https://pypi.python.org/pypi/selenium

Write attacker which tries to exploit file uploads.

After Testing with Gruyere( #44 ), File Uploading vulnerability wasn't detected. It should try the following:

• Upload .svg Files with embed Javascript
• Try to upload html code
• Try to upload executable code
  • Python
  • PHP

Add an option to show each request

In one invocation, webvulnscan crashes. I'd like to see what the last request was without having to resort to wireshark. This would also be a great preparation for #19 .

Generate a report/more beautiful output

It should be possible to generate from scan results reports in reStructuredText, as example:

========================================
http://localhost/ - Vulnerability Report
========================================
123 XSS Vulnerabilities
456 CSRF Vulnerabilities
...
15 Warnings

http://localhost/perform_login
------------------------------
Found a CSRF vulnerability

This should be handled by a seperate script and not by the module it self, as the output has been designed to be parseable. Maybe a look at NLTK as optional feature for dynamic sentence generation would be cool.

Remove dead code in Client

Currently the client still has in the functions download_page() and download() the optional parameter remember_visit. However, the history is now saved in the crawler and therefore this code has become useless. Remove it.

Unfiltered Output option

It should set the do_print option in the log module to true.

Built a system for running attacks distributed.

This shouldn't be so difficult as it sounds, basically only the Crawler-Propertys to_visit and already_visited have to be synchronized acroos all clients.

Test Gruyere

We should evaluate webvulnscan by running it against gruyere.

Check for handling of redirectors

Web application sometimes have a redirector under a specific URL which they use to monitor the user, etc. Check if these are present and warn the user when they redirect to every given site. See https://www.owasp.org/index.php/Unvalidated_Redirects_and_Forwards_Cheat_Sheet

Implement multithreading for attack driving.

It should be relatively easy to implement multitasking for driving the attacks.

Handle links to images

It is perfectly sensible to link to an image, but when webvulnscan reports:

Warning: http://localhost:8666/pathtraversal/get?file=logo.png Strange content type image/png

Allow cookie-import from Firefox

Some login mechanism are to difficult to specify using the --auth-Options so it should be possible for a user to sign in the page with his browser and export his cookies to use them with webvulnscan.

Clickjacking should try to find Javascript protection

Unicode character handling is broken in broken_unicode_characters attack

With python 2.7, running it on vulnsrv gives:

$ python -m webvulnscan http://localhost:8666/   --broken_unicode_characters
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/phihag/projects/webvulnscan/webvulnscan/__main__.py", line 18, in <module>
    webvulnscan.main()
  File "webvulnscan/__init__.py", line 232, in main
    messages = run(options, arguments)
  File "webvulnscan/__init__.py", line 98, in run
    drive_all(page, attacks, client)
  File "webvulnscan/attacks/__init__.py", line 20, in drive_all
    attack(page, client)
  File "webvulnscan/attacks/broken_unicode_characters.py", line 33, in broken_unicode_characters
    try_on_form(client, form, symbol)
  File "webvulnscan/attacks/broken_unicode_characters.py", line 11, in try_on_form
    result = form.send(client, attack_parameters)
  File "webvulnscan/form.py", line 42, in send
    return client.download_page(self.action, parameters)
  File "webvulnscan/client.py", line 86, in download_page
    status_code, html, headers = self.download(url, parameters)
  File "webvulnscan/client.py", line 48, in download
    data = urlencode(parameters).encode("utf-8")
  File "/usr/lib/python2.7/urllib.py", line 1329, in urlencode
    v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\uffff' in position 0: ordinal not in range(128)

It works fine on Python 3.

Add a attack module which scans for sessions identifiers in the url

Sessions in the url aren't a direct vulnerability, but they aren't a best practice also. As example, there is a library site which offers books to be readable on the web. They are also "saving" their sessions in the URL, now somebody wants to give somebody other a link to a specific page. Now the other somebody nows the session of the user. This could be used in the combination which Social Engineering to access information/overtake user accounts.

Related:

• https://www.owasp.org/index.php/Top_10_2013-A4-Insecure_Direct_Object_References

Make output more machinereadable

The current output would be difficult to parse. The format should look like the following:

Vulnerability: <Name of the Vulnerability> <site> <additional message>

Also, the log messages should be adjusted. The type of the vulnerability should directly be something like "XYZ Injection" and not "XYZ Vulnerability".

Ignore # in URLs

As the # symbol doesn't provide a new page, only a specific navigation point, it is complety useless to run test against them.

Exploit module

It would be an intresting option to be able to generate exploits for specific vulnerabilities. This could be used for testing to avoid false prositives or simply for easier demonstration.

Rewrite attacks to use Object-Orientation

It should be more abstractred and have the following functions:

• search()
• run()

Authentication via form fill

This should allow the user to instead of a specifing a post target, allow him to set a site, specify the form and filling some inputs manually. This could be handy to avoid CSRF-Tokens in application. As Example:

$ python -m webvulnscan --form-page http://test/login --form-id login --form-data field=value

Make output order determined

Currently, the order of the default output is random (due to hash randomization only being implemented in newer cPython versions, this can only be observed in 3.3+). That can lead to differently sorted outputs even if the whole scanning process is deterministic, which is confusing. We should always have the same order in the output.

Tests should be run with python2 and python3

As we want to support both versions of python, it would be logically to test with both. This should also be done via travis.

Simplify EtreeParser

EtreeParser doesn't care about the URL, it should be simply given a warn or log method.