thomaspatzke/wase

The Web Audit Search Engine - Index and Search HTTP Requests and Responses in Web Application Audits with ElasticSearch

WASE

WASE is a shortcut for Web Audit Search Engine. It's a framework for indexing HTTP requests/responses while web application audits in an ElasticSearch instance and enriching it with useful data. The indexed data can then be searched and aggregated with ElasticSearch queries or with Kibana.

Currently WASE contains the following parts:

• doc_HttpRequestResponse.py: a library that implements the DocHTTPRequestResponse class. This class is an elasticsearch_dsl-based storage class of HTTP requests/responses (derived from Burps data structures and API).
• ElasticBurp: a Burp plugin that feeds requests/responses into ElasticSearch.

ElasticBurp

Scared about the weak searching performance of Burp Suite? Are you missing possibilities to search in Burp? ElasticBurp combines Burp Suite with the search power of ElasticSearch. It can be installed directly from the Burp BApp Store.

Installation

1. Install ElasticSearch and Kibana.
2. Configure both - For security reasons it is recommend to let them listen on localhost:

• Set network.host: 127.0.0.1 in /etc/elasticsearch/elasticsearch.yml.
• Set host: "127.0.0.1" in /opt/kibana/config/kibana.yml.

3. Install dependencies in the Jython environment used by Burp Extender with: $JYTHON_PATH/bin/pip install -r requirements.txt
4. Load ElasticBurp.py as Python extension in Burp Extender.

Currently there seem to be incompatibilities with the new Python Elasticsearch packages. Specify the 2.2 version when installing with pip: $JYTHON_HOME/bin/pip install elasticsearch_dsl==2.2

Usage

See this blog article for usage examples.

WASEProxy

A generic intercepting HTTP(S) proxy server that stores extracted data into an ElasticSearch index.

Installation with pip: pip install -r requirements-proxy.txt

WASEQuery

Search ElasticSearch indices created by WASE for

• responses with missing headers
• responses with missing parameters
• all values that were set for a header (e.g. X-Frame-Options, X-XSS-Protection, X-Content-Type-Options, Content-Security-Policy, ...)

...or do arbitrary search queries.

Invoke WASEQuery.py for help message. This blog article shows some examples for usage of WASEQuery.

wase's Issues

Timezone information error on Windows

When running under jython, it appears unix.py recognizes the system as non-win32 and as such tries to look for timezone information in unix locations. I copied a \etc\localtime file from a linux machine to C:\etc\localtime and that seems to work ok. I'm not sure if or where this would be fixed but it might be worth noting in the extension docs for those trying the extension on windows. It was the first error I ran into and prevented the extension from loading.

Traceback (most recent call last):
File "C:\Users\user1\AppData\Roaming\BurpSuite\bapps\67f5c31f93d04ad3a3b0a1808b3648fa\ElasticBurp.py", line 22, in
from doc_HttpRequestResponse import DocHTTPRequestResponse
File "C:\Users\user1\AppData\Roaming\BurpSuite\bapps\67f5c31f93d04ad3a3b0a1808b3648fa\doc_HttpRequestResponse.py", line 26, in
tz = get_localzone()
File "C:\Users\user1\AppData\Roaming\BurpSuite\bapps\67f5c31f93d04ad3a3b0a1808b3648fa\BappModules\tzlocal\unix.py", line 123, in get_localzone
_cache_tz = _get_localzone()
File "C:\Users\user1\AppData\Roaming\BurpSuite\bapps\67f5c31f93d04ad3a3b0a1808b3648fa\BappModules\tzlocal\unix.py", line 117, in _get_localzone
raise pytz.UnknownTimeZoneError('Can not find any timezone configuration')
pytz.exceptions.UnknownTimeZoneError: 'Can not find any timezone configuration'

at org.python.core.PyException.doRaise(PyException.java:226)

TypeError: unicode indices must be integers

Hello @thomaspatzke ,

Thanks for writing this wonderful extension. I'm getting exceptions while connecting it to elastic server.

• I have elastic server running on localhost:9200 with cluster name mycluster1

Tested on BApp version of elasticburp.

Connecting to 'localhost:9200', index 'wase-burp'

Errors

Traceback (most recent call last):
  File "C:\Users\arbaz\AppData\Roaming\BurpSuite\bapps\67f5c31f93d04ad3a3b0a1808b3648fa\ElasticBurp.py", line 64, in registerExtenderCallbacks
    self.applyConfig()
  File "C:\Users\arbaz\AppData\Roaming\BurpSuite\bapps\67f5c31f93d04ad3a3b0a1808b3648fa\ElasticBurp.py", line 81, in applyConfig
    JOptionPane.showMessageDialog(self.panel, "<html><p style='width: 300px'>Error while initializing ElasticSearch: %s</p></html>" % (str(e)), "Error", JOptionPane.ERROR_MESSAGE)
  File "C:\jython2.7.0\Lib\site-packages\elasticsearch\exceptions.py", line 55, in __str__
    cause = ', %r' % self.info['error']['root_cause'][0]['reason']
  File "C:\jython2.7.0\Lib\site-packages\elasticsearch\exceptions.py", line 55, in __str__
    cause = ', %r' % self.info['error']['root_cause'][0]['reason']
TypeError: unicode indices must be integers

	at org.python.core.Py.TypeError(Py.java:259)
	at org.python.core.SequenceIndexDelegate.checkIdxAndFindItem(SequenceIndexDelegate.java:74)
	at org.python.core.SequenceIndexDelegate.checkIdxAndGetItem(SequenceIndexDelegate.java:61)
	at org.python.core.PySequence.seq___getitem__(PySequence.java:377)
	at org.python.core.PySequence.__getitem__(PySequence.java:373)
	at elasticsearch.exceptions$py.__str__$8(C:\jython2.7.0\Lib\site-packages\elasticsearch\exceptions.py:58)
	at elasticsearch.exceptions$py.call_function(C:\jython2.7.0\Lib\site-packages\elasticsearch\exceptions.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:167)
	at org.python.core.PyBaseCode.call(PyBaseCode.java:138)
	at org.python.core.PyFunction.__call__(PyFunction.java:413)
	at org.python.core.PyMethod.__call__(PyMethod.java:126)
	at org.python.core.PyMethod.__call__(PyMethod.java:117)
	at org.python.core.PyBaseExceptionDerived.__str__(PyBaseExceptionDerived.java:65)
	at org.python.core.PyString.str_new(PyString.java:164)
	at org.python.core.PyString$exposed___new__.new_impl(Unknown Source)
	at org.python.core.PyType.invokeNew(PyType.java:494)
	at org.python.core.PyType.type___call__(PyType.java:1706)
	at org.python.core.PyType.__call__(PyType.java:1696)
	at org.python.core.PyObject.__call__(PyObject.java:461)
	at org.python.core.PyObject.__call__(PyObject.java:465)
	at org.python.pycode._pyx6.applyConfig$3(C:\Users\arbaz\AppData\Roaming\BurpSuite\bapps\67f5c31f93d04ad3a3b0a1808b3648fa\ElasticBurp.py:81)
	at org.python.pycode._pyx6.call_function(C:\Users\arbaz\AppData\Roaming\BurpSuite\bapps\67f5c31f93d04ad3a3b0a1808b3648fa\ElasticBurp.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:167)
	at org.python.core.PyBaseCode.call(PyBaseCode.java:138)
	at org.python.core.PyFunction.__call__(PyFunction.java:413)
	at org.python.core.PyMethod.__call__(PyMethod.java:126)
	at org.python.pycode._pyx6.registerExtenderCallbacks$2(C:\Users\arbaz\AppData\Roaming\BurpSuite\bapps\67f5c31f93d04ad3a3b0a1808b3648fa\ElasticBurp.py:64)
	at org.python.pycode._pyx6.call_function(C:\Users\arbaz\AppData\Roaming\BurpSuite\bapps\67f5c31f93d04ad3a3b0a1808b3648fa\ElasticBurp.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:167)
	at org.python.core.PyBaseCode.call(PyBaseCode.java:307)
	at org.python.core.PyBaseCode.call(PyBaseCode.java:198)
	at org.python.core.PyFunction.__call__(PyFunction.java:482)
	at org.python.core.PyMethod.instancemethod___call__(PyMethod.java:237)
	at org.python.core.PyMethod.__call__(PyMethod.java:228)
	at org.python.core.PyMethod.__call__(PyMethod.java:218)
	at org.python.core.PyMethod.__call__(PyMethod.java:213)
	at org.python.core.PyObject._jcallexc(PyObject.java:3626)
	at org.python.core.PyObject._jcall(PyObject.java:3658)
	at org.python.proxies.__main__$BurpExtender$6.registerExtenderCallbacks(Unknown Source)
	at burp.b0h.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:745)
Traceback (most recent call last):
  File "C:\Users\arbaz\AppData\Roaming\BurpSuite\bapps\67f5c31f93d04ad3a3b0a1808b3648fa\ElasticBurp.py", line 93, in applyConfigUI
    self.applyConfig()
  File "C:\Users\arbaz\AppData\Roaming\BurpSuite\bapps\67f5c31f93d04ad3a3b0a1808b3648fa\ElasticBurp.py", line 81, in applyConfig
    JOptionPane.showMessageDialog(self.panel, "<html><p style='width: 300px'>Error while initializing ElasticSearch: %s</p></html>" % (str(e)), "Error", JOptionPane.ERROR_MESSAGE)
  File "C:\jython2.7.0\Lib\site-packages\elasticsearch\exceptions.py", line 55, in __str__
    cause = ', %r' % self.info['error']['root_cause'][0]['reason']
  File "C:\jython2.7.0\Lib\site-packages\elasticsearch\exceptions.py", line 55, in __str__
    cause = ', %r' % self.info['error']['root_cause'][0]['reason']
TypeError: unicode indices must be integers

TCP port exhaustion

When enabling the plugin and sending multiple burp requests to elasticsearch via context window, each request will cause a new http requests to be made from the burp client to the elasticsearch server. The tcp connection will not be torn down and will sit in a wait state after data has been sent. This leads to port tcp port exhaustion on the client and server. Is there a way to resolve this or do some type of connection pooling so the http requests go over a single tcp connection? I didn't find anything so far but I'm new to ES so I may be missing something. My ES install is a clean 16.04 ubuntu, and client is windows server 2012.

ElasticSearch 7.12.0 responds with "Content-Type header [] is not supported"

There appears to be a requirement from elasticsearch 7.12.0 to have a content-type header.

The configuration used here is the docker image for elasticsearch and kibana 7.12.0 in a network, a separate container running DVWA and burp pro on the local host pointed at elasticsearch.

The post from elastic burp does not include a Content-Type header

The response from elasticsearch gives the impression that the Content-Type header is necessary

A text file following the HTTP thread. The data is from a burp scan of DVWA (no sensitive date)
elasticBurp.txt

Docker command used to run DVWA
docker run --name=dvwa -d vulnerables/web-dvwa

Docker-Compose Yaml file for ElasticSearch and Kibana
docker-compose.yml.txt

pymiproxy not in requirements.txt

It looks like the pymiproxy requirement for WASEProxy.py isn't in requirements.txt. Let me know if you'd like a PR to add pymiproxy to the current file or a separate requirements file for the proxy.

# pip install -r requirements.txt
# ./WASEProxy.py -h
Traceback (most recent call last):
  File "./WASEProxy.py", line 4, in <module>
    from miproxy.proxy import ProxyHandler, MitmProxy, AsyncMitmProxy
ImportError: No module named miproxy.proxy

"OnlyResp" Conf value will always return True

In this line the expression will always return true since the default value is true and the configuration value, while a bool initially, is converted to a string "True" or "False" at conf save and restoration.

WASE/ElasticBurp.py

Line 52 in b8c70f6

self.confBurpOnlyResp = bool(self.callbacks.loadExtensionSetting("elasticburp.onlyresp") or Burp_onlyResponses)

The value needs to be changed from string "True"/"False" to bool True/False. Is it better to include another library (distutils.util.strtobool()) or add a helper function for this purpose?

*Actually I think the code also needs changed in that before it was checking for None the defaulting to True. Now if the first is True/False and "or True" it will also always return True even if str2bool is fixed.

Installing Requirements-proxy break Encrypted connections

When I tried to install elastic-burp i encountered an error while installing the requirements. Within the requirements-proxy the dependency cryptography==2.8 is specified.

This leads to error: ERROR: pyopenssl 19.1.0 has requirement cryptography>=2.8, but you'll have cryptography 1.8.1 which is incompatible.

After that pip does not work anymore due to the following error: Error WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),)': /simple/elasticsearch/

Steps to reproduce

virtualenv venv
source venv/bin/activate
pip install -r requirements-proxy.txt # includes the dependency error 
pip install -r -r requirements-proxy.txt # Repeat of the previously working command (now fails)

(My) solution

Removing the stickied version from cryptography within the requirements-proxy.txt seemed to resolve the problem.

New requirements-proxy.txt content:

cryptography==2.8
elasticsearch==5.1.0
elasticsearch-dsl==5.1.0
pymiproxy==1.0
pyparsing==2.2.0
six==1.10.0
tzlocal==1.3

Pip freeze output afterwards

cffi==1.14.0
cryptography==2.8
elasticsearch==6.0.0
elasticsearch-dsl==6.0.0
enum34==1.1.9
ipaddress==1.0.23
pycparser==2.19
pymiproxy==1.0
pyOpenSSL==19.1.0
pyparsing==2.2.0
python-dateutil==2.8.1
pytz==2019.3
six==1.10.0
tzlocal==1.3
urllib3==1.22

Remarks

On a sidenote, there is still elasticsearch==5.1.0 and elasticsearch-dsl==5.1.0 specified within the requirements-proxy.txt, which causes Issue #12 for the Burp plugin.