mushorg / glastopf Goto Github PK
View Code? Open in Web Editor NEWWeb Application Honeypot
Home Page: http://glastopf.org
Web Application Honeypot
Home Page: http://glastopf.org
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 121, in regular_generate_dork
self.generate_dork_pages()
File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 98, in generate_dork_pages
with codecs.open("{0}/{1}".format(self.pages_path, page_md5), "w", "utf-8") as dork_file:
File "/usr/lib/python2.7/codecs.py", line 881, in open
file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 13] Permission denied: '/opt/honeypot/data/dork_pages/942af692e6b84eae76b8f54b9da9219b'
It would be useful to have a request path -> web application mapping at least for the major web applications. E.g. wp-content belongs to WordPress. This way we can customize the honeypots response (server a WordPress template instead of the default one).
Downloading/unpacking glastopf
Downloading Glastopf-3.0.0.tar.gz (531Kb): 531Kb downloaded
Running setup.py egg_info for package glastopf
The required version of distribute (>=0.6.35) is not available,
and can't be installed while this script is running. Please
install a more recent version first, using
'easy_install -U distribute'.
(Currently using distribute 0.6.24dev-r0 (/usr/lib/python2.7/dist-packages))
Complete output from command python setup.py egg_info:
The required version of distribute (>=0.6.35) is not available,
This currently is not really remote but fetches data from the local collection. Should be replaced to get the actual data from http://exploit-db.com
Current code: https://github.com/glastopf/glastopf/blob/master/modules/handlers/emulators/dork_list/remote_exploits.py
I've got a relatively fresh install of Ubuntu 12.04 and followed the glastopf Ubuntu setup guide. When I try to start up glastopf, I get an error right after the message about bootstrapping the dork database. Here is the output.
:/opt/glastopf$ sudo python webserver.py
2013-02-27 12:29:35,272 (root) Webserver running on: 0.0.0.0:8080 waiting for connections.
2013-02-27 12:29:35,273 (glastopf) Starting Glastopf (Git: 42529b4)
2013-02-27 12:29:35,274 (glastopf) Connecting to main database with: sqlite:///db/glastopf.db
2013-02-27 12:29:35,286 (modules.handlers.emulators.dork_list.dork_page_generator) Bootstrapping dork database.
Traceback (most recent call last):
File "webserver.py", line 111, in <module>
a = WebSockListener(host, port)
File "webserver.py", line 38, in __init__
self.glastopf_honeypot = glastopf.GlastopfHoneypot()
File "/opt/glastopf/glastopf.py", line 78, in __init__
self.dork_generator = self.setup_dork_generator(conf_parser)
File "/opt/glastopf/glastopf.py", line 124, in setup_dork_generator
mnem_service_instance=mnemosyne_service)
File "/opt/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 53, in __init__
self.bootstrap_dorkdb()
File "/opt/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 141, in bootstrap_dorkdb
dorks = self.mnem_service.get_dorks()
File "/opt/glastopf/modules/handlers/emulators/dork_list/mnem_service.py", line 36, in get_dorks
response = sess.post(base_url + '/login', payload, timeout=timeout, verify=False)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 241, in post
return self.request('post', url, data=data, **kwargs)
TypeError: request() got an unexpected keyword argument 'verify'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/evnet-1.0_5-py2.7.egg/evnet/__init__.py", line 561, in _readable
self._event('read', data)
File "/usr/local/lib/python2.7/dist-packages/evnet-1.0_5-py2.7.egg/evnet/util.py", line 16, in _event
cb(*args)
File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/EGG-INFO/scripts/glastopf-runner.py", line 67, in read
response = self.glastopf_honeypot.handle_request(d, self.addr, self.c)
File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/glastopf/glastopf.py", line 264, in handle_request
emulator.handle(attack_event)
File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/glastopf/modules/handlers/emulators/tomcat_status.py", line 27, in handle
with open(tomcat_manager_path, 'r') as tomcat_manager_file:
IOError: [Errno 2] No such file or directory: '/tomcat/manager_status.html'
Hi,
When we choose to host the honeypot on 0.0.0.0 port 80 and we submit data into user/password at the top of the page we are redirect to http://localhost:8080/ which broke the website.
libinjection might be a good replacement for ANTLR: https://github.com/client9/libinjection
Test is not passing at the moment.
======================================================================
FAIL: Objective: Emulator testing for PHP CGI remote code execution CVE-2012-1823
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jkv/glas-dev/glastopf/testing/test_emulators.py", line 232, in test_phpcgi_rce_emulator
self.assertTrue("""testing""" == self.event.response)
AssertionError: False is not true
-------------------- >> begin captured stdout << ---------------------
Successfully parsed with sandbox
Return value:
--------------------- >> end captured stdout << ----------------------
----------------------------------------------------------------------
Haven't tried this on a clean machine but it seems like we are dropping privileges before binding the WSGI server to port 80. This causes a permission error...
We don't handle malformed requests properly (basically the parser just fails and the exception is not handled). We should raise a proper exception and return an error code to the client.
See also #33
SQLite throws an error:
Insert into database Error: database is locked
Replacement of the current way too static and complicated solution.
i started working on it: https://github.com/glastopf/glastopf/tree/master/modules/handlers/emulators/surface
Some code is not in the repository yet.
Another missing directory: testing/reports
We should start to make the honeypot less obvious to detect. This will be part of the new attack surface creation. So with a more variable layout/design/etc. you will not be able to "see" the honeypot.
I'm not a big fan to ship a dork collection with the honeypot. We might want to ship some for bootstrapping and then remove them as soon as the honeypot is collecting attacks. But what Google has seen, Google will remember so this might cause issues.
Integrating additional dorks from a exploit database is great but could make us vulnerable for detection.
After merging with the sqlalchemy patch I got the following error:
File log_sql.py:
WARNING:root:Error caught while inserting 1 events into SQL, will retry in 1 seconds. ((OperationalError) table events has no column named time u'INSERT INTO events (time, source, request_method, request_url, request_parameters, request_version, request_header, request_body, pattern, filename, response) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' ('2012-12-08 10:32:57', '68.235.36.193:52957', 'GET', '/guestbook.pl', '', 'HTTP/1.0', [...]
We should give this a try: https://github.com/antlr/antlr3/tree/master/runtime/Python
Not sure if they have any major changes compared to the zip but might be worth a look.
Pretty close to closing the sqlalchemy deal, but battling with a few unit tests. Is the unit test below supposed to pass at the moment?
As far as i understand the test is supposed to query the dork database, but at the moment it is querying the events db.
https://github.com/glastopf/glastopf/blob/master/testing/test_dorks.py:
def test_dork_links(self):
"""Objective: Test if a random link from the dork page exists in the database.
Input: A random link from a created dork page.
Expected Results: The path of the link should be at least once in the db.
Notes: Links have the parameters truncated, so multiple entries are likely."""
dirname = 'modules/handlers/emulators/dork_list/pages/'
sample_file = choice(gen_dork_list.get_old_dork_pages_list(dirname))
print "Randomly selected dork page:", sample_file.rsplit('/', 1)[1]
with open(sample_file, 'r') as sample_data:
data = fromstring(sample_data)
links = data.cssselect('a')
test_link_path = choice(links).get('href')
print "Randomly selected path:", test_link_path
data = self.db.select_entry(test_link_path) #Problem here? Supposed to query dork.db instead?
print "Done searching for the entry."
self.assertTrue(len(data) > 0)
print "The dork db returned:",
print str(len(data)), "entries,",
print "which equates our expectation."
This can be triggered sometimes by issuing requests when the Glastopf is being set up.
I am using "39ea1e6a3236e6e" version.
After this, database logging gets crashed, although the webserver runs OK. I used sqlite3 for this.
2013-01-24 05:23:40,685 (glastopf) 192.168.1.107 requested GET / on 192.168.1.109:8080
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 505, in run self.__target(*self.__args, **self.__kwargs)
File "/home/phani/glastopf/glastopf.py", line 202, in post_processer self.maindb.insert(attack_event)
File "/home/phani/glastopf/modules/reporting/main/log_sql.py", line 51, in insert conn.execute(self.events_table.insert(entry))
File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1194, in execute params)
File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1274, in _execute_clauseelement return self.__execute_context(context)
File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1305, in __execute_context context.parameters[0], context=context)
File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1404, in _cursor_execute context)
File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1397, in _cursor_execute context)
File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 299, in do_execute
cursor.execute(statement, parameters)
OperationalError: (OperationalError) database is locked u'INSERT INTO events (time, source, request_method, request_url, request_parameters, request_version, request_header, req
uest_body, pattern, filename, response) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' ('2013-01-24 05:23:35', '192.168.1.107:56271', 'GET', '/', '', 'HTTP/1.1', '{"Accept-Language":
"en-US,en;q=0.8", "Accept-Encoding": .........
Most basic attack and recon tools (sqlmap, havij, nikto, metasploit, etc) attacks in a pretty detectable way. (most of the times anyways)
Capability should be added to the profiler to guess attack tool.
Make Glastopf $exploit_tool_name exploitable.
Running : 'python webserver.py' gives me this error:
2013-02-22 08:14:11,674 (glastopf) Generating initial dork pages - this can take a while.
Traceback (most recent call last):
File "webserver.py", line 111, in <module>
a = WebSockListener(host, port)
File "webserver.py", line 38, in __init__
self.glastopf_honeypot = glastopf.GlastopfHoneypot()
File "/home/ubuntu/glastopf/glastopf.py", line 85, in __init__
self.dork_generator.regular_generate_dork(0)
File "/home/ubuntu/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 116, in regular_generate_dork
self.generate_dork_pages()
File "/home/ubuntu/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 74, in generate_dork_pages
clusters = self.clustere.cluster(inurl_list)
File "/home/ubuntu/glastopf/modules/handlers/emulators/dork_list/cluster.py", line 31, in cluster
vectorizer = CountVectorizer(preprocessor=self.preprocessor, token_pattern=self.pattern, min_df=self.min_df)
TypeError: __init__() got an unexpected keyword argument 'token_pattern'
Various print statements are spread around the code base. These statements ought to be converted to use the logging module instead.
Which loglevels goes to console could then be configured in glastopf.cfg/glastopf.py.
johnnykv, do you think that makes sense? It might remove quite some overhead but probably requires some work...
Request:
http://192.168.9.34/index.php?id=%0D%0AH%
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/evnet-1.0_5-py2.7.egg/evnet/__init__.py", line 566, in _readable
self._event('read', data)
File "/usr/local/lib/python2.7/dist-packages/evnet-1.0_5-py2.7.egg/evnet/util.py", line 16, in _event
cb(*args)
File "webserver.py", line 63, in read
response = self.glastopf_honeypot.handle_request(d, self.addr, self.c)
File "/opt/glastopf/glastopf.py", line 108, in handle_request
attack_event.parsed_request = self.HTTP_parser.parse_request(raw_request)
File "/opt/glastopf/modules/HTTP/util.py", line 91, in parse_request
parsed_request.method = re_grp.group(1)
AttributeError: 'NoneType' object has no attribute 'group'
Install dependencies and various setup tasks. Maybe even install with virtualenv for required modules.
Downloading/unpacking evnet>=1.0-5 (from glastopf)
Could not find a version that satisfies the requirement evnet>=1.0-5 (from glastopf) (from versions: )
No distributions matching the version for evnet>=1.0-5 (from glastopf)
The requirements.rst still mentions the Sample DB which should not be needed any more.
Gevent is more mature and probably better suited for our case.
Goal is to leverage collected information to improve the honeypots classification, attack surface generation and emulation capabilities.
We don't really analyze data on the fly right now. Having this would put us in a position from which we can change the honeypots behavior based on the attacks we collect.
This will help us to spot trends and automatically react on it.
This allows rapid modification of the honeypot emulation on a global scale based on an event collected by a single sensor.
Request:
http://192.168.9.34/index.php?id=%27%20%3D%27 (resend request a couple of times, less then 5 should do in order to kill the webserver.
Error:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/evnet-1.0_5-py2.7.egg/evnet/init.py", line 48, in loop
else: l.start()
OSError: [Errno 9] Bad file descriptor: '<pyev.Io object at 0x266ae20> has been stopped'
oserror [Errno 9] Bad file descriptor: '<pyev.Io object at 0x266ae20> has been stopped' (9, 'Bad file descriptor')
2013-03-01 19:50:08,293 (glastopf.modules.privileges) Could not change file owner: [Errno 2] No such file or directory: 'modules/handlers/emulators/dork_list/pages'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/glastopf/modules/privileges.py", line 34, in drop
run_uid, run_gid)
OSError: [Errno 2] No such file or directory: 'modules/handlers/emulators/dork_list/pages'
Python version is 2.7.1 on an Ubuntu 11.04
pymongo version: 1.8.1
I get the following error:
Traceback (most recent call last):
File "webserver.py", line 11, in
import glastopf
File "/home/phani/glastopf/glastopf.py", line 38, in
from modules.handlers.emulators.dork_list import database_mongo
File "/home/phani/glastopf/modules/handlers/emulators/dork_list/database_mongo.py", line 19, in
from pymongo import MongoClient, uri_parser
ImportError: cannot import name MongoClient
The BFR related part in the documentation lacks complete instructions (php.ini part is missing).
os x, ubuntu, freebsd
Get this message when using the dorks.db:
Traceback (most recent call last):
File "/usr/local/openhoney/glastopf/modules/handlers/emulators/dork_list/dork_db.py", line 57, in trueInsert
self.cursor.execute(sql, (content, 1, datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")))
OperationalError: table intext has 5 columns but 4 values were supplied
2013-01-18 11:20:17,166 (modules.handlers.emulators.dork_list.dork_db) Error while inserting into dork_db: table intext has 5 columns but 4 values were supplied
Using the dorks.db from http://glastopf.org/dork.db.tar.
The PHP sandbox is currently a for of https://github.com/glastopf/phpox which some features missing. We want to remove the PHP sandbox from Glastopf in the future and install it externally as a module or run it as a service.
We could also run a central sandbox which should reduce processing time significantly if your sensor has a decent connection as we don't have to run the sample twice and can send the sensor historic data. This can be easily achieved if the sandbox is deployed as a service.
The files in the virtualdocs
directory are static and very easy to fingerprint. Might be useful to create them randomized on first start-up of Glastopf.
We could benefit from continuous integration, especially to test out the various supported database types in my "soon to come" pull request.
Lukas, if you enable the travis-ci hooks for the glastopf project i will try to set it up.
Seems like the SQL test makes the test process hung. Ctrl+C causes the following traceback:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.7/multiprocessing/util.py", line 295, in _exit_function
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/opt/glastopf/modules/reporting/log_sql.py", line 89, in consumer
time.sleep(self.wait_seconds)
KeyboardInterrupt
p.join()
File "/usr/lib/python2.7/multiprocessing/process.py", line 145, in join
res = self._popen.wait(timeout)
File "/usr/lib/python2.7/multiprocessing/forking.py", line 148, in wait
return self.poll(0)
File "/usr/lib/python2.7/multiprocessing/forking.py", line 133, in poll
pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.7/multiprocessing/util.py", line 295, in _exit_function
p.join()
File "/usr/lib/python2.7/multiprocessing/process.py", line 145, in join
res = self._popen.wait(timeout)
File "/usr/lib/python2.7/multiprocessing/forking.py", line 148, in wait
return self.poll(0)
File "/usr/lib/python2.7/multiprocessing/forking.py", line 133, in poll
pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
aux is a reserved directory name which cannot be used in windows. modules/reporting/aux must be reanmed to modules/reporting/auxiliary
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/evnet-1.0_5-py2.7.egg/evnet/__init__.py", line 561, in _readable
self._event('read', data)
File "/usr/local/lib/python2.7/dist-packages/evnet-1.0_5-py2.7.egg/evnet/util.py", line 16, in _event
cb(*args)
File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/EGG-INFO/scripts/glastopf-runner.py", line 67, in read
response = self.glastopf_honeypot.handle_request(d, self.addr, self.c)
File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/glastopf/glastopf.py", line 264, in handle_request
emulator.handle(attack_event)
File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/glastopf/modules/handlers/emulators/comments.py", line 41, in handle
with codecs.open(os.path.join(self.data_dir, 'comments.txt'), "r", "utf-8") as comments_txt:
File "/usr/lib/python2.7/codecs.py", line 881, in open
file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 2] No such file or directory: '/opt/honeypot/data/comments.txt'
When installing from a barebone ubuntu it is missing the libevent package.
fix: apt-get install libevent-dev
There are no supported loggers in the profiler. Besides traditional logging, it might be usefull to transmit this information on a hpfeeds channel.
I started creating the documentation based on Sphinx.
Minor issue .. but here goes ..
Request:
GET /cgi-bin/index.php?filepath=http://XXXXXXXXXX on 192.168.7.34
Response:
2013-01-19 23:20:39,413 (modules.handlers.emulators.rfi) Failed to fetch injected file, I/O error:
Traceback (most recent call last):
File "/opt/glastopf/modules/handlers/emulators/rfi.py", line 56, in download_file
injected_file = urllib2.urlopen(req, timeout=4).read()
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 401, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 419, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1211, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1181, in do_open
raise URLError(err)
URLError: <urlopen error timed out>
We might get some clues on how to react more appropriate if we do some static analysis.
At the moment some tests takes very long time to complete. I think that development in general would improve if we could run through all testcases quickly.
Is it really necessary that we use a input set of around 12k of entries?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.