circl/ail-framework

AIL framework - Analysis Information Leak framework. Project moved to https://github.com/ail-project

ail-framework's Issues

/usr/bin/ld: cannot find -lsnappy

Installing on an updated Ubuntu 14.04 and getting this error when running installing_deps.sh

+ make
cd src && make all
make[1]: Entering directory `/data/AIL-framework/redis-leveldb/src'
g++ rl_util.o rl_server.o rl_connection.o rl.o rl_request.o rl_kv.o rl_set.o rl_hash.o rl_list.o ../vendor/libleveldb.a  ../vendor/libleveldb.a -lm -L/usr/lib -lev -L/usr/lib -lgmp -lsnappy -lpthread -o ../redis-leveldb
/usr/bin/ld: cannot find -lsnappy
collect2: error: ld returned 1 exit status
make[1]: *** [redis-leveldb] Error 1
make[1]: Leaving directory `/data/AIL-framework/redis-leveldb/src'
make: *** [all] Error 2

"libsnappy1" is installed.

Ran the installing_deps.sh script also after following the manual installation steps but no luck either.

Error

Hi
I run update_thirdparty.sh Then encountered an error when running Flask_server.py with the following error message :

Traceback (most recent call last):
File "Flask_server.py", line 14, in <module>
sys.path.append(os.path.join(os.environ['AIL_BIN'], 'packages/'))
File "/usr/lib/python2.7/UserDict.py", line 40, in __getitem__
raise KeyError(key)
KeyError: 'AIL_BIN'

python error

Hi CIRCL team,

I was trying to install AIL-framework and found an python error during run "pip install -r pip_packages_requirement.txt" section :
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-5h5xZQ/ssdeep/

pyerror.txt

Best Wishes

Hardcoded path in ZMQ_Sub_Curve.py

There are two hardcoded path in ZMQ_Sub_Curve.py:

lib_words.create_curve_with_word_file(r_serv1, "/home/user/AIL/var/www/static/csv/wordstrendingdata","/home/user/AIL/files/wordfile", int(PST.p_date.year), int(PST.p_date.month))

The path should be built based on path configuration.

Webstats.py

Suddenly my webstats.py decided to stop working :/, what do you think is happening which shouldn't?

Traceback (most recent call last):
File "./WebStats.py", line 133, in
month)
File "/home/olivier/AIL-framework/bin/packages/lib_words.py", line 89, in create_curve_with_word_file
with open(feederfilename, 'rb') as f:
IOError: [Errno 2] No such file or directory: '/home/olivier/AIL-framework/AILENV/faup/src/data/mozilla.tlds'

Issues

Hi I am seeing u guys are in full update mode so I will just point out some stuff I have seen which are abit weird, or not working to well. Sry that i cant just push myself but it's company policy where I work.
Curve.py and Webstats.py have connection erros on localhost:2013, Also the Creditcard regex seem to match way more than it should, my AIL is completely filled with false positives.

Traceback (most recent call last):
File "./Curve.py", line 68, in
prev_score = r_serv1.hget(low_word, date)
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/redis/client.py", line 1857, in hget
return self.execute_command('HGET', name, key)
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/redis/client.py", line 578, in execute_command
connection.send_command(_args)
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/redis/connection.py", line 563, in send_command
self.send_packed_command(self.pack_command(_args))
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
self.connect()
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/redis/connection.py", line 442, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to localhost:2013. Connection refused.

Exception in Wordtrending section ([Errno32] Broken pipe)

I have an error that doesn't allow the framework to work correctly
While browsing the Wordtrending section, the "Broken Pipe" error , illustrated below, is emerging:

Could you help please .
Is it because I'm using a VM on my personnal computer ?

Implementing SIMHASH for document similarities

Implementing SIMHASH for document similarities in AIL

https://github.com/liangsun/simhash

Configuration documentation?

Hello,

I gave a try to AIL with docker. It starts correctly. after I filled some terms in Terms Management but I don't see anywhere where I can configure where AIL should fetch data, which websites (pastebin...) and at which frequency.

Where are those options?
Didn't find any official documentation (No wiki, AIL-framework/doc/ seems more about code/architecture)

Thanks

dns.exception.Timeout not catched (ZMQ_Sub_Mail.py)

dns.exception.Timeout not catched (ZMQ_Sub_Mail.py)

hardcoded IP in the Flask/web server

There is an hardcoded IP address in the Flask/web server:

app.run(host='172.16.100.123' ,port=7000, threaded=True)

Could you replace it with localhost or 0.0.0.0?

Configuration & Setup & Flask-Server

I might be just stupid..
Backstory:
I cloned the repository.
Installed all dependencies.

Path 1:
Ran the AIL-Framework.
With:
LevelDB
Redis
Logs
Queues
Scripts

Ran pystemon with my own configured yaml.
Ran the Flask-Server.
Nothing except the main interface. No data, no listing, nothing.

Path 2:
Set the path in the pystemon-feeder script to '/opt/pystemon/' (where pystemon is located)
Ran the feeder
Ran the AIL-Framework with the options in path 1
Ran the Flask-server.
Again: Nothing except the main UI.

Did I miss to configure something?
Also:
I can't kill the Flask-Server with CTRL+C, it just gets unresponsive.
And the AILENV variable needs to be reset after reboot.

Running on Ubuntu 14.04 LTS without xServer (plain cli rootserver)

Error 404 for adns-python module installation

Hi CIRCL team,

I was trying to build a docker container with AIL framework and found a 404 error during the build process due to the following URL located in pip_packages_requirement.txt that could not be accessed anymore:

http://adns-python.googlecode.com/files/adns-python-1.2.1.tar.gz

There is however an archive link available at the following URL (but I dont know for how long):

https://web.archive.org/web/20151223111838/https://adns-python.googlecode.com/files/adns-python-1.2.1.tar.gz

Best regards,

Tabs disappearing

The tabs keep disappearing when I go to click on them.

Before:

Hover over tabs:

Any idea on how to fix this?

not working

Hi Dear
I've installed the tool without error.
However, no data will be displayed on the dashboard.
There's no such paste does not download.
How can I tell where is the problem.

Installation Issues & Starting the Flask server

Hello Team,

Really thanks for the great project. Do you have any better documentation for installation (or) any recommendations to choose which version of distro will work out of the box.

matplotlib dependency: incorrect Distribute version detection

distribute in /usr/local/lib/python2.7/dist-packages/distribute-0.7.3-py2.7.egg
setuptools>=0.7 in /usr/local/lib/python2.7/dist-packages/setuptools-16.0-py2.7.egg (from distribute)

current Distribute version installed is 'distribute-0.7.3-py2.7' but the auto installer can't detect it, even using easy_install:

Error Trace:

Downloading/unpacking matplotlib (from -r pip_packages_requirement.txt (line 11))
Downloading matplotlib-1.4.3.tar.gz (50.4Mb): 50.4Mb downloaded
Running setup.py egg_info for package matplotlib
The required version of distribute (>=0.6.28) is not available,
and can't be installed while this script is running. Please
install a more recent version first, using
'easy_install -U distribute'.

Currently using distribute 0.6.24 (/root/AIL-framework/AILENV/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg)

Complete output from command python setup.py egg_info:
The required version of distribute (>=0.6.28) is not available, and can't be installed while this script is running. Please install a more recent version first, using

'easy_install -U distribute'.

(Currently using distribute 0.6.24 (/root/AIL-framework/AILENV/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg))

Pystemon to AIL connector

Add a sample Python script to publish new Pastes into AIL

Problem with launching AIL

After installing dependencies, and changing paths in LAUNCH.sh and attempting to launch the framework I see the following processes, but there is no listener on port 7000 as the documentation suggests.

root 18487 18473 0 14:09 pts/19 00:00:00 bash -c redis-server /home/wsalusky/Github/AIL-framework/configs/6379.conf ; read x
root 18491 18473 0 14:09 pts/20 00:00:00 bash -c redis-server /home/wsalusky/Github/AIL-framework/configs/6380.conf ; read x
root 18494 18473 0 14:09 pts/21 00:00:00 bash -c redis-server /home/wsalusky/Github/AIL-framework/configs/6381.conf ; read x
root 18511 18497 0 14:09 pts/23 00:00:00 bash -c redis-leveldb -H 127.0.0.1 -D /home/wsalusky/Github/AIL-framework/AILENV/LEVEL_DB_DATA/201
root 18512 18511 0 14:09 pts/23 00:00:00 redis-leveldb -H 127.0.0.1 -D /home/wsalusky/Github/AIL-framework/AILENV/LEVEL_DB_DATA/2013/ -P 20
root 18514 18497 0 14:09 pts/24 00:00:00 bash -c redis-leveldb -H 127.0.0.1 -D /home/wsalusky/Github/AIL-framework/AILENV/LEVEL_DB_DATA/201
root 18516 18514 0 14:09 pts/24 00:00:00 redis-leveldb -H 127.0.0.1 -D /home/wsalusky/Github/AIL-framework/AILENV/LEVEL_DB_DATA/2014/ -P 20
root 18531 18517 0 14:09 pts/26 00:00:00 bash -c ./log_subscriber -p 6380 -c Queuing -l ../logs/; read x
root 18534 18517 0 14:09 pts/27 00:00:00 bash -c ./log_subscriber -p 6380 -c Script -l ../logs/; read x
root 18551 18537 0 14:09 pts/29 00:00:00 bash -c ./ZMQ_Feed_Q.py; read x
root 18555 18537 0 14:09 pts/30 00:00:00 bash -c ./ZMQ_Sub_Duplicate_Q.py; read x
root 18559 18537 0 14:09 pts/31 00:00:00 bash -c ./ZMQ_Sub_Attributes_Q.py; read x
root 18563 18537 0 14:09 pts/32 00:00:00 bash -c ./ZMQ_PubSub_Lines_Q.py; read x
root 18567 18537 0 14:09 pts/33 00:00:00 bash -c ./ZMQ_PubSub_Categ_Q.py; read x
root 18571 18537 0 14:09 pts/34 00:00:00 bash -c ./ZMQ_PubSub_Tokenize_Q.py; read x
root 18575 18537 0 14:09 pts/35 00:00:00 bash -c ./ZMQ_Sub_CreditCards_Q.py; read x
root 18579 18537 0 14:09 pts/36 00:00:00 bash -c ./ZMQ_Sub_Onion_Q.py; read x
root 18583 18537 0 14:09 pts/37 00:00:00 bash -c ./ZMQ_Sub_Mails_Q.py; read x
root 18587 18537 0 14:09 pts/38 00:00:00 bash -c ./ZMQ_Sub_Urls_Q.py; read x
root 18590 18537 0 14:09 pts/39 00:00:00 bash -c ./ZMQ_Sub_Curve_Q.py; read x
root 18607 18593 0 14:09 pts/41 00:00:00 bash -c ./ZMQ_Feed.py; read x
root 18611 18593 0 14:09 pts/42 00:00:00 bash -c ./ZMQ_Sub_Duplicate.py; read x
root 18615 18593 0 14:09 pts/43 00:00:00 bash -c ./ZMQ_Sub_Attributes.py; read x
root 18619 18593 0 14:09 pts/44 00:00:00 bash -c ./ZMQ_PubSub_Lines.py; read x
root 18623 18593 0 14:09 pts/45 00:00:00 bash -c ./ZMQ_PubSub_Categ.py; read x
root 18627 18593 0 14:09 pts/46 00:00:00 bash -c ./ZMQ_PubSub_Tokenize.py; read x
root 18631 18593 0 14:09 pts/47 00:00:00 bash -c ./ZMQ_Sub_CreditCards.py; read x
root 18635 18593 0 14:09 pts/48 00:00:00 bash -c ./ZMQ_Sub_Onion.py; read x
root 18639 18593 0 14:09 pts/49 00:00:00 bash -c ./ZMQ_Sub_Mails.py; read x
root 18643 18593 0 14:09 pts/50 00:00:00 bash -c ./ZMQ_Sub_Urls.py; read x
root 18646 18593 0 14:09 pts/51 00:00:00 bash -c ./ZMQ_Sub_Curve.py; read x

Feeding

Hello,

I have a doubt about "Feeding AIL with pystemon". In this section, one of the steps is: "Edit the file bin/feeder/pystemon-feeder.py and modify the pystemonpath path accordingly", but if I edit the pystemonpath variable inside the file: pystemon-feeder.py, (my pystemonpath is: /home/pystemon/pystemon.py), and then launch pystemon.py and pystemon-feeder.py, when opening the web browser, I don't see any information about pastesbins in AIL-Framework homepage. Could it be a problem with redis? Or Have I been editing the pystemonpath incorrectly?

Thank you!

Auto Installation: symbolic link error

** file: https://github.com/CIRCL/AIL-framework/blob/master/installing_deps.sh
** line 21: test ! -L /usr/include/ft2build.h && ln -s freetype2/ft2build.h /usr/include/

• test '!' -L /usr/include/ft2build.h
• ln -s freetype2/ft2build.h /usr/include/

ln: failed to create symbolic link «/usr/include/ft2build.h»: File exists

There is no data

Just fresh installed your software without errors (install-dep.sh, etc.)

launch everything through LAUNCH.sh with success (listen on 2016, 3016, 6379, etc...)

started the flask_server

then nothing

Notification when the feed is empty for a long period

An alarm should be sent when the feed is empty for a defined period (5 minutes).

Storage of pastes in case of (very) large dataset

Following the discussion about the storage of pastes/feeds in case of (very) large dataset, we could evaluate the use of https://github.com/chrislusf/seaweedfs

Testing Bokeh

I tested a bit Bokeh (http://bokeh.pydata.org) and it's quite nifty for drawing graphs and so on. It might be worth to evaluate it for AIL.

Pystemon Problems

Hi, i managed to install AIL-framework, but i am having problems figuring out how i activate the pystemon feeder to the AIL-framework, any help would be very helpful.
Thank you in advance

Notification queue based on threshold value

Notification queue based on threshold value (e.g. if a certain number of valid emails is reached).

Broken pipe

Since I run the webserver on the VM provided by you, I start see some broken pipes...

192.168.56.1 - - [27/Aug/2014 12:39:08] "GET /wordstrending/ HTTP/1.1" 200 -
192.168.56.1 - - [27/Aug/2014 12:39:08] "GET /static//csv/wordstrendingdata.csv HTTP/1.1" 200 -
192.168.56.1 - - [27/Aug/2014 12:39:12] "GET /_logs HTTP/1.1" 200 -
----------------------------------------
Exception happened during processing of request from ('192.168.56.1', 33317)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 593, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
192.168.56.1 - - [27/Aug/2014 12:39:25] "GET / HTTP/1.1" 200 -
192.168.56.1 - - [27/Aug/2014 12:39:25] "GET /_stuff HTTP/1.1" 200 -
192.168.56.1 - - [27/Aug/2014 12:39:52] "GET /_logs HTTP/1.1" 200 -
----------------------------------------
Exception happened during processing of request from ('192.168.56.1', 33322)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 593, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------

Displayed information

Two Questions:

1. I'm only just seeing the follow information on webinterface which differs from your screenshot on documentation page. Is it ok?
   

2. What means each icon in message column?
   http://s7.postimg.org/vobprz64r/image2.png

Module to add - Hash Algorithm Identifier

https://github.com/AnimeshShaw/Hash-Algorithm-Identifier

Module to add - Hash Algorithm Identifier

Edit graph words.

In the words trending tab, edit word list does not work. If it is clicked on, nothing happens.

Adding support for Automatic Keyphrase Extraction Task

Adding support for Automatic Keyphrase Extraction Task as an external metadata value for each element.

http://www.aclweb.org/anthology/S10-1004
https://github.com/snkim/AutomaticKeyphraseExtraction

Documentation - configs/ requires editing

Please note for future installers that it is necessary to update the 'dir' directive in each of the configuration files located in AIL-framework/configs/

docker_start.sh needs 2017 and Mixer.py adding

docker_start.txt

dns.exception.Timeout not catched (ZMQ_Sub_Urls.py)

dns.exception.Timeout not catched (ZMQ_Sub_Urls.py) (everything is in the title ;-).

Thank you

Link to download a paste is undefined when in DEBUG info

Link to download a paste is undefined when in DEBUG info

Impossible to use the latest version of python-tlsh

See: trendmicro/tlsh#25

Dirty fix: https://github.com/CIRCL/AIL-framework/blob/master/installing_deps.sh#L100

Dashboard graphs and logs disapear when switching to the Words Trending tab.

When I switch over to the Words Trending tab and then go back to the Dashboard, all of the data that use to be in the Dashboard is reset. Also, if the browser is closed, all of the Dashboard data is lost.

Data not showing on 'Browse important pastes' tab

Hello,
I have some pastes containing a mail address, I know this mail address has been received by AIL because I can retrieve it in the "Terms frequency: Management interface" tab. But my 'Browse important pastes' tab is still empty (I am using pystemon to feed AIL):

Thank you.

get_p_content should add a redis cache

As get_p_content from the class Paste is called by all modules, it would be nice to add a simple expiring cache in Redis on get_p_content. To avoid read from the disk and improve I/O.

Redis error

Hello,

I've these errors on a fresh install :

Running on http://0.0.0.0:9000/ (Press CTRL+C to quit) Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 763, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.7/SocketServer.py", line 596, in process_request_thread self.handle_error(request, client_address) File "/usr/lib/python2.7/SocketServer.py", line 593, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__ self.handle() File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/werkzeug/serving.py", line 216, in handle rv = BaseHTTPRequestHandler.handle(self) File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle self.handle_one_request() File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/werkzeug/serving.py", line 251, in handle_one_request return self.run_wsgi() File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/werkzeug/serving.py", line 193, in run_wsgi execute(self.server.app) File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/werkzeug/serving.py", line 183, in execute for data in application_iter: File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/werkzeug/wsgi.py", line 703, in __next__ return self._next() File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/werkzeug/wrappers.py", line 81, in _iter_encoded for item in iterable: File "./Flask_server.py", line 75, in event_stream pubsub.psubscribe("Script" + '.*') File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/redis/client.py", line 2199, in psubscribe ret_val = self.execute_command('PSUBSCRIBE', *iterkeys(new_patterns)) File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/redis/client.py", line 2161, in execute_command self._execute(connection, connection.send_command, *args) File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/redis/client.py", line 2172, in _execute connection.connect() File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/redis/connection.py", line 442, in connect raise ConnectionError(self._error_message(e)) ConnectionError: Error 111 connecting to localhost:6380. Connection refused.

Any idea ? Thx.

Improve the Phone module

@Starow pointed me this library: https://github.com/daviddrysdale/python-phonenumbers

This would be a great improvement for the Phone module. Cc @Alainfou

Issues with big files containing tons of UUIDs

Tokenize and Sentiment analysis were stuck on our platform because of huge files containing huge lists of UUIDs (more than one million per file).

Some examples:

teruteru128_cceb0e34fb4153d454596c03f953d8f6
teruteru128_292c5a69b229bfbd2875240ca1587755
teruteru128_f01fa322f03f3c341421a0c4bab5f508
teruteru128_17bf2ebc3b47c9846ec0d5248e34f55a
teruteru128_968d53c6287ed0435e1c205c01d40e64

Sentiment Analysis Stuck

The "Sentiment Analysis" page is not reporting any data and the "SentimentAnalysis" queue under the Dashboard page is showing red (stuck).
The AIL logs are not reporting any issues that I can see.
Anyone experience this and know how to fix?
Thanks!

Missing ".sh" within the installing_deps.sh

Hello,

just ran into an issue while installing.
Lien 42 within above scripts only mentions "./make" but it should be "./make.sh" to run from within the script.
At least on my Ubuntu. :-)

Cheers
Marcus

Daemon Run

I hall,
I`m trying to use AIl in daemon mode. How can i run AIL, pystmon feed on boot

pyFaub

Hi i am trying out the new version of AIL and almost everything seems to work, but only question is what is pyfaub? and how come i am getting a error saying it cant be imported from my webstats.py

Traceback (most recent call last):
File "./WebStats.py", line 16, in
from pyfaup.faup import Faup
ImportError: No module named pyfaup.faup

Thank you in advance.

Indexer crash (due to gzip)

Traceback (most recent call last):
File "ZMQ_Sub_Indexer.py", line 59, in
PST = Paste.Paste(message.split(" ", -1)[-1])
File "/home/adulau/AIL-framework/bin/packages/Paste.py", line 79, in init
self.p_mime = magic.from_buffer(self.get_p_content(), mime=True)
File "/home/adulau/AIL-framework/bin/packages/Paste.py", line 108, in get_p_content
paste = f.read()
File "/usr/lib/python2.7/gzip.py", line 254, in read
self._read(readsize)
File "/usr/lib/python2.7/gzip.py", line 325, in _read
self._read_eof()
File "/usr/lib/python2.7/gzip.py", line 343, in _read_eof
crc32 = read32(self.fileobj)
File "/usr/lib/python2.7/gzip.py", line 25, in read32
return struct.unpack("<I", input.read(4))[0]
struct.error: unpack requires a string argument of length 4

Leveldb error

Pastebin IP block

Hey, I have been using AIL-framework with pystemon, for awhile now and have managed to get my ip banned by pastebin.com, how do you manage to access the site continously? Did you pay for the subscription to avoid bans?

Thanks you in advance.