Redis error

[circl]

Redis error

[adulau]

It seems that the Redis server are not running. Could you check on the screen session if the Redis servers are properly running?

[Nicknam3]

My bad, i forgot to launch ./docker_start.sh script, but i've this error now :
./docker_start.sh Traceback (most recent call last): File "Flask_server.py", line 15, in <module> import Paste File "/opt/AIL/bin/packages/Paste.py", line 30, in <module> from Hash import Hash File "/opt/AIL/bin/packages/Hash.py", line 5, in <module> import tlsh ImportError: No module named tlsh

I tried to modify tlsh version by using the trick provided by @Rafiot here [(https://github.com//issues/82)]
git checkout a67c69b0cdfd168c62c159d41b8a3612ee2b0df1

but no success.

[Rafiot]

Did you reinstall it within the virtualenv?

[Nicknam3]

Sorry for the delay. I tried again on a fresh install with :

docker build -t test-ail-framework .
docker run -it test-ail-framework /bin/sh

On the docker shell :

./installing_deps.sh

and i've another error about urlib3 :

ImportError : cannot import name HeaderParsingError

Any idea .

[Rafiot]

I never had this error before. Can you give me more context on when you get this error? (the full stacktrace for example)

[Nicknam3]

I'm on my docker install and when i launch ./installing_deps.sh, i've these errors:

 pip install -U pip
Traceback (most recent call last):
  File "/opt/AIL/AILENV/bin/pip", line 7, in <module>
    from pip import main
  File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/pip/__init__.py", line 11, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/pip/download.py", line 22, in <module>
    from pip._vendor import requests, six
  File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py", line 58, in <module>
    from . import utils
  File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/pip/_vendor/requests/utils.py", line 25, in <module>
    from .compat import parse_http_list as _parse_list_header
  File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/pip/_vendor/requests/compat.py", line 7, in <module>
    from .packages import chardet
  File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/__init__.py", line 3, in <module>
    from . import urllib3
  File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/__init__.py", line 16, in <module>
    from .connectionpool import (
  File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/connectionpool.py", line 33, in <module>
    from .connection import (
  File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/connection.py", line 41, in <module>
    from .util import (
  File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/__init__.py", line 5, in <module>
    from .response import is_fp_closed
  File "/opt/AIL/AILENV/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/response.py", line 4, in <module>
    from ..exceptions import HeaderParsingError
ImportError: cannot import name HeaderParsingError

I tried to update request with pip but when i've the same error.

(Edit by @Rafiot, proper formatting of the stack trace)

[Rafiot]

This link seems to be related to the issue you have: https://community.codeship.com/t/update-of-pip-not-properly-persisted-in-dependency-cache/1245

Do you already have a partial install or AIL? if yes, delete the AILENV directory, and re-run the script, that should do the trick. But as you're using cocker, I'd recommend to spawn a brand new image and re-do it from scratch.

[Nicknam3]

Nop, it's the first time i install the docker on my VM (i tried to install it on another desktop friday).
I'm using kali linux as VM and i launch docker for the first time this morning to install AIL-Framework and i've this HeaderParsingError problem.

Does it work for you with a fresh install by using the Dockerfile ?

I tried without installing the framework with Docker and it's work.

[Nicknam3]

But now, when i use the search function i've some Internal Server Error (500).

[Rafiot]

I never tried the docker file, but maybe @rivenz can help on that part?

[Rafiot]

If you get an error 500, I need the stack trace from the web server.

[Nicknam3]

Here is the stack trace from the web server:

127.0.0.1 - - [17/Sep/2016 19:40:10] "GET /_stuff HTTP/1.1" 200 -
127.0.0.1 - - [17/Sep/2016 19:40:12] "GET /?_=1474131253201 HTTP/1.1" 200 -
127.0.0.1 - - [17/Sep/2016 19:40:12] "GET /_stuff HTTP/1.1" 200 -
127.0.0.1 - - [17/Sep/2016 19:40:13] "GET /?_=1474131253202 HTTP/1.1" 200 -
127.0.0.1 - - [17/Sep/2016 19:40:13] "GET /_stuff HTTP/1.1" 200 -
127.0.0.1 - - [17/Sep/2016 19:40:14] "GET /?_=1474131253203 HTTP/1.1" 200 -
127.0.0.1 - - [17/Sep/2016 19:40:14] "GET /_stuff HTTP/1.1" 200 -
set([])
[2016-09-17 19:40:15,087] ERROR in app: Exception on /search [POST]
Traceback (most recent call last):
  File "/home/AIL-framework/AILENV/local/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/AIL-framework/AILENV/local/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/AIL-framework/AILENV/local/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/AIL-framework/AILENV/local/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/AIL-framework/AILENV/local/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "./Flask_server.py", line 350, in search
    ix = index.open_dir(indexpath)
  File "/home/AIL-framework/AILENV/local/lib/python2.7/site-packages/whoosh/index.py", line 123, in open_dir
    return FileIndex(storage, schema=schema, indexname=indexname)
  File "/home/AIL-framework/AILENV/local/lib/python2.7/site-packages/whoosh/index.py", line 421, in __init__
    TOC.read(self.storage, self.indexname, schema=self._schema)
  File "/home/AIL-framework/AILENV/local/lib/python2.7/site-packages/whoosh/index.py", line 616, in read
    gen = cls._latest_generation(storage, indexname)
  File "/home/AIL-framework/AILENV/local/lib/python2.7/site-packages/whoosh/index.py", line 593, in _latest_generation
    for filename in storage:
  File "/home/AIL-framework/AILENV/local/lib/python2.7/site-packages/whoosh/filedb/filestore.py", line 81, in __iter__
    return iter(self.list())
  File "/home/AIL-framework/AILENV/local/lib/python2.7/site-packages/whoosh/filedb/filestore.py", line 525, in list
    files = os.listdir(self.folder)
OSError: [Errno 2] No such file or directory: '/home/AIL-framework/indexdir'
127.0.0.1 - - [17/Sep/2016 19:40:15] "POST /search HTTP/1.1" 500 -

[Rafiot]

This directory is created by: https://github.com/CIRCL/AIL-framework/blob/master/bin/Indexer.py

The next steps after ./installing_deps.sh are described here:
https://github.com/CIRCL/AIL-framework#installation

[Nicknam3]

I forgot to validate with ENTER on the interface providing by LAUNCH.sh. All seems good now. Thank you very much.

Do you have some user guide or specific documentation to use the framework ? because i've no data on the web interface. So i don't know if i forgot to do some specific changes on config files or other.

Thx again for your help :)

[Rafiot]

Oh, yeah, you need to have a pystemon instance to fetch the pastes and feed AIL: https://github.com/CIRCL/pystemon

The alternative is to send us a mail to [email protected] with the source IP you want to use for AIL and we give you access to our own feed.

[Nicknam3]

Thanks for the information. But if i use your feed, it's not possible for me to modify the pystemon config file and search my specific patterns right ?

Edit : oh i see that there is some specific file to configure pystemon on /bin/feeder directory

[Rafiot]

if you are only looking for one specific pattern, and receive a mail when it appears, you don't need AIL :)

We use pystemon as a fetcher and we create a ZMQ feed we send to AIL an do the processing. The patterns in pystemon ave no impact on AIL

[Nicknam3]

Oh i see thanks ;) no i want to analyze datas publicy available on multiple source like pastebin, etc.
I will continue and i let you know if i've some other issues, but i think the docker currently does not work.

Thank you again Raphaël.

[Rafiot]

Great, closing then :)

[Nicknam3]

Sorry for my late response ! Thank you again and for all your work ! :)