stamusnetworks/amsterdam

Docker based Suricata, Elasticsearch, Logstash, Kibana, Scirius aka SELKS

amsterdam's Issues

Elasticsearch staying yellow

I've noticed that elasticsearch stays in a yellow status and doesn't change to green. Apparently this is a common issue with the default deployment of it, and its due to how the default configuration handles nodes. I managed to fix this by following this

"This is a common issue arising from the default index setting, in particularly, when you try to replicate on a single node. To fix this with transient cluster setting, do this:

curl -XPUT http://localhost:9200/_settings -d '{ "number_of_replicas" :0 }'

Next, enable the cluster to reallocate shards (you can always turn this on after all is said and done):

curl -XPUT http://localhost:9200/_cluster/settings -d '
{
"transient" : {
"cluster.routing.allocation.enable": true
}
}'

Now sit back and watch the cluster clean up the unassigned replica shards. If you want this to take effect with future indices, don't forget to modify elasticsearch.yml file with the following setting and bounce the cluster:

index.number_of_replicas: 0
"

from here http://stackoverflow.com/questions/23656458/elasticsearch-what-to-do-with-unassigned-shards

of course i used docker exec -it to do it

Error installing from setup

[admin@myserver Amsterdam-master]$ amsterdam -d ams -i eth0 setup
Traceback (most recent call last):
File "/usr/bin/amsterdam", line 4, in
import('pkg_resources').run_script('amsterdam==1.2', 'amsterdam')
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 738, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 1499, in run_script
exec(code, namespace, namespace)
File "/usr/lib/python2.7/site-packages/amsterdam-1.2-py2.7.egg/EGG-INFO/scripts/amsterdam", line 46, in
ams = Amsterdam(args.name, args.iface, basepath)
File "/usr/lib/python2.7/site-packages/amsterdam-1.2-py2.7.egg/amsterdam/amsterdam.py", line 58, in init
self.check_environment()
File "/usr/lib/python2.7/site-packages/amsterdam-1.2-py2.7.egg/amsterdam/amsterdam.py", line 140, in check_environment
self.get_api_version()
File "/usr/lib/python2.7/site-packages/amsterdam-1.2-py2.7.egg/amsterdam/amsterdam.py", line 123, in get_api_version
self.api_version = Client().version()['ApiVersion']
File "/usr/lib/python2.7/site-packages/docker/api/daemon.py", line 76, in version
return self._result(self._get(url), json=True)
File "/usr/lib/python2.7/site-packages/docker/utils/decorators.py", line 47, in inner
return f(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/docker/client.py", line 139, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 488, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 473, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(2, 'No such file or directory'))

check for docker-compose

If docker-compose is missing from the system 'amsterdam' doesn't give a clear error message. It would help new users to give a clean warning in such cases.

Managing logs

Hi,

In SELKS there is a https://github.com/StamusNetworks/SELKS/wiki/Logrotate. How do you manage logs here?
Is it good idea to manage them through the main host and SIGHUP with 'docker exec'?

Python3 Support

I'm guessing this project is in need of an update, some of the required libs aren't even available under python2 on my distro now.
Any change of an update in the future?

Unable to modify threshold.conf

With SELKS I was able to modify threshold.conf so I was able to add supress rules. With Amsterdam however I am not.

I found 2 docker images with /etc/suricata.
One of them was on a read only file system, changing the other didn't reflect my changes.

How am I supposed to edit threshold.conf?

Guidance - suricata.yaml\\\" caused \\\"not a directory\\\"\""

Hey,

I'm working on some changes to the repo in order to open a PR, namely getting Amsterdam itself to run in a container and therefore requiring no littering of the underlying host. I'm very close, however, Amsterdam is having an issue deploying the suricata service in particular as per the below error:

Starting data_suricata_1 ... error

ERROR: for data_suricata_1  Cannot start service suricata: oci runtime error: container_linux.go:262: starting container process caused "process_linux.go:339: container init caused \"rootfs_linux.go:57: mounting \\\"/opt/Amsterdam/data/config/suricata/suricata.yaml\\\" to rootfs \\\"/var/lib/docker/overlay2/c63115e1ff44923ae65ece97b40a3b8bcec448e1c7c42b8d0c5672d05ed8d58c/merged\\\" at \\\"/var/lib/docker/overlay2/c63115e1ff44923ae65ece97b40a3b8bcec448e1c7c42b8d0c5672d05ed8d58c/merged/etc/suricata/suricata.yaml\\\" caused \\\"not a directory\\\"\""

Other containers start up just fine. Any ideas?

For reference, my repo is https://github.com/monokal/Amsterdam.git and the Amsterdam container can be built and run like so:

./dbuild.sh build && ./dbuild.sh test

Suricata can't start due to missing file

Suricata fails to start and supervisor states too many failed attempts to start suricata.

The following were the steps I took to fix the issue (missing file). If I have time I will make a pull request.

/usr/bin/suricata -T /etc/suricata/suricata.yaml
26/12/2015 -- 21:26:03 - - Running suricata under test mode
26/12/2015 -- 21:26:03 - - This is Suricata version 3.0RC3 RELEASE
26/12/2015 -- 21:26:13 - - [ERRCODE: SC_ERR_FOPEN(44)] - Error opening file: "/etc/suricata//threshold.config": No such file or directory
26/12/2015 -- 21:26:13 - - Configuration provided was successfully loaded. Exiting.

touch /etc/suricata/threshold.config

/usr/bin/suricata -T /etc/suricata/suricata.yaml
26/12/2015 -- 21:26:46 - - Running suricata under test mode
26/12/2015 -- 21:26:46 - - This is Suricata version 3.0RC3 RELEASE
26/12/2015 -- 21:26:56 - - Configuration provided was successfully loaded. Exiting.

Using Python 2 only?

Hi,

After installing a clean python3 virtualenv I get:

~$ amsterdam -d ams-test -i wlan0 setup
Traceback (most recent call last):
File "/home/gmuller/amsterdam-test/bin/amsterdam", line 46, in
ams = Amsterdam(args.name, args.iface, basepath)
File "/home/gmuller/amsterdam-test/lib/python3.6/site-packages/amsterdam/amsterdam.py", line 47, in init
self.check_environment()
File "/home/gmuller/amsterdam-test/lib/python3.6/site-packages/amsterdam/amsterdam.py", line 112, in check_environment
self.name.decode('ascii')
AttributeError: 'str' object has no attribute 'decode'

According to:
https://stackoverflow.com/questions/26014209/python-3-4-str-attributeerror-str-object-has-no-attribute-decode#comment40743496_26014209

The problem is that the code is written for python2...

Would have been great to read a wraning about that somewhere, before installing the whole Python3 env...
e.g. on pages:
https://www.stamus-networks.com/2016/06/02/amsterdam-1-0-selks-and-docker/
or
https://github.com/StamusNetworks/Amsterdam

Thanks for the job anyway, ELK+Suricata+Scirius+Evebox all look fabulous!

Update full is not changing iface name

Ethtool configuration for supervisor in suricata container is not updated after a full update.

change command: option in docker-compose.yml

i was wondering if it would make sense for you to change the command option in the docker-compose file. How it is currently configured means expanding logstash to take other inputs is difficult. A small change from:
'command: -f /etc/logstash/logstash.conf'
to
'command: /opt/logstash/bin/logstash -f /etc/logstash/conf.d/'
and of course changing the setup script to place the logstash.conf in the conf.d folder.

However it would mean the the logstash would be able to be configured inline with the documentation for it.

Just a suggestion. Great work guys

Arr0n

Stack addon

I'd like to suggest that Bro IDS be added as part of the stack. I know how useful it is in real world.

Which architectures are supported?

Hi!
First of all thanks for this project! It's really cool and useful!
Sorry, newbie here, so maybe this issue is more like a feature request...
I haven't found clear info, so my question: could I run it on something like raspberry pi?
That would be really cool adding the support for arm.
Thank you.

not working, alas

Really love the idea of Amsterdam, but the install has been a problem.
I'm probably just missing a large collection of things obvious to people
who are closer to the project.

Install took quite awhile to get through. Many missing and sometimes
circular dependencies. I think I have it installed properly now (at least
the setup.py runs to completion without errors).

However, when I run the configure command:

amsterdam -d ams -i <my_device_port> setup
I get a backtrace that ends with:

requests.exceptions.ConnectionError: ('Connection aborted.', error(2, 'No such file or directory'))

Tried several different versions of the above command with different ./data directories.
Same error. Also looked through other issues filed -- nothing seemed to
match this, but again, I do not have much context yet.

My install: CentOS7; Python 2.7.

Add init/upstart/systemd services for Amsterdam

We need the ability to easily stop and start Amsterdam with the operating system. Please add init/upstart/systemd services for Amsterdam.

Amsterdam won't start if path contains utf-8

All is in the subject.

High I/O usage

When running Amsterdam I am taking a huge hit on my datastore on:

Picture

I am not swapping to the disk:

          total        used        free      shared  buff/cache   available

Mem: 4799 2768 461 16 1569 1745
Swap: 2063 0 2063

How can I trace the high I/O usage back to a specific part of Amsterdam?

Run PCAP through SELKS

Not sure if this is the appropriate forum for this, so please feel free to redirect me.

I'm trying to run PCAP through the docker image for SELKS, but the suricata.yaml file shows regular suricata settings...

Is there a way to reference the docker image to run suricata... In other words, If I run amsterdam -d ams start it will start suricata just fine, but I want to run PCAP through the docker suricata. What is the best way to do this?

Setup fails - API conflict on 16.04

On a clean install of 16.04, following installation instructions, I get a conflict between client API and server API.

root@ubuntu1604VM:/home/fbegin1# docker -v
Docker version 1.11.2, build b9f10c9

root@ubuntu1604VM:/home/fbegin1# amsterdam -d ams -i enp0s3 setup
Traceback (most recent call last):
File "/usr/local/bin/amsterdam", line 46, in
ams = Amsterdam(args.name, args.iface, basepath)
File "/usr/local/lib/python2.7/dist-packages/amsterdam/amsterdam.py", line 47, in init
self.check_environment()
File "/usr/local/lib/python2.7/dist-packages/amsterdam/amsterdam.py", line 119, in check_environment
self.api_version = Client().version()['ApiVersion']
File "/usr/local/lib/python2.7/dist-packages/docker/api/daemon.py", line 76, in version
return self._result(self._get(url), json=True)
File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 178, in _result
self._raise_for_status(response)
File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 174, in _raise_for_status
raise errors.APIError(e, response, explanation=explanation)
docker.errors.APIError: 400 Client Error: Bad Request ("client is newer than server (client API version: 1.24, server API version: 1.23)")

Error in src/amsterdam.py

Hello,

There is an error in this file src/amsterdam.py in this line "from docker import Client". It's a small 'c' letter not a capital one.

Just doesn't work

First issue - you did not define any of the prerequisites that you will need in your documentation before installing via pip - ESPECIALLY - the 1.9.0 version of docker-compose (otherwise you get a client error)

Second issue - after all the prereq's were done, I finally was able to get through an install via pip. However, when attempting to access via browser, it gives an 'Internal Server Error'

Is this meant to monitor a single machine or a network of machines? What if you wanted to run this on one machine and monitor a group of web servers in the same subnet?

Timelion not working

When I open Timelion I am getting a blank page.

This are the errors in the console:

Failed to load resource: the server responded with a status of 400 (BAD REQUEST)

Error: [illegal_argument_exception] mapper [hits] cannot be changed from type [long] to [int]
at respond (timelion.bundle.js?v=9892:35254)
at checkRespForFailure (timelion.bundle.js?v=9892:35217)
at timelion.bundle.js?v=9892:33835
at processQueue (commons.bundle.js?v=9892:41839)
at commons.bundle.js?v=9892:41855
at Scope.$eval (commons.bundle.js?v=9892:43083)
at Scope.$digest (commons.bundle.js?v=9892:42894)
at Scope.$apply (commons.bundle.js?v=9892:43191)
at done (commons.bundle.js?v=9892:37640)
at completeRequest (commons.bundle.js?v=9892:37838)(anonymous function) @ commons.bundle.js?v=9892:39571

System stops working after a few weeks

Hi.

I've deloyed amsterdam 1.0 and it runs good for a while, but after a few weeks it's just stop responding (i mean web interface shows no activity). Evebox loading endlessly, scirius shows "Fetching data". Elasticsearch become red. Eve Insert rate is about 20 (but when it works good its about 100) Some log:

> elasticsearch_1  | [2016-07-18 17:52:42,687][DEBUG][action.admin.cluster.node.stats] [Dum-Dum Dugan] failed to execute on node [peHmChnxQ5qjNYNvCJkjhw]
> elasticsearch_1  | ReceiveTimeoutTransportException[[Dum-Dum Dugan][172.17.0.3:9300][cluster:monitor/nodes/stats[n]] request_id [1456381] timed out after [17231ms]]
> elasticsearch_1  |      at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:679)
> elasticsearch_1  |      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> elasticsearch_1  |      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> elasticsearch_1  |      at java.lang.Thread.run(Thread.java:745)
> elasticsearch_1  | [2016-07-18 17:52:47,052][WARN ][transport                ] [Dum-Dum Dugan] Received response for a request that has timed out, sent [21596ms] ago, timed out [4365ms] ago, action [cluster:monitor/nodes/stats[n]], node [{Dum-Dum Dugan}{peHmChnxQ5qjNYNvCJkjhw}{172.17.0.3}{172.17.0.3:9300}], id [1456381]
> kibana_1         | {"type":"log","@timestamp":"2016-07-18T17:52:59+00:00","tags":["status","plugin:elasticsearch","error"],"pid":11,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 30000ms","prevState":"green","prevMsg":"Kibana index ready"}
> kibana_1         | {"type":"log","@timestamp":"2016-07-18T17:53:04+00:00","tags":["status","plugin:elasticsearch","info"],"pid":11,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 30000ms"}

Configuration is:
SW:
Debian Jessie 8.5 x86_64
HW:
core i3-3220
16Gb RAM
2x500Gb SATA 7200RPM @ RAID1
eth - 1Gbps (real traffic is about 30-40Mbps)

It there performance bottle neck? or maybe it's configuration problem?

Thank you

amsterdam install not works on Ubuntu desktop 18.04 64 bit

pip install amsterdam

Collecting amsterdam
Using cached https://files.pythonhosted.org/packages/ac/f2/13c4dab827cd19271aca2786d712ba203da276d09d96804b3bcdcf36613a/amsterdam-1.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-62cdWj/amsterdam/setup.py", line 3, in
from src.amsterdam import AMSTERDAM_VERSION
File "src/init.py", line 2, in
from .amsterdam import *
File "src/amsterdam.py", line 28, in
from docker import Client
ImportError: cannot import name Client

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-62cdWj/amsterdam/

Integrate manage.py (scbackups/screstore/listbackups) and curator with the "amsterdam" commands

Give us the ability to backup, restore and listbackups and manage elasticsearch data with the amsterdam command.

Now we have to execute:

docker exec amsterdam_scirius_1 python /opt/selks/scirius/manage.py scbackup
docker exec amsterdam_scirius_1 python /opt/selks/scirius/manage.py screstore
docker exec amsterdam_scirius_1 python /opt/selks/scirius/manage.py listbackups

docker exec amsterdam_elasticsearch_1 curator delete indices --prefix logstash-alert*

Kibana Init issue?

I just installed and am getting the below when I connect to Kibana. I also noticed a few "this will be depreacated messages". Wondering what I need to do to update/resolve this?

Configure an index pattern
In order to use Kibana you must configure at least one index pattern. Index patterns are used to identify the Elasticsearch index to run search and analytics against. They are also used to configure fields.
Index contains time-based events
Use event times to create index names [DEPRECATED]
Index name or pattern
Patterns allow you to define dynamic index names using * as a wildcard. Example: logstash-*
logstash-*

logstash_1 | Error: The setting host in plugin elasticsearch is obsolete and is no longer available. Please use the 'hosts' setting instead. You can specify multiple entries separated by comma in 'host:port' format. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask.
logstash_1 | You may be interested in the '--configtest' flag which you can
logstash_1 | use to validate logstash's configuration before you choose
logstash_1 | to restart a running system.
amsterdam_logstash_1 exited with code 1

scirius_1 | /usr/local/lib/python2.7/dist-packages/django_tables2/tables.py:175: RemovedInDjango19Warning: SortedDict is deprecated and will be removed in Django 1.9.
scirius_1 | attrs["base_columns"] = SortedDict(parent_columns)
scirius_1 |
scirius_1 | /usr/local/lib/python2.7/dist-packages/django_tables2/tables.py:197: RemovedInDjango19Warning: SortedDict is deprecated and will be removed in Django 1.9.

scirius_1 | Your models have changes that are not yet reflected in a migration, and so won't be applied.
scirius_1 | Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.

terminated by SIGSEGV (core dumped)

Hi.
I've installed amsterdam 0.7 on two different machines
And sometimes suricata crashes(?) with log:

suricata_1       | 2016-03-28 20:57:09,631 INFO exited: suricata (terminated by SIGSEGV (core dumped); not expected)
suricata_1       | 2016-03-28 20:57:10,634 INFO spawned: 'suricata' with pid 40
suricata_1       | 28/3/2016 -- 20:57:10 - <Notice> - This is Suricata version 3.0 RELEASE
suricata_1       | 2016-03-28 20:57:20,655 INFO success: suricata entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
suricata_1       | 28/3/2016 -- 20:57:22 - <Warning> - [ERRCODE: SC_ERR_FOPEN(44)] - Error opening file: "/etc/suricata/rules/threshold.config": No such file or directory
suricata_1       | 28/3/2016 -- 20:57:22 - <Notice> - all 4 packet processing threads, 4 management threads initialized, engine started.

What am i doing wrong?

Error on the command pip install amesterdam

Hi Stamus dev's and community,

While setup amesterdam I am getting module error on line 26
from OpenSSL import crypto
ImportError: No module named OpenSSL

Any Ideas to solve the issue will be appreciated.

Internal Server Error

Hi,

I've installed the latest version of Amsterdam and have containers running. When accessing localhost I can see everything. The problem is when accessing remotely by IP address. I get "Internal Server Error".

Do I have to configure additionally something?

Scirius not starting due to missing +x

Hi,

On a fresh ubuntu 14.04, I had to add a statement in the Dockerfile:
"RUN chmod ugo+x /opt/selks/bin/scirius.sh" to force execution right on scirius entrypoint script - otherwise container will not start with error "System error: exec: "/opt/selks/bin/scirius.sh": permission denied".

diff.txt

https access not working

After doing a fresh install on Ubuntu 16.04, everything appears to be working except the https access. I can open the individual apps by going to http://localhost:8000 and the other ports.

HTTP 404 when updating ruleset

Sometimes when updating the rules I am getting:
502 Bad Gateway

nginx/1.11.0

This happens on this page:

https://ips.corp.domain.com/suricata/update

scirius and user accounts

maybe i'm being stupid, which is possible of course, but how to you make accounts for scirius permanent. Now that you have to login to kibana via it, i'd like to set something that sticks rather than it disappearing every time the containers start.

Have i just overlooked something, or is there a config i can supply with pass details in it?

No dashboards under scirius menu

Disable offloading on sniffing iface

The start script should be upgraded to take care of offloading on sniffing iface.

Scirius - Internal Server Error

I am running the Scirius container and added ports 8000:8000 to the compose file. The container is running, and I have configured nginx to forward requests on port 8001 to 8000. I go to the web browser and do http://my_url:8001 - it returns 'Internal Server Error'

Kibana dashboards

I'm trying to load https://github.com/pevma/Suricata-Logstash-Templates/ into the docker, but i'm not sure where the dashboards are stored. Can you point me at the right spot? I tried importing through the web interface but its not accepting them for some reason.

Thanks.