stamparm/maltrail

Malicious traffic detection system

maltrail's Issues

Config issues

Hi,

On FreeBSD, 10.2 (64bits), I have this issue:

[i] using configuration file '/usr/maltrail/maltrail.conf'
free: not found
[!] unable to determine total physical memory. Please use absolute value for 'CAPTURE_BUFFER'

Sincerely

Add support for GeoIP

http://dev.maxmind.com/geoip/geoip2/downloadable/
http://dev.maxmind.com/geoip/legacy/downloadable/
https://www.maxmind.com/en/geoip2-city

Preventing IP blocked to be analyzed.

When I want to block an attacker, I null route him with "route add" (http://www.cyberciti.biz/tips/how-do-i-drop-or-block-attackers-ip-with-null-routes.html) but even after he is still displayed on the web interface in the first position in the column "last seen".
Maybe you should add an option to hide automatically IP who has been blocked.

Pushing lists from server to sensors

Hi,

It would be great if malware feeds are being pushed from server to sensors instead of sensors are downloading lists for distributed architectures where only server can reach the internet for rule-updates etc.. (Like a distributed security onion installation etc)

Filter by Severity High

When I filter with the word "high" or when i click "high" in the severity column not only filter "high" because in the info column we have "high consonant no such domain name".

bug cannot allocated memory

Trying to start the soft in an ubuntu 14.04 :

:~/maltrail$ sudo python sensor.py
Maltrail (sensor) #v0.8.187

[i] using configuration file '/home/user/maltrail/maltrail.conf'
[i] loading trails file...
[i] 921,479 trails loaded
[i] using '/var/log/maltrail' for log storage
[!] in case of any problems with packet capture on virtual interface 'any', please put all monitoring interfaces to promiscuous mode manually (e.g. 'sudo ifconfig eth0 promisc')
[i] opening interface 'any'
[i] setting filter '(tcp[13] == 2) or (tcp[13] & 8 != 0) or not tcp'
[i] creating 3 more processes (4 CPU cores detected)
[!] unhandled exception occurred ('[Errno 12] Cannot allocate memory')

[x] please report the following details at 'https://github.com/stamparm/maltrail/issues':

'Traceback (most recent call last):
File "sensor.py", line 531, in
main()
File "sensor.py", line 524, in main
init()
File "sensor.py", line 443, in init
_init_multiprocessing()
File "sensor.py", line 470, in _init_multiprocessing
_buffer = mmap.mmap(-1, BUFFER_LENGTH) # http://www.alexonlinux.com/direct-io-in-python
error: [Errno 12] Cannot allocate memory

'

If a can adding another data for fix this bug...

Enchant: Select more then one day of data

it could be very cool if we can select more then one day of available data to see more complete picture

Sending logs to Logstash (ELK) and Local

Hi there I think is good idea to let us setup in the config file config more than one "LOG_SERVER" or one LOG_SERVER and save that in local :-)

I use ELK solution and to let me use that with web server side I modify the "core/log.py" and in the section "def log_event(event_tuple)" in the first "if":

if config.LOG_SERVER:
remote_host, remote_port = config.LOG_SERVER.split(':')
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.sendto("%s %s" % (sec, event), (remote_host, int(remote_port)))

I add:

            handle = get_event_log_handle(sec)
            os.write(handle, event)

Then now I can send and save the logs :-)

What do you think about that improvement ?

Thanks a happy holidays

BUG: OverflowError: in Voipbl.py

here the exception:

[i] updating trails...
 [o] 'https://feodotracker.abuse.ch/blocklist/?download=domainblocklist'
[!] something went wrong during remote data retrieval ('https://feodotracker.abuse.ch/blocklist/?download=domainblocklist')
 [o] 'https://www.badips.com/get/list/any/2?age=30d'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/ri_web_proxies_30d.ipset'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/proxylists_30d.ipset'
 [o] 'http://www.botscout.com/last_caught_cache.htm'
 [o] 'http://vxvault.siri-urz.net/URL_List.php'
 [o] 'http://malwaredomains.lehigh.edu/files/domains.txt'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/socks_proxy_30d.ipset'
 [o] 'https://zeustracker.abuse.ch/blocklist.php?download=badips'
 [o] 'https://openphish.com/feed.txt'
 [o] 'https://myip.ms/files/blacklist/htaccess/latest_blacklist.txt'
 [o] 'https://www.maxmind.com/en/proxy-detection-sample-list'
 [o] 'https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist'
 [o] 'http://malwareurls.joxeankoret.com/normal.txt'
 [o] 'http://talosintel.com/feeds/ip-filter.blf'
 [o] 'https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.1.1.1'
 [o] 'https://rules.emergingthreats.net/open/suricata/rules/emerging-dns.rules'
 [o] 'http://www.nothink.org/blacklist/blacklist_malware_irc.txt'
 [o] 'http://danger.rulez.sk/projects/bruteforceblocker/blist.php'
 [o] 'http://malwared.malwaremustdie.org/rss.php'
 [o] 'http://www.voipbl.org/update/'
[!] Unhandled exception occurred ('Python int too large to convert to C long')
[x] Please report the following details at 'https://github.com/stamparm/maltrail/issues':

---
'Traceback (most recent call last):
  File "sensor.py", line 389, in <module>
    main()
  File "sensor.py", line 382, in main
    init()
  File "sensor.py", line 287, in init
    update_timer()
  File "sensor.py", line 275, in update_timer
    _ = update(server=config.UPDATE_SERVER)
  File "/opt/maltrail/maltrail/core/update.py", line 82, in update
    results = function()
  File "/opt/maltrail/maltrail/trails/feeds/voipbl.py", line 29, in fetch
    for address in xrange(start_int, end_int + 1):
OverflowError: Python int too large to convert to C long
'

---

Debian 6
Python 2.7.3

Thanks for your amazing work!!

stuck in loading screen

Browser keeps request http://serverip:8338/events?date=2015-12-16
until chrome tab crash

probably too many events? after i stop server.py, The dashboard shows some data

Button ban IP

As @stonfute said in this #30 (comment).
It would be awesome if you can implement a function to ban an IP on the web interface with a button !

Enhancement: Windows Sensor

Hello!
Would be great if there was a Windows sensor for this.
Use-Case: In most setups there is a mix of both Windows and Linux machines. I'd like to monitor both.

Cloudflare

If server is behind cloudflare it shows coudflare ip, not the real ip :) Pls fix it.

Issue with ports

Hey again. So I've been running the latest git pull for about 4 hours now. After fixing the Cisco VLAN issue (thanks much!) I am seeing src and dst IP's that look correct, but the ports do not. An example:

Src: 64.74.133.82
Src Port: 50887
Dst: x.x.x.x
Dst Port: 56863, 56902

Yet packet capturing during this time shows no hits on port 50887. Bro-ids does show 64.74.133.82, but Src port ranges from 33573-38544, with Dst port ranges of 33440-33444, this is a traceroute. Betting something isn't getting translated correctly. Thank you.

Multiple sensors

Ability to manage multiple sensors with the web interface.

OSX 10.11.1 psutil

core/Settings.py line 111
-- retval = psutil.phymem_usage().total (depreciated in my psutil package?)
++ retval = psutil.virtual_memory().total (works)

https://mail.python.org/pipermail/python-announce-list/2012-August/009575.html

unpack requires a string argument of length 8

Hi! Thanks for you work.

Trying to run the system on FreeBSD 10.0-RELEASE (Python 2.7.6). Default configs. Got this traceback:

root@iota:/tmp/3/maltrail-master # python2.7 sensor.py
Maltrail (sensor) #v0.8.359

[i] using configuration file '/tmp/3/maltrail-master/maltrail.conf'
[i] using '/var/log/maltrail' for log storage
[?] at least 512MB of free memory required
[i] loading trails file...
[i] 867,140 trails loaded
[x] virtual interface 'any' missing. Replacing it with all interface names
[i] opening interface 'em0'
[i] opening interface 'taurus0'
[i] opening interface 'libra0'
[i] opening interface 'em0.5'
[i] opening interface 'em0.100'
[i] opening interface 'em0.101'
[i] opening interface 'em0.102'
[i] opening interface 'em0.103'
...
[i] opening interface 'em0.600'
[i] opening interface 'lo0'
[i] setting filter 'ip and (not tcp or (tcp[13] == 2) or (tcp[13] & 8 != 0))'
[i] creating 1 more processes (2 CPU cores detected)
[?] please install 'schedtool' for better CPU scheduling
[o] running...
Process 0:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/tmp/3/maltrail-master/core/parallel.py", line 92, in worker
    sec, usec = struct.unpack("=II", content[:8])
error: unpack requires a string argument of length 8
[x] Ctrl-C pressed

The server starts fine:

root@iota:/tmp/3/maltrail-master # python2.7 server.py
Maltrail (server) #v0.8.359

[i] using configuration file '/tmp/3/maltrail-master/maltrail.conf'
[i] starting HTTP server at 'http://0.0.0.0:8338/'
[o] running...

The web-interface is up and running. I can see some threats there. But data doesn't update. It seems it analyse the traffic once (when starting) and then stops.

Can this error "error: unpack requires a string argument of length 8" cause the problem? Tnx!

Feature Request: Regular expression filters

Would be great to be able to apply some basic regex or lucene (http://www.lucenetutorial.com/lucene-query-syntax.html) type filtering in the input. For example:

NOT mass scanner 
malicious AND malware

Thank you!

Filtering requests

Would love to be able to filter by a port number. Would be nice also to have src/dst/port/protocol filtering like maybe something below:

USER_WHITELIST src_192.168.1.1_tcp_25
USER_WHITELIST src-192.168.1.1-udp-53

Thank you.

new features added in forked maltrail_dev

these are only my 2cents, unfortunately i usually use bitbucket for my "projects", so i show you what i have added to your very good project for meet my needed inside my honeynet sensor:

• i have just added User-Agent Heuristic as an optional configuration that can or cant be activated
• it will check UA for various string for Malware, Scanner, Crawler, Attacker
• it will report it in log because i think that UA will be a good information during IR for every type of HTTP events, so IMHO it could be added to all other HTTP Heuristic events if you think that can be useful.
• next improvement i think that will be to switch from regex to a trail so we can treat UA like IP and Domain with some auto updating feeds.

https://bitbucket.org/n3ophyte/maltrail_dev/commits/branch/develop-ua

i can recreate it in github and then make a pull-request to a development branch of your repo if you think that this can also by useful for you, obv iam not a developer so if you think that there will be a better way to deal with this implementation no problems :)

my next development will be to create a python instance and a caching mechanism for scanning all logfiles for create a single JSON document that contains all historical matches for further search and a better overview of aggregate data that every sensor are collecting, something like:

[IP/Domain] { 
     [first_seen]
     [last_seen]
     [hits]
     [UA] { [1] => .. [2] => ... }
     [Reliability] = from 10 to 0
}

Reliability can be from 10 to 0, where 10 is added when we first add this information to the JSON document and then every day or two (or whatever we think that could be a good security timeframe) we decrease that number if we dont see that object anymore in the sensor...
this could be integrated in the portal so we have a single method for searching all the events over the time without scanning billions of lines of files that can contains many time the same ip/domain/url

Not logging -ping

ping 202.99.224.68
not loging,why?
but nmap is loging.

1.➜ maltrail git:(master) ✗ sudo python sensor.py
[sudo] password for asrr:
Maltrail (sensor) #v0.8.288

[i] using configuration file '/home/asrr/tools/maltrail/maltrail.conf'
[i] loading trails file...
[i] 814,061 trails loaded
[i] using '/var/log/maltrail' for log storage
[i] opening interface 'eth0'
[i] setting filter '(tcp[13] == 2) or (tcp[13] & 8 != 0) or not tcp'
[i] creating 3 more processes (4 CPU cores detected)
[o] running...

2.➜ maltrail git:(master) ✗ ip route get 202.99.224.68
202.99.224.68 via 172.16.9.70 dev eth0 src 172.16.9.67
cache

3.➜ maltrail git:(master) ✗ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 3c:97:0e:e8:3b:30
inet addr:172.16.9.67 Bcast:172.16.9.71 Mask:255.255.255.248
inet6 addr: fe80::3e97:eff:fee8:3b30/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:21314 errors:0 dropped:1 overruns:0 frame:0
TX packets:20767 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:17267930 (17.2 MB) TX bytes:2672368 (2.6 MB)

4.➜ maltrail git:(master) ✗ cat ~/.maltrail/trails.csv |grep 202.99.224.68
➜ maltrail git:(master) ✗

5.➜ maltrail git:(master) ✗ cat maltrail.conf | grep MONITOR_INTERFACE

MONITOR_INTERFACE \Device\NPF_{70D09A93-BDAB-4F2E-B4BE-5DCAE73AAF64}

MONITOR_INTERFACE eth0
➜ maltrail git:(master) ✗

6.cat 2015-12-26.log|grep 202.99.224
"2015-12-26 12:17:54.146010" weizi-linux 172.16.9.67 59850 202.99.224.68 53 UDP DNS bluereader.org malware otx.alienvault.comu

but not ping

Authentication Issue

Even when using the default password, the authentication does not work properly. I've tried setting a new password for the admin account but the submitted password never seems to match so I cannot view the user interface.

My suspicion is that it has to do with the server.py running on windows for some reason. I've tried looking at the authentication code in httpd.py but the main thing is that somewhere along there, the Session value never gets set properly due to issues reading from the config file. The only difference I really have is with the log paths being windows based () instead of unix based (/). Any thoughts on what might be causing the authentication issue with the default setup?

Why SHA256 and not SHA512

How can I use SHA512 and not SHA256 password?

False Positive?

Hi,

I have the sensor and server on installed on the same box.

It seems that a legitimate traffic to a feed server (vxvault) is flagged as malicious (high).

Sorry if I am confused :-)

Sensor and Server shutdown

Hi, both of services shutdown 2-3 hours since start up.
I launch it so:

python sensor.py &
python server.py &

What I doing bad ?

Regards!

Filter by country

Would be great to double-click a flag and have it pop the two character country code into the filter field and filter by country. Thank you.

Sensor shutdown

When I scan IPs range sensor shutdown and need scan again... Where is problem? Is there timeout for sensor or what?

Feature Request: Virustotal integration

Would be awesome to have the Trail popup use virustotal to lookup the IP address. Thank you!

Running but not logging anything

root@xyz:/var/log/maltrail# ping -c 1 136.161.101.53
PING 136.161.101.53 (136.161.101.53) 56(84) bytes of data.
64 bytes from 136.161.101.53: icmp_seq=1 ttl=54 time=81.6 ms

--- 136.161.101.53 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 81.669/81.669/81.669/0.000 ms

root@xyz:/var/log/maltrail# cat /var/log/maltrail/$(date +"%Y-%m-%d").log
cat: /var/log/maltrail/2015-12-16.log: No such file or directory

Cannot look at one day ago - No connection to the server

One day all was OK. Also I could look one day back. Now when I want to do it still got the same error as in Screenshot.
No exception in screen process with server none in error.log.
After reload all is OK, but I can see only today.

Maltrail sensor use a bunch of my ram after a long period of running.

My sensor was running since 26 december and I saw that the process "sensor" was using 60% of my total ram (2GO).
I don't really know why it using so much ram because if I restart it, sensor now use only 13% of my ram.

root@unixfox:~# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 16893 0.0 39.9 1080768 837880 ? Ssl 2015 10:08 /usr/bin/python2 /root/maltrail/sensor.py
root 16896 0.9 21.9 994892 461316 ? Sl 2015 146:11 /usr/bin/python2 /root/maltrail/sensor.py

login

Hello,
first congrats for your job.

I wrote you because i try to change the password in the maltrail.conf, and now i can't connect to the server...
here is the mailtrail.conf user's section
USERS
#admin:$ff0ae5570e1f39a8$10000$d42e622afe0b0ede53b64b97a59d65c221edbf9dde2f0e95:0:0.0.0.0/0 # changeme!
toto:$8c81af41b9a90583$10000$650a0db5a4df76fee8a2557c1388be3c2b0b16362c423680:0:0.0.0.0/0

i ran on windows and all was ok before i change the password.

Thanks for your help

VPN Configuration

Good evening.

I write to ask if anyone knows how to make a confguration so you can connect to the server through a VPN.
In local network work correctly. But when trying to connect to the server through a VPN, which is not put in the IP address settings.
For your reply, thanks.

Add support for The WebGL Globe

http://www.chromeexperiments.com/globe
https://github.com/dataarts/webgl-globe

Not seeing traffic

So I have a machine that listens to several netblocks. I am wanting to listen to just the external traffic. This external traffic only has routable IP addresses, we don't see any internal only (10.0.0.0/8, 192.168.0.0/24 for example) traffic. This traffic is on eth2:

MONITOR_INTERFACE eth2

This interface is in promiscuous mode:

eth2      Link encap:Ethernet  HWaddr 00:00:00:00:00:01
          UP BROADCAST RUNNING NOARP PROMISC MULTICAST  MTU:1500  Metric:1

I do not see any events on this interface, even though I do see hits from say, bro-ids:

139.196.104.39  58928   x.x.x.x       443     -       -       -       139.196.104.39  Intel::ADDR     Conn::IN_ORIG   bro     alienvault

If I set maltrail to any, I see hits, but only internal. Is there something I can do to troubleshoot this? Thaank you.

Feature Request: Refresh

First off THIS APP IS AWESOME. Just started with it today and it rocks. Second, any chance we can get an autorefresh workin on this? Thanks so much!

Enhancement: Support for vhost

Support for showing which vhost is targeted would be really useful, and fairly easy to extract from the traffic.

Integrate with firewall rules, perhaps via a plugin architecture?

Maltrail is great!

However, it captures packets before host-based firewalls see those packets, resulting in a lot of false positives. That is, if your firewall is already blocking the packets from blacklisted IPs, your host is protected from that threat. While it is interesting and educational to know what is being directed towards your hosts, it would be more useful to know what got past the firewall.

I don't want to whitelist my blacklisted IP addresses, in case the firewall rules change (or the firewall is off) and the firewall inadvertently is allowing some blacklisted IP addresses through.

Ideally, maltrail would use my firewall rules/blacklists to prepend a comment character to the logged line. Commented log lines would normally be ignored by the client when the web display is generated, but could be re-instated in the display if desired to peek at activity behind the firewall.

Perhaps allowing plugins at critical points in processing would be a better idea and more general. We would then write the code to do the special filtering that we desire.

Problem with logging (i think)

Hello stamparm,

i'm glad to discover this tool, could be useful to use it. That's why i've started to try it and at this time i'm not able to do what is it supposed to do.

First of all, i've nothing on the webserver, saying there's no event. But the calendar show event and the number increase.
I've started in debug, it's an ubuntu 14.04.

To begin, in documentation, you don't use sudo to start the webserv daemon, and i got a perm error to read logs. It could be awesome to dos something who will work at the begin without needs of chown or chmod.

And the problem i have when starting the webserver as sudo/root :

[i] starting HTTP server at 'http://0.0.0.0:8338/'
[o] running...
Traceback (most recent call last):
  File "/home/admin/maltrail/core/httpd.py", line 514, in _counts
    current = datetime.datetime.strptime(os.path.splitext(os.path.basename(filename))[0], DATE_FORMAT)
  File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
    (data_string, format))
ValueError: time data 'error' does not match format '%Y-%m-%d'

My sensor log file looks like this :

"2015-12-29 03:57:27.818642" maltrail 149.202.238.199 41049 92.x.x.x 8888 TCP IP 149.202.238.199 "bad reputation (malicious)" alienvault.com
"2015-12-29 03:59:53.915040" maltrail 92.x.x.x - 136.161.101.53 - ICMP IP 136.161.101.53 "conficker (malware)" (static)
"2015-12-29 04:03:56.389039" maltrail 92.x.x.x - 136.161.101.53 - ICMP IP 136.161.101.53 "conficker (malware)" (static)

I'm trying too to put a sensor on an other server, but actually it's not able to send logs to webserver. I'll tell you more in other ticket if i don't find how to fix this.

Thanks again for this good job.

Add a link to go back to home on the top left logo

Hello,

could be useful to have a buton to click to go back to the main view, without the need to clear the filter.

Thank you !

Not logging - sorry first issue closed

Hello.

I have problem with logging like this user on #14 . My server run on 0.0.0.0:8338 and sensors running too (opening interface 'any'). Server run without sudo and sensors with sudo command. But there is no log when ping some IP. I can login on my server panel successfully. First I try with screen on server but nothing happen then I install VNC and run on VNC and this not problem. My configuration is default.

Please help me, thank you.

Some new sources - having a loooot unique IPs

https://github.com/firehol/blocklist-ipsets

install Sensor on Centos?

Hi, I am trying to install a sensor on Centos 6? Is possible?

Thanks in advance!

Whitelist doesn't appear to be working

So here's what I got from my maltrail.conf:

USER_WHITELIST 127.0.0.1,localhost,71.21.81.200

But I continue to see this IP address in the list. Sensor running in debug doesn't show anything. I'm also specifically looking at the last seen time. Thank you!

Feature Request : autobans/manual bans

I love the software well done, easy to use, but so far it's a passive tool.
Adding a button that would launch :
iptables -I INPUT -s xxx.xxx.xxx.xxx -j DROP (xxx beeing the offender IP) on each offender line.
Could even add it as an autobanning feature, with maybe a treshold like fail2ban,
That would certainly make my day :)
(could even make it an option to autoban the known mass scanners at first start)

(I'm not forgiving, my jail2ban jails ban the 1st time a week, in case of recidive a year, but fail2ban is not that easy to configure, especially for new jails, this, with banning features, could really be a better solution)

Feature Request: Right-click options

Would be awesome to have the ability to, perhaps, highlight an IP address and say copy it...or perform additional tasks (look up in virustotal or urlquery or urlvoid). Thank you!

whoami not found

After starting sensor and server, on web interface I get 'The requested url /maltrail/html/whoami not found'

Responsive

My mobile device can't really load maltrail because it've to render the interface like a PC.
Is it possible to have responsive on the web interface or support mobile devices ?

javascript error

Worked fine for a few hours, then I'm getting hundreds of javascript alerts

Uncaught TypeError: Cannot read property 'html' of undefined Script: http://xxxxxx.com:8338/js/main.js Line:1387

The URL of vxvault has changed.

From http://vxvault.siri-urz.net/URL_List.php to http://vxvault.net/URL_List.php.

related to /trails/feeds/vxvault.py