A couple issues

[mitchellkrogza]

A couple issues

[m411b]

I did actually try adding sudo to the sort command and there was no change.
It's weird for sure. Changing the sort command to two lines seems to be the trick. I'll leave it that way since it works and focus on blacklist banning other jails. I'll report back.

Thanks so much

[lls-hcr]

I had a very similar problem recently on debian (buster). Unfortunately, I could not find a solution. If a solution is found, I'll be very much interested. Maybe sharing your jail.local file could be interesting.

[m411b]

I think the sort command in the action.d blacklist.conf file is not working correctly. It's not removing duplicate entries into the file ip.blacklist. The -u option should do both sort and remove, but it's only sorting and not removing. By the time the -u option gets done, sort is already adding the new IP entry into the ip.blacklist file with -o option. Which creates the dupe in the ip.blacklist file and consequently on iptables.

sort -u /etc/fail2ban/ip.blacklist -o /etc/fail2ban/ip.blacklist

EDIT:
I fixed my problem by separating the sort command in the /etc/fail2ban/action.d/blacklist.conf file like this -

sort -u /etc/fail2ban/ip.blacklist
sort -o /etc/fail2ban/ip.blacklist

No more dupes and everything is sorted correctly.

[mitchellkrogza]

Weird it should work in a one line, always has and does work this way on most distributions.
What distro are you using?

[mitchellkrogza]

Is there a way to get this to watch other filters than just SSH? Like say [Postfix-Auth] or any other custom filter? So far this filter seems to only care about SSH.

@m411b this filter blacklists anything not just SSH. You happen to see a lot of SSH because that's the most common exploit. This filter does not pick and choose what to blacklist ... it blacklists anything that was blacklisted from any jail by Fail2ban. It works by only processing what is blacklisted in the fail2ban.log

You can see the regex pattern used to scan through the fail2ban.log file

https://github.com/mitchellkrogza/Fail2Ban-Blacklist-JAIL-for-Repeat-Offenders-with-Perma-Extended-Banning/blob/master/filter.d/blacklist.conf#L61

GitHub

mitchellkrogza/Fail2Ban-Blacklist-JAIL-for-Repeat-Offenders-with-Perma-Extended-Banning

A customised jail with action and filter file for Fail2Ban. This jail is based on the recidive jail but makes use of a simple text file to enable extended and permanent bans. - mitchellkrogza/Fail2...

[m411b]

@mitchellkrogza Why no blacklist ban on 185.234.219.113? Maxretry is set to 2. For postfix-sasl I'm using the built in settings in jail.conf/local just like with SSH. But the blacklist filter ignores everything other than SSH. I've gone through the settings and filters multiple times and cannot figure out why? I love the blacklist filter and thank you very much for sharing it. It works great, just not for anything other than SSH on my system. At least that's the way it seems.

2019-12-19 09:19:37,210 fail2ban.filter [7456]: INFO [postfix-sasl] Found 185.234.219.113 - 2019-12-19 09:19:37
2019-12-19 09:19:37,381 fail2ban.filter [7456]: INFO [postfix-auth] Found 185.234.219.113 - 2019-12-19 09:19:37
2019-12-19 09:33:36,275 fail2ban.filter [7456]: INFO [postfix-sasl] Found 185.234.219.113 - 2019-12-19 09:33:36
2019-12-19 09:33:36,443 fail2ban.filter [7456]: INFO [postfix-auth] Found 185.234.219.113 - 2019-12-19 09:33:36
2019-12-19 09:47:53,375 fail2ban.filter [7456]: INFO [postfix-sasl] Found 185.234.219.113 - 2019-12-19 09:47:53
2019-12-19 09:47:53,498 fail2ban.filter [7456]: INFO [postfix-auth] Found 185.234.219.113 - 2019-12-19 09:47:53
2019-12-19 10:02:08,254 fail2ban.filter [7456]: INFO [postfix-sasl] Found 185.234.219.113 - 2019-12-19 10:02:08
2019-12-19 10:02:08,437 fail2ban.filter [7456]: INFO [postfix-auth] Found 185.234.219.113 - 2019-12-19 10:02:08
2019-12-19 10:16:12,342 fail2ban.filter [7456]: INFO [postfix-sasl] Found 185.234.219.113 - 2019-12-19 10:16:12
2019-12-19 10:16:12,343 fail2ban.filter [7456]: INFO [postfix-auth] Found 185.234.219.113 - 2019-12-19 10:16:12
2019-12-19 10:30:12,378 fail2ban.filter [7456]: INFO [postfix-sasl] Found 185.234.219.113 - 2019-12-19 10:30:12
2019-12-19 10:30:12,406 fail2ban.filter [7456]: INFO [postfix-auth] Found 185.234.219.113 - 2019-12-19 10:30:12
2019-12-19 10:44:00,272 fail2ban.filter [7456]: INFO [postfix-sasl] Found 185.234.219.113 - 2019-12-19 10:44:00
2019-12-19 10:44:00,416 fail2ban.filter [7456]: INFO [postfix-auth] Found 185.234.219.113 - 2019-12-19 10:44:00
2019-12-19 10:58:05,390 fail2ban.filter [7456]: INFO [postfix-sasl] Found 185.234.219.113 - 2019-12-19 10:58:05
2019-12-19 10:58:05,390 fail2ban.filter [7456]: INFO [postfix-auth] Found 185.234.219.113 - 2019-12-19 10:58:05
2019-12-19 11:12:21,108 fail2ban.filter [7456]: INFO [postfix-sasl] Found 185.234.219.113 - 2019-12-19 11:12:21
2019-12-19 11:12:21,210 fail2ban.filter [7456]: INFO [postfix-auth] Found 185.234.219.113 - 2019-12-19 11:12:21
2019-12-19 11:26:38,355 fail2ban.filter [7456]: INFO [postfix-sasl] Found 185.234.219.113 - 2019-12-19 11:26:38
2019-12-19 11:26:38,484 fail2ban.filter [7456]: INFO [postfix-auth] Found 185.234.219.113 - 2019-12-19 11:26:38
2019-12-19 11:33:08,716 fail2ban.filter [7456]: INFO [sshd] Found 218.92.0.178 - 2019-12-19 11:33:08
2019-12-19 11:33:08,922 fail2ban.filter [7456]: INFO [sshd] Found 218.92.0.178 - 2019-12-19 11:33:08
2019-12-19 11:33:09,625 fail2ban.filter [7456]: INFO [sshd] Found 218.92.0.178 - 2019-12-19 11:33:09
2019-12-19 11:33:09,629 fail2ban.actions [7456]: NOTICE [sshd] Ban 218.92.0.178
2019-12-19 11:33:09,633 fail2ban.filter [7456]: INFO [blacklist] Found 218.92.0.178 - 2019-12-19 11:33:09
2019-12-19 11:33:11,246 fail2ban.filter [7456]: INFO [sshd] Found 218.92.0.178 - 2019-12-19 11:33:11
2019-12-19 11:36:31,803 fail2ban.filter [7456]: INFO [sshd] Found 37.24.51.142 - 2019-12-19 11:36:31
2019-12-19 11:36:31,864 fail2ban.filter [7456]: INFO [sshd] Found 37.24.51.142 - 2019-12-19 11:36:31
2019-12-19 11:36:31,961 fail2ban.filter [7456]: INFO [sshd] Found 37.24.51.142 - 2019-12-19 11:36:31
2019-12-19 11:36:32,023 fail2ban.filter [7456]: INFO [sshd] Found 37.24.51.142 - 2019-12-19 11:36:32
2019-12-19 11:36:32,186 fail2ban.actions [7456]: NOTICE [sshd] Ban 37.24.51.142
2019-12-19 11:36:32,191 fail2ban.filter [7456]: INFO [blacklist] Found 37.24.51.142 - 2019-12-19 11:36:32
2019-12-19 11:36:33,981 fail2ban.filter [7456]: INFO [sshd] Found 37.24.51.142 - 2019-12-19 11:36:33
2019-12-19 11:36:34,184 fail2ban.filter [7456]: INFO [sshd] Found 37.24.51.142 - 2019-12-19 11:36:34
2019-12-19 11:36:34,572 fail2ban.actions [7456]: NOTICE [sshd] 37.24.51.142 already banned
2019-12-19 11:40:56,181 fail2ban.filter [7456]: INFO [postfix-auth] Found 185.234.219.113 - 2019-12-19 11:40:56
2019-12-19 11:40:56,183 fail2ban.filter [7456]: INFO [postfix-sasl] Found 185.234.219.113 - 2019-12-19 11:40:56
2019-12-19 11:43:09,133 fail2ban.actions [7456]: NOTICE [sshd] Unban 218.92.0.178
2019-12-19 11:46:35,448 fail2ban.actions [7456]: NOTICE [sshd] Unban 37.24.51.142

[mitchellkrogza]

Post your jail settings all of them

What is your logrotate set to for fail2ban.log ???

[m411b]

jail.local -

[DEFAULT]
bantime = 10m
findtime = 10m
maxretry = 2
ignoreip = 127.0.0.1/8 ::1 xxx.xxx.xxx.xxx
port = 0:65535
filter = %(name)s

ACTIONS

destemail = xxx
sender = xxx

sendername = Fail2ban
mta = sendmail
action = %(action_mwl)s

JAILS

[pure-ftpd]
enabled = true
port = ftp
filter = pure-ftpd
logpath = /var/log/syslog
maxretry = 2

[dovecot]
enabled = true
filter = dovecot
action = iptables-multiport[name=dovecot-pop3imap, port="pop3,pop3s,imap,imaps", protocol=tcp]
logpath = /var/log/mail.log
maxretry = 2

[postfix]
enabled = true
port = smtp
filter = postfix
logpath = /var/log/mail.log
maxretry = 2

[postfix-auth]
enabled = true
filter = postfix.auth
action = iptables-multiport[name=postfix, port="http,https,smtp,submission,pop3,pop3s,imap,imaps,sieve", protocol=tcp]
logpath = /var/log/mail.log
maxretry = 2

[postfix-sasl]
enabled = true
filter = postfix[mode=auth]
port = smtp,465,submission,imap,imaps,pop3,pop3s
logpath = /var/log/mail.log
backend = %(postfix_backend)s
maxretry = 2

[blacklist]
enabled = true
logpath = /var/log/fail2ban.*
filter = blacklist
banaction = blacklist
bantime = 31536000 ; 1 year
findtime = 31536000 ; 1 year
maxretry = 3

[recidive]
enabled = false
logpath = /var/log/fail2ban.log
banaction = %(banaction_allports)s
bantime = 1w
findtime = 1d

logrotate -
/var/log/fail2ban.log {

monthly
rotate 13
compress

delaycompress
missingok
postrotate
    fail2ban-client flushlogs 1>/dev/null
endscript

# If fail2ban runs as non-root it still needs to have write access
# to logfiles.
# create 640 fail2ban adm
create 640 root adm

}

[m411b]

By the way. Thanks so much for the help.

[m411b]

I realized I had 3 postfix jails enabled. I've disabled postfix-auth and postfix. Postfix-sasl is the only postfix jail currently enabled.

[m411b]

Revisiting the sort issue where sort - sorts, but does not remove dupes. I wanted to see if changing the sort command back to the original one line command with options -u and -o in the /action.d/blacklist.conf file would fix itself after squaring everything else away. It did not. I did some looking around and found out sort will not make changes to the original data file. The sort command in /action.d/blacklist.conf with the option -o has sort writing back to the original data file. I wonder if making sort write to a second ip.blacklist file and then using that file for the blacklist will clear this up?

Changing the command back with options -u and -o separated into two lines fixed the issue again. Not sure exactly why. But I suspect it has something to do with sort writing back to the original data file.

On to the issue where the blacklist jail does nothing for other jails. Blacklist did ban a few pure-ftpd attackers. But nothing else. Only sshd and pure-ftpd are being blacklisted. While still ignoring postfix-sasl and apache jails. No idea why.

@mitchellkrogza - I did go read and pay very well attention to your fail2ban tutorial and went back through my jail.local file to set everything up correctly and in line with your tutorial. Basically to see if my jail.local file was out of wack. Still no change.

For reference -
Ubuntu Server 18.04.3 - 4.15.0-72-generic
Fail2ban 0.10.2

Thanks so much

[mitchellkrogza]

I use sort -u file -o file in so many projects, in fact every single one here on GitHub and it always does the replacement in place on the file I feed it. The only thing that could prevent it doing so is permissions. Have you tried changing that sort command line, just to test, by adding sudo to the beginning? Would be interesting to know as I suspect a permission issue somewhere. If all else fails there's nothing wrong with giving a different input and output file and just making sure this filter load the blacklist from the correct, sorted, output file. Although this should not be needed as I run it just the way it is and the file is always sorted. I cannot understand why some jail's are ignored or those repeat offenders are not being blacklisted. I have repeat offenders from all my jail's automatically blacklisted. Very strange.