waterbywind/edgeos-bl-mgmt

Automated updating of EdgeOS firewall network-group to be used as source address blacklist

Ubiquiti Networks EdgeRouter Firewall BlockList Management

---

Automated management of network and host address blocklists, for use in EdgeRouter (EdgeOS) firewall rules.

Quick Start

To get started, perform these steps on your EdgeRouter from a CLI configure prompt:

1. For IPv4: set firewall group network-group Nets4-BlackList description 'Blacklisted IPv4 Sources'
2. For IPv6: set firewall group ipv6-network-group Nets6-BlackList description 'Blacklisted IPv6 Sources'
3. cp updBlackList.sh /config/scripts/updBlackList.sh
4. cp fw-BlackList-URLs.txt /config/user-data/fw-BlackList-URLs.txt
5. cp loadBlackList.sh /config/scripts/post-config.d/loadBlackList.sh
6. set system task-scheduler task Update-Blacklists executable path /config/scripts/updBlackList.sh
7. set system task-scheduler task Update-Blacklists interval 12h
8. sudo /config/scripts/updBlackList.sh

You will also need to create a firewall rule to deny inbound source addresses that match this network-group Nets4-BlackList. An example using a zone-based firewall might look like:

1. set firewall name wan-dmz-4 rule 1 source group network-group Nets4-BlackList
2. set firewall name wan-dmz-4 rule 1 action drop
3. set firewall name wan-dmz-4 rule 1 protocol all
4. set firewall name wan-lan-4 rule 1 source group network-group Nets4-BlackList
5. set firewall name wan-lan-4 rule 1 action drop
6. set firewall name wan-lan-4 rule 1 protocol all
7. set firewall name wan-self-4 rule 1 source group network-group Nets4-BlackList
8. set firewall name wan-self-4 rule 1 action drop
9. set firewall name wan-self-4 rule 1 protocol all

Similar for IPv6:

1. set firewall ipv6-name wan-dmz-6 rule 1 source group ipv6-network-group Nets6-BlackList
2. set firewall ipv6-name wan-dmz-6 rule 1 action drop
3. set firewall ipv6-name wan-dmz-6 rule 1 protocol all
4. set firewall ipv6-name wan-lan-6 rule 1 source group ipv6-network-group Nets6-BlackList
5. set firewall ipv6-name wan-lan-6 rule 1 action drop
6. set firewall ipv6-name wan-lan-6 rule 1 protocol all
7. set firewall ipv6-name wan-self-6 rule 1 source group ipv6-network-group Nets6-BlackList
8. set firewall ipv6-name wan-self-6 rule 1 action drop
9. set firewall ipv6-name wan-self-6 rule 1 protocol all

To use the optional iprange for optimization and reduction you will need to install the binary. There is an existing iprange .deb package for both mips and mipsel that may be used.

1. If apt repositories have been configured: sudo apt install iprange
2. If apt has not been configured:
   1. mkdir -p /config/data/firstboot/install-packages
   2. cd /config/data/firstboot/install-packages
   3. For Cavium-based platforms (MIPS):
      1. curl -O http://http.us.debian.org/debian/pool/main/i/iprange/iprange_1.0.3+ds-1_mips.deb
      2. sudo dpkg --install iprange_1.0.3+ds-1_mips.deb
   4. For MTK-based platforms (MIPSEL; including ER-X, ER-X-SFP, EP-R6):
      1. curl -O http://http.us.debian.org/debian/pool/main/i/iprange/iprange_1.0.3+ds-1_mipsel.deb
      2. sudo dpkg --install iprange_1.0.3+ds-1_mipsel.deb

That should get you going with minimal effort. However, you really should review fw-BlackList-URLs.txt and edit as appropriate. The two scripts both have configuration sections that you should also review and edit as appropriate.

Network and host blocklist IPset creation script

updBlackList.sh is the actual tool that will fetch various blocklists, consolidate into one group, and load into a pre-existing IPset already used with firewall rulesets.
You may schedule this appropriately for your needs, but the quick start above provides for a twice-daily update at noon and midnight. Note that most list providers have guidelines on frequency of retrieval so these should be taken into account when setting the schedule.

List of URLs containing blacklists

The file fw-BlackList-URLs.txt should contain the list of URLs to individual blocklist of networks and hosts. Blank lines and comments (beginning with a hash (#) are acceptable).
The sample provided here has many sources listed as a starting point, though only some are enabled (uncommented). Note that several of the indicated lists either overlap or fully include other lists so it is not necessary nor is it recommended to blindly uncomment all URLs here. 'cURL' will be used to fetch each of these individually, exactly as listed.

Boot-time IPset reload script

Since the updBlackList.sh tool directly manipulates IPsets and does not reflect any changes in the EdgeOS config.boot, contents of the blocklist network-groups are lost upon reboot/restart of the EdgeRouter.
The loadBlackList.sh script addresses this by restoring the previously saved network-groups at boot time.
Alternatively, the updBlackList.sh script may be re-run at boot-time. This will take longer, however, since it will recreate the list newly.

Using iprange

iprange may be used to optimize lists of network addresses, particularly when merging multiple different lists from multiple sources. Overlaps and consolidation is standard with additional ability to reduce the number of prefix lengths for runtime efficiency when using the created IPsets.

iprange also provides for a true proper whitelist mechanism, automatically splitting address blocks around the whitelisted addresses followed by the above optimization and reduction.

Unfortunately iprange only supports IPv4 currently so there would be no optimization for IPv6 IPsets (yet)

By default, if iprange is found it will be used with nothing more than the installation noted above being required. Note that if 'apt' is used to install iprange you will need to re-install after each firmware update, though 'apt' sources would need to be reconfigured at the same time anyway. If you download the .deb and place it in the directory referenced above, however, this will be re-installed automatically after each firmware upgrade so that may actually be a preferred option.

More detail

IPset documentation may be found at http://ipset.netfilter.org

IPrange documentation may be found at https://github.com/firehol/iprange/wiki

This work was inspired by the Emerging Threats Blacklist discussion thread in the Ubiquiti Networks community forums.

This utility is one option among several for ingress filtering, and is primarily intended to be used in lieu of, or perhaps in supplement of, (egress-based) blackhole-routing and BGP route filtering.

Note

Arbitrary lists should not be summarily added or enabled for blocking. Review of each list should be performed combined with a period of careful monitoring after enabling to ensure legitimate traffic is not affected. Some public "blocklists" are known to be rather aggressive and add addresses/netblocks too easily.

edgeos-bl-mgmt's Issues

Silence sed warning

To silence the following error message from sed, apply the patch below:
sed: character class syntax is [[:space:]], not [:space:]

Patch:

diff --git a/updBlackList.sh b/updBlackList.sh
index d04caca..246b9d0 100755
--- a/updBlackList.sh
+++ b/updBlackList.sh
@@ -475,7 +475,7 @@ doProcess4()
     cat ${flBlockList} | \
         sed -e '/^[;#]/d' \
             -e '/ERROR/d' \
-            -e '/[:\::]/d' \
+            -e '/[\:\::]/d' \
             -e 's/ .*//g' \
             -e 's#//.*##g' \
             -e 's/[^0-9,.,/]*//g' \

Firewall rules not accepted

I pretty new to this, but trying to commit the example fw rules gives me this?

sav[ firewall ipv6-name wan-dmz-6 ]
Firewall config error: 'action' must be defined

e[ firewall name wan-dmz-4 ]
Firewall config error: 'action' must be defined

[ firewall name wan-lan-4 ]
Firewall config error: 'action' must be defined
[ firewall name wan-self-4 ]
Firewall config error: 'action' must be defined

[ firewall ipv6-name wan-self-6 ]
Firewall config error: 'action' must be defined

[ firewall ipv6-name wan-lan-6 ]
Firewall config error: 'action' must be defined

Can't confirm rulesets are applied

Can you provide steps to confirm that the new rulesets have been applied?

EdgeRouter PoE v1.9.7+hotfix.3

Update process stops when trying to add a wrong IP syntax

One of the sources contained a wrong IP (195.78.82.0/234 China)

which returned a error
ipset v6.23: Error in line 32227: Syntax error: '234' is out of range 0-32

This stops the whole process of updating.
Could there be a filter that would ignore such wrong entries?

Trying to modify, getting error line 255: /dev/fd/62: No such file or directory

Any idea why I might be seeing this error? I'm trying to modify this script as per: https://community.ubnt.com/t5/EdgeMAX/statuscake-auto-address-group-population/m-p/1509015#M102637

But I can't seem to get it to work. This is on edgeos 1.9.0.

Thanks!

Firewall Block List Not Populating

UDM PRO

Anyone looked at making this work on UDM/UDMPRO?

No longer works on USG

USG has moved to a new labelling system for firewall groups - instead of using the actual name they use 'hashed names'. Would be great to fix this somehow...

Config question

More a question than an issue.
I have this setup as per the readme however it is not 100% clear to me, should the FW policies (e.g. WAN - LAN ) have interfaces defined?
same question for Nets4-BlackList etc, should they have a network defined? (currently empty)

Many thanks :)

What about applying these rules also for egress traffic?

I'm thinking of the specific case of a malware talking to its C&C server via UDP. These rules won't block the upload traffic. Given the growing number of ransomware with the goal of stealing private data, a upload-only UDP connection should be enough for them for their job.

user error - please ignore

woops, wrong project

https://isc.sans.edu/ipsascii.html not being parsed correctly

Including the https://isc.sans.edu/ipsascii.html blacklist results in the following error:

iprange: Invalid address 185.176.026.10018701290432019021320190417.
iprange: Cannot understand line No 56515 from /tmp/.BL/updBlackList.2-5TS8vD: 185.176.026.10018701290432019021320190417

IPv6 Parsing

Just want to start by saying the Blacklist Management project is awesome, and you're awesome for maintaining it! Just wanted to report an issue using the default lists FW-Blacklist-URLs, it seems parsing IPv6 is broken at this time. I had to remove each of the IPv6 lists, including BlockList.de which had a few IPv6 addresses, in order for updBlackList.sh to succeed. After removing each list which contained IPv6 addresses, I was sucessfully able to import the IPSet:
IPv4 blocklist items fetched: 23470, unique: 21522, final: 21522
Total IPv4 prefix length count (including hosts): 27
IPv6 blocklist items fetched: 0, unique: 0, final: 0
Total IPv6 prefix length count (including hosts): 0

Currently on EdgeRouter X v2.0.8 but also tried on 1.10.10

ubnt@ubnt:~$ sudo /config/scripts/updBlackList.sh Missing executable '/usr/bin/iprange'. Will not optimize IPsets Starting at 00:52:56 MST Wed 11 Dec 2019 Fetching 'https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt' Fetching 'https://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt' Fetching 'https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt' Fetching 'https://www.spamhaus.org/drop/drop.txt' Fetching 'https://www.spamhaus.org/drop/dropv6.txt' Fetching 'https://www.spamhaus.org/drop/edrop.txt' Fetching 'https://lists.blocklist.de/lists/all.txt' Fetching 'https://iplists.firehol.org/files/firehol_level1.netset' Fetching 'https://www.okean.com/chinacidr.txt' Processing block file list (IPv4): ' 00_www.team-cymru.org_fullbogons-ipv4.txt 01_www.team-cymru.org_fullbogons-ipv6.txt 02_rules.emergingthreats.net_emerging-Block-IPs.txt 03_www.spamhaus.org_drop.txt 04_www.spamhaus.org_dropv6.txt 05_www.spamhaus.org_edrop.txt 06_lists.blocklist.de_all.txt 07_iplists.firehol.org_firehol_level1.netset 08_www.okean.com_chinacidr.txt' Applying IPset (IPv4) Processing block file list (IPv6): ' 00_www.team-cymru.org_fullbogons-ipv4.txt 01_www.team-cymru.org_fullbogons-ipv6.txt 02_rules.emergingthreats.net_emerging-Block-IPs.txt 03_www.spamhaus.org_drop.txt 04_www.spamhaus.org_dropv6.txt 05_www.spamhaus.org_edrop.txt 06_lists.blocklist.de_all.txt 07_iplists.firehol.org_firehol_level1.netset 08_www.okean.com_chinacidr.txt' Applying IPset (IPv6) ipset v6.30: Error in line 2: Syntax error: cannot parse 1000::: resolving to IPv4 address failed FATAL: inet6 ipset restore failed: error 1

Error when running update script

When running the update script, I get all the way past the download of the files and IPV4 ipset, then get this message. I am using an Edge Router 4.

Applying IPset (IPv6)
ipset v6.23: Error in line 2: Syntax error: cannot parse 1000::: resolving to IPv4 address failed
FATAL: inet6 ipset restore failed: error 1

When using sudo /config/scripts/updBlackList.sh

Hi!

When i try to run the update script, I get this error - and I cant figure out why.

lundaa@LundaaDK-ER6P# sudo /config/scripts/updBlackList.sh

/config/scripts/updBlackList.sh: line 606: syntax error near unexpected token `then'

/config/scripts/updBlackList.sh: line 606: `trap atExit EXIT startup doFetch if [[ -n "${fwGroupNets4}" ]]; then'

[edit]

What to do ?

Firewall Policies configuration

Hi,

Do we need to actually specify which "Interfaces" for this Firewall Blocklist filtering method to apply?

Thanks for your help,
JR

Issue Parsing Talos Feed

Not sure if anyone still maintains this, but there appears to be an issue pulling the Cisco Talos feed (https://www.talosintelligence.com/documents/ip-blacklist). My guess is the fact that the URL redirects to an Amazon S3 bucket is the issue.

Suggesting additional ban list sources

http://blocklist.greensnow.co/greensnow.txt
http://charles.the-haleys.org/ssh_dico_attack_hdeny_format.php/hostsdeny.txt
http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt
http://talosintel.com/feeds/ip-filter.blf
http://www.binarydefense.com/banlist.txt
https://www.badips.com/get/list/any/2?age=30d

Update README to new iprange version

Thanks again for this great solution. Works like a charm.

You might want to update the README to the "new" (2018) 1.0.4 version of iprange.
Version 1.0.3 is not in the pool anymore.

curl -O http://http.us.debian.org/debian/pool/main/i/iprange/iprange_1.0.4+ds-2_mips.deb
curl -O http://http.us.debian.org/debian/pool/main/i/iprange/iprange_1.0.4+ds-2_mipsel.deb

I am using 1.0.4+ds-2_mipsel on an ERX SFP - nothing broken.

Cheers

Informational: thanks!

This is great work, and hugely helpful. Thank you for making this available to the community!

IP Range won't install - ER-X

I have an ER-X and following the install options for IPRange I had an error:

package architecture (mips) does not match system (mipsel)
Errors were encountered while processing:
 iprange_1.0.3+ds-1_mips.deb

Checked out that repository and there was a mipsel out there so I downloaded that:
curl -O http://http.us.debian.org/debian/pool/main/i/iprange/iprange_1.0.3+ds-1_mipsel.deb

Just wondering if your script will use this version and wanted you to be aware of it.

Error when running this on a Unifi Gateway

Previously the script worked well, however now I get the following error:

updBlackList.sh: line 322: syntax error near unexpected token `<'

updBlackList.sh: line 322: ` done < <(sed -e 's/#.$//g' -e '/^[[:space:]]$/d' ${fpUrlList})'

Any ideas?

Can't Parse Malc0de Blacklist

http://malc0de.com/bl/IP_Blacklist.txt

For some reason, adding the blacklist above inflates the generated IP set so that it cannot be created even when MaxElem is set to something ridiculous like a million entries or so.