m-parashar/adbhostgen

Script to generate massive block lists for DD-WRT

Ad-blocking on DD-WRT

deprecated; use adblock.sh

Script to generate a MEGA hosts file for DD-WRT

https://github.com/m-parashar/adbhostgen Copyright 2018 Manish Parashar

Developed and tested on: NETGEAR R8500 / DD-WRT v3.0-r35030M kongac Hardware requirements: Minimum 4GB USB drive. Partitioned, formatted, and mounted as /jffs (1-2GB), swap (256-512MB), and optionally /opt (1-2GB)

NOTE: DO NOT try to run this script on your internal jffs.

Software requirements: DD-WRT (preferably latest), cURL

DOWNLOAD: https://github.com/m-parashar/adbhostgen/releases

Installation:

1a. Download installer.sh and make it executable (chmod +x), then run it in /tmp or /jffs. It will automatically create /jffs/dnsmasq and extract required files into it. OR

1b. Download and extract adbhostgen.7z or adbhostgen.tar.gz into /jffs/dnsmasq directory.

2. Run adbhostgen.sh in /jffs/dnsmasq

3. Use --update or -u command line option to update to the latest version. [only available since 20180315]

Settings:

1. Disable internal flash (JFFS2) if you already have a USB drive mounted as /jffs.

2. Enable DNSMasq and local DNS for LAN and WAN. Add these lines under the additional options section.

   conf-file=/jffs/dnsmasq/mpdomains
   addn-hosts=/jffs/dnsmasq/mphosts
   

3. Enter additional options for dnsmasq if required, for example:

   domain-needed bogus-priv

4. Under Administration -> Cron, enter this or choose your own schedule:

   0 6 * * 1,4 root /jffs/dnsmasq/adbhostgen.sh
   

5. Reboot

Usage:

Add your blacklist or whitelist domains to myblacklist and mywhitelist files respectively. Do not save anything in blacklist or whitelist files as they will be overwritten every time the script is executed.

FOR DETAILS AND SCREENSHOTS: https://www.dd-wrt.com/phpBB2/viewtopic.php?t=307533

--

Should work on R7000, R8000 and other >128MB RAM routers too. Any feedback is welcome.

mode	size	number of domains blocked
BLITZ=0 mphosts	2.7 MB	~103743
BLITZ=1 mphosts	9.9 MB	~358430
BLITZ=2 mphosts	25.6 MB	~866434
BLITZ=3 mphosts	31.9 MB	~1159037

mode	processing time (minutes)
BLITZ=0	1:40
BLITZ=1	3:50
BLITZ=2	7:00
BLITZ=3	10:14

Status:

adbhostgen's Issues

Delete

Delete

Adding list of domains to myblacklist/mywhitelist

Hi,

Thank you for creating a great utility!!! 👍 🥇

I am curious to know if is it a way to add list of domains (talking about 1k+) to the myblacklist/mywhitelist. What I am trying to do is to add domain list of YouTube ads to myblacklist which is around 8k.

To do this manually for each domain will take ages, So is there any way by which we can push the list directly, From what I understood it is not advisable to manually edit the file.

YouTube Ads Domain List.txt

Thank you so much in advance.

Segmentation fault during setup

VERSION: 20180331
on Netgear R6400 running Linux DD-WRT 4.4.159 #4001 SMP Wed Oct 10 09:28:16 CEST 2018 armv7l DD-WRT (build 37305)

Size of /tmp/mphosts.tmp: 9.9M
Size of /tmp/mpdomains.tmp: 3.6M

Processing blacklist/whitelist files
Processing final mphosts/mpdomains files
Segmentation fault
Removing temporary files
Size of /jffs/dnsmasq/mphosts: 0
Size of /jffs/dnsmasq/mpdomains: 3.1M
Number of ad domains blocked: approx 0

result of free
total used free shared buffers cached
Mem: 254576 51496 203080 0 6292 22312
-/+ buffers/cache: 22892 231684
Swap: 487420 0 487420

For testing purposes, I used -0 to see if a smaller file would work, but, same problem.

I've narrowed it down to this line:
LC_ALL=C cat $tmphosts | sed -r 's/^[[:blank:]]//; s/[[:blank:]]$//; /^$/d; /^\s*$/d' | tr -cd '\000-\177' | cat tmpbl - | grep -Fvwf tmpwl | sort -u | awk -v "IP=$ADHOLEIP" '{sub(/\r$/,""); print IP" "$0}' > $mphosts

if I replace $tmphosts with $tmpdomains it completes, successfully, so, something is going on with the $tmphosts file... I think I'm going to add some code to split the file up, into smaller files, and, see if it can process them, and then concatenate the files back into one file, when complete, or, maybe, narrow it down to some error with one (or, more) of the hosts in the file...

OK, so, I added some clunky code to break the mphosts.tmp file into smaller files, process each one, and add them to the final mphosts file, and, this is what happened, first, I broke it into files with 100000 lines in each, and, a couple of them, segfaulted, so, i dropped it to 50000 lines, each, and, they finished up, without segfaulting, and this seems to have worked, although, I have not yet verified that everything works as expected, yet.

OK, I cleaned up my code, to the best of my abilities, and, it ran it several times, and, it seems to be working fine, I haven't tried changing the number of lines above 50000 to see where the actual limit is, although, from my previous tests, it seems, more than 2 MB worth of lines was about where the segfaults were occurring. If anyone is interested in my code, and, would like to help me make sure it's good enough to make a pull request, just let me know, here.

Delete

Delete