Giter Site home page Giter Site logo

trinib / black-mirror Goto Github PK

View Code? Open in Web Editor NEW

This project forked from t145/black-mirror

0.0 1.0 0.0 131.22 MB

Automatically maintained malicious host blacklists and false-positive whitelists.

License: GNU Affero General Public License v3.0

Shell 79.60% Python 6.37% Dockerfile 11.89% Perl 2.14%

black-mirror's Introduction

logo

🌓 Reflection | 💿 Redundancy | ✅ Reliability

Automatically maintained malicious host blacklists and false-positive whitelists.

humane-tech last_updated visits release release release docker

🛡️ Privacy Protectors

🖋️ Manifesto

Defines the logic behind why a host is permitted or blocked. Please report any hosts that are wrongly blocked or sources that do not wholly align in an issue. Reference the contribution guidelines.

📋 Attributes

  1. Produced in domain-only, IPv4-only, IPv4-CIDR-only, and IPv6-only builds.
  2. Updates at 1:27 AM & PM UTC.
  3. No excess or trailing whitespace.
  4. No lingering webscraper garbage.
  5. Lines are terminated with lf.
  6. No blank lines.
  7. No comments.

🚚 Deliverables

⚓ Hyperlinks

List Name File Content Unique Entries File Size Mirror 1 Mirror 2
black_domain Domain entries 7,316,618 165M [✔️,🔗] [✔️,🔗]
black_ipv4 IPv4 addresses 742,694 11M [✔️,🔗] [✔️,🔗]
black_ipv4_cidr IPv4 CIDR blocks 22,313 373K [✔️,🔗] [✔️,🔗]
black_ipv6 IPv6 addresses 9,984 302K [✔️,🔗] [✔️,🔗]
white_domain Domain entries 21,964 334K [✔️,🔗] [✔️,🔗]
white_ipv4 IPv4 addresses 12,050 166K [✔️,🔗] [✔️,🔗]
white_ipv4_cidr IPv4 CIDR blocks 1,511 24K [✔️,🔗] [✔️,🔗]
white_ipv6 IPv6 addresses 2,788 107K [✔️,🔗] [✔️,🔗]

🧮 Checksum Evaluation

cat black_domain.txt | sha256sum -c black_domain.checksums --status && echo $?

A return code of 0 means the check was successful. The specific checksum command can be any of the following:

  • md5sum
  • b2sum
  • sha1sum
  • sha224sum
  • sha256sum
  • sha384sum
  • sha512sum

🐙 Fetching GitHub Releases

Provided below are some examples to fetch release artifacts leveraging the GitHub API.

Get all build artifacts

curl --proto '=https' --tlsv1.3 -H 'Accept: application/vnd.github.v3+json' -sSf https://api.github.com/repos/T145/black-mirror/releases/latest | jq -r '.assets[].browser_download_url'

Get a build artifact & its checksum

curl --proto '=https' --tlsv1.3 -H 'Accept: application/vnd.github.v3+json' -sSf https://api.github.com/repos/T145/black-mirror/releases/latest | jq -r '.assets[] | select(.name | startswith("black_domain")).browser_download_url'

Get a single build artifact

curl --proto '=https' --tlsv1.3 -H 'Accept: application/vnd.github.v3+json' -sSf https://api.github.com/repos/T145/black-mirror/releases/latest | jq -r '.assets[] | select(.name | startswith("black_domain")) | select(.name | endswith(".txt")).browser_download_url'

🛠️ Usage

Desktop OS Hosts File

mawk '{print "0.0.0.0 " $0}' black_domain.txt >>hosts
# mawk '{print ":: " $0}' black_domain.txt >>hosts
mawk '{print "0.0.0.0 " $0}' black_ipv4.txt >>hosts
mawk '{print ":: " $0}' black_ipv6.txt >>hosts

dnsmasq

Many popular platforms such as OpenWRT, DDWRT, and Pihole use DNSmasq as their choice TCP powerhouse. After inspecting many domain blocklists you'll inevitably run across a list in the dnsmasq.conf format. This list doesn't support it because you can use the addn-hosts parameter to add hosts in the list. Target a file that has the hosts in a format similar to the Desktop OS Hosts File format.

If you're using the RADVD daemon, prepend any listed hosts with ::. Otherwise, even if you have IPv6 support set up, prepend hosts with 0.0.0.0.

This has been tested across all the mentioned platforms using dig{6} on a small sample size and had each host null-routed successfully. DNSmasq's man page discusses configuration further, and DDWRT's ad blocking wiki page provides some examples.

Amazon EC2 DNS Resolver

Follow this guide to create a DNS server on a Amazon EC2 instance.

pihole

If you'd like to update when some sources do or not extract a production build, just use the single-line list sources.pihole. Note that this list only contains Pihole-compatible sources, and not every handled source. Some manual configuration may also be required.

unbound

Similar to dnsmasq, but requires more manual configuration. Name any products as a *.conf file. Then follow Steffinstanly's instructions on how to apply blocklists.

personalDNSfilter

Use the domain list.

📚 Sources

Please report any redundant sources in an issue!

⬛ Blacklists

⬜ Whitelists

Applied to generated blacklists.

🥢 Duplicates

Sources that contain duplicate and potentially deprecated data.

🧟 Zombies

Sources that are dead or deprecated and not included but may be worth mentioning.

🕵🏻 Lamers Unwelcome

🎶 Notes

R Language

Docker installs

RUN apt-get -y install r-base

# install libarchive manually since libarchive-dev is at version 3.4.3
# https://github.com/libarchive/libarchive/wiki/BuildInstructions#using-configure-for-building-from-the-command-line-on-linux-freebsd-solaris-cygwin-aix-interix-mac-os-x-and-other-unix-like-systems
# https://www.zhouchun.net/blog/show/439 (run all commands together to prevent spawning subcontainers)
RUN aria2c https://github.com/libarchive/libarchive/releases/download/v3.5.2/libarchive-3.5.2.tar.gz \
&& tar xzf libarchive-3.5.2.tar.gz \
&& cd libarchive-3.5.2 \
&& ./configure \
&& make \
&& make check \
&& make install \
&& cd .. \
&& rm libarchive-3.5.2.tar.gz

# install R libarchive bindings
# https://github.com/r-lib/archive
RUN echo 'install.packages("archive", repos="https://cloud.r-project.org/")' | R --vanilla \
&& echo 'install.packages("data.table", repos="https://cloud.r-project.org/")' | R --vanilla

Boosting speeds

Need for speed

References

Together we'll make a better internet!

A project by T145 made with 💖

black-mirror's People

Contributors

dependabot[bot] avatar github-actions[bot] avatar snyk-bot avatar t145 avatar trinib avatar

Watchers

avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.