collinbarrett/filterlists

:shield: The independent, comprehensive directory of filter and host lists for advertisements, trackers, malware, and annoyances.

FilterLists is the independent, comprehensive directory of filter and host lists for advertisements, trackers, malware, and annoyances.

Table of Contents

• Background
• Services
• Contributing
  • Adding or Updating Lists
  • Adding or Updating Rules
  • Building and Running Locally
• Acknowledgements
• Miscellany
  • Disclaimer
  • Privacy

Background

FilterLists helps to protect your privacy and security when using the internet. It provides a comprehensive directory of subscription lists to block advertisements, malware, trackers, and other general annoyances. First, install one of the software tools that supports these lists. Then you can subscribe to a set of these lists inside of that software. If you need help or have a comment/question, start a Discussion.

Using a software firewall or adblocker has become a key tool in protecting one's privacy online. While the current business model of the "free" internet relies upon advertising revenue, publishers have become too flippant about tracking users without their consent. Advertising networks track users' browsing habits around the internet building lucrative profiles to exploit for targeted marketing or big data research. Until internet publishers develop more transparency or a new business model, we must continue to take the protection of our privacy into our hands.

Typical adblockers run as an extension in popular web browsers. As we browse the internet, they compare HTTP requests to their list of hosts and filters to selectively block advertisements, trackers, and malware. This filtering helps to protect the surfer's privacy, prevents malvertising attacks, and reduces bandwidth requirements. Thousands of contributors work hard to maintain these FilterLists for anyone to use. Locating and selecting these lists, however, has not historically been very easy. This project was launched in 2015 to solve that problem.

Services

Service	Description	More
Website	A React & TypeScript UI built with Ant Design.
Directory API	An ASP.NET Core API serving the core FilterList information.
Archival API	An ASP.NET Core API archiving and serving copies of FilterLists for mirrors and analysis.
Reverse Proxy	An NGINX instance forwarding requests to the respective services above.

Contributing

Adding or Updating Lists

To submit a new list or update data about an existing list, please submit a pull request to data in conjunction with the data model described here. Once you open the pull request, our Migrate bot will create an Entity Framework Core migration with your changes and test them. Please ensure to grant the bot permissions to push to your pull request branch. You can find more details on this process here.

Alternatively, you can open a new issue providing information for all of the fields described in the data model.

In the future, we hope to have an easier form UI to submit updates. Follow this issue for any progress on that feature.

Note: Since the primary FilterLists.com site does not support Internet Explorer, we also have a separate page for tracking protection lists (TPL) here and viewed here.

Adding or Updating Rules

FilterLists does not maintain any of these lists. It serves only as a discovery tool to direct users to lists that they may want to use. If you want to request addition, modification, or removal of a rule from a list, you will need to contact the maintainers of that list directly. FilterLists provides a variety of ways you can get in contact with the maintainers to do so.

Building and Running Locally

We have containerized FilterLists to make it as easy as possible for contributers to get the project up and running locally.

1. Install Docker CE. Docs
2. Install the current version of Node.js. Docs
3. Clone the FilterLists git repository to your computer. Docs
4. Navigate to the root directory of your locally cloned FilterLists git repository in a command-line interface.
5. Start the APIs:
   docker-compose -f docker-compose/docker-compose.yml -f docker-compose/docker-compose.override.yml up -d
   You can then view the API docs and execute API calls here: http://localhost:8080/api/
6. Start the Web app:
   npm i --cwd web && npm start --prefix web
   You can then view the Web app calling your local instance of the Directory API here: http://localhost:3000

Acknowledgements

• Imre Kristoffer Eilertsen @DandelionSprout : major contributor
• Raymond Hill @gorhill : uBlock Origin owner, advised on some technical aspects of FilterLists
• Andrey Meshkov @ameshkov : AdGuard CTO, advised on some technical aspects of FilterLists
• More contributors

Miscellany

Disclaimer

FilterLists does not condone subscribing to any particular FilterList or blocking any particular advertisement network. Make sure to financially support your favorite sites in some way if you choose to block their advertisements. FilterLists is also in no way officially affiliated with any of the software vendors or list maintainers referenced on this site.

Privacy

We respect your privacy. That is the whole point of this project. The only minor data we collect about visitors is what our infrastructure providers collect. These include:

• GitHub
• Azure DevOps
• Application Insights
• DigitalOcean
• Cloudflare

filterlists's Issues

Add Mute

Add to Global:
http://mute.bradconte.com/

frequency of list link clicks

Use the frequency of clicks of the View/Add links (unique per visitor per day) from FilterLists to calculate the FilScore.

This data is collected from our anonymized, self-hosted Piwik installation.

Automation of moving lists to Stale

Currently, lists are included in the Stale list type if they have not been updated in the past year. This is a manual process, and it will be hard to keep updated. We should eliminate the Stale list type .json file (moving all of its current lists to their most appropriate other list type) and instead display lists on the Stale page by an automated check of their last updated date.

Requires #6

Add Adblock-Persian list

Add to Regional:
.txt
.java

age

Lists that have been around (and active) for longer should receive a boost in FilScore.

In conjunction with #44 .

Weight: Medium

Add logo and favicon

Add logo and favicon

add notifications for urls with HTTP response other than 200 OK

Add email notifications to investigate any URLs in FilterLists that return an HTTP response other than 200 OK.

Change Description font to differentiate between quoted and non-quoted

Many of the list descriptions are blockquotes from copy on the list's homepage. Lists that do not provide a concise description, however, are written by one of the maintainers of FilterLists. There should be a difference in font or some annotation that differentiates the two types of descriptions.

Filter regions by checkboxes

Filter regions by checkboxes rather than selecting individual regions and region combinations. This will allow lists that target more than one region to be included in the filters of any of the regions selected rather than its own combination region filter selection. For an immediate fix, this will be dependent on the ability to customize the wpdatatables plugin.

support comparing lists

Allow users to compare two or more lists for coverage overlap. Think browser-based WinMerge for FilterLists.

community vibrancy

Lists who have more activity in GitHub Issues, GitHub stars, the list's forum, etc. should receive a boost in FilScore.

Weight: Low

support creating custom combination lists

In the same vein as Steven Black's project (but for filter lists as well as hosts lists), it could be useful for FilterLists to be able to easily merge and de-duplicate (likely server-side) multiple lists into custom combination lists. Depending on list licensing, FilterLists could even automatically maintain these custom lists so that users could subscribe to them for future updates.

Add FED List

Add to Global:
https://firime.wordpress.com/projects/others/fedlist/

Add internal links to list references in Descriptions

Where another list is referenced in the text of a Description, add an internal link to that list on FilterLists.

Add property or perhaps new list type for exception lists

There are some lists that are purely for "exceptions". The most popular of which is probably ABP's Acceptable Ads list. There are others, though, put out by sites such as cashback or shopping sites that whitelist enough tracking for their users to gain rewards.

While most FilterLists visitors will not be interested, these may still be worth indexing and flagging as the type of list that they are.

rule search

Allow users to search for a specific domain or rule across all lists. This would be in the same vein as Adblock Monitor or Adblock-filter search but across all lists indexed on FilterLists.

Requires #6 .

Add adblock-youtube-ads

Add to Global:
https://github.com/kbinani/adblock-youtube-ads

Populate language for former "Combo" lists

Regions for combo lists should be pulled from all of their master lists.

add lists' syntaxes and the software that supports them

Adblock lists are only useful in a browser-based solution. Host lists can be used in a browser-based solution as well as in a host file to block entire domains/IPs machine-wide. FilterLists should index lists based on their syntax to assist users.

Add EasyList Thailand

Add to Regional:
https://github.com/easylist-thailand/easylist-thailand

(discovered via GitHub's new "topics" feature, which is great!)

license

Lists with more permissive licenses should receive a boost in FilScore.

Requires #18 .

Weight: Medium

Remove filter row from Google Featured Snippets

Google Featured Snippets shows the filter row as a duplicate header row. We should figure out a way around that for SEO purposes.

Internal linking of forks, combos, and master lists

Lists in the "Forks & Combos" and "Master" fields should internally link to the list's information on FilterLists. For example, if a Global list has a list in its "Forks & Combos" field, that list name should be linked to the list's anchor on it's list type page.

Implement frequent fetches of all lists

Fetch copies of all lists daily and cache older versions. This would allow determining list update frequency, list change notifications, etc. This project likely should not serve as an official mirror of the lists (at least without verifying list licensing, etc.), but implementing this system would allow for some useful features on FilterLists.

Add a "New" property/badge

Add a "New" property or badge to give lists that have been recently added to FilterLists or recently started a little extra boost.

add Blacklists UT1 lists

Per @indolering,

Add all lists from Blacklist UT1.

Low priority until #201 is resolved.

Add adblock-leadgenerator-list

Add to Global:
https://github.com/Rpsl/adblock-leadgenerator-list

add Issue template

Finish the template for Issues:
https://github.com/collinbarrett/FilterLists/blob/master/.github/ISSUE_TEMPLATE.md

automatic URL validation

All URLs in the Directory should be periodically validated for the following:

• Is valid URI syntax
• Availability (i.e., HTTP 200 vs 404)
• Prefer direct URL if 301/302 redirected
• Prefer HTTPS if available
• Malware (using VirusTotal API or similar)

add property for confirmed software support

Add a property for signaling what software tools each list has been verified to support. For each list, provide a list of labels such as "uBlock Origin", "AdBlock Plus", etc. for software tools that the list has been confirmed to work with.

Verify all lists from Steven Black's Hosts are indexed

Verify all lists from Steven Black's Hosts are indexed.

dynamic "Subscribe" buttons (function/visibility)

Add a flag property for if a list is directly "subscribe-able". If it is a .7z, .zip, etc., this flag will indicate that the "Subscribe" button should not be displayed.

Instead of a property, this could be something that is automated as well. (i.e., if file is a .txt vs a .zip, etc.). That would need extensive testing/debugging.

Feature overlap/similar to #7

Eventually, it could be neat to provide a subscribable mirror of these lists if the maintainer permits it.

Though this could encroach user's privacy, it could be neat to try to detect which (if any) browser extension Software the user has installed. Then, only show the "Subscribe" button for lists that are actually supported by that extension. Feedback wanted on this idea, though, if you feel that would violate user's privacy. If not, does anyone know how this could be achieved?

Updated:
Subscribe buttons should be conditionally visible. By default, they should be visible only if they link to a .txt file (to accommodate most browser-based software tools). We could enable the button for compressed (.zip, .7z) or other formats per this if needed after #370 is implemented.

Updated 2:

• We should hide subscribe buttons if the viewUrl is not plain-text (e.g. .7z, .zip).
• We should hide subscribe buttons if the software selected in the software filter select does not support abp:/ubo:-style subscribe links. (isAbpSubscribable)
• We should change "Subscribe" to "Import" if the software selected in the software filter select is AdGuard.

cuz AdGuard doesn't have real subscribe, it'll only import lists to user filters without possible updates. -via

• We should change abp: subscribe links to ubo: subscribe links if the list is only supported by uBO/Nano. -via
• We should support IE/TPL subscribe links. -#289
• We should support Little Snitch subscribe links. -#507
• See if there is a subscribe link format that the Opera browser can support. -via

Add entity for list license

Add a json property for each list (when available) denoting it's license

Add uBO-Personal-Filters

Add to Global:
https://github.com/uBlock-user/uBO-Personal-Filters

frequency

Lists who are updated (rules added and removed) more frequently should receive a boost in FilScore.

Requires #6 .

Weight: Medium

uniqueness / furthest upstream

Lists that are more unique should receive a boost in FilScore. Preference should be given to upstream lists in merges and forks (with the caveat that a fork whose upstream list is discontinued should be scored greater).

Requires #9

Weight: High

user rating

Collect user ratings (upvote/downvote, five-star, etc.) so that higher rated lists should receive a boost in FilScore.

As much as is feasible, we would want to prevent the same user from voting many times for the same list. But, the system should be friction-less and private (no accounts/logins).

Weight: Medium

Increase size of tap targets on mobile

The small + icons to expand list info on mobile are too small. The entire list row should be made "tap-able" to expand the list info.

Launch internal links to other lists in modal

It would be neat to launch modal popups when clicking an internal link to another list. For example, when looking at the Combo lists, clicking on the name of one of the master lists from which it was created would launch a modal with all of the relevant information about that master list.

Add Satterly's ABP Filters

Add to Global:
https://satterly.neocities.org/abp_filters.html

Add prominent link to GitHub in header or footer

Slim resource loading and verify font support

Currently, CSP blocks loading 3rd-party resources included by the default WordPress theme. However, these resources should be removed altogether. Any resources from the first party domain that are unneeded should be removed as well. Also, self-host all fonts including a slimmed-down version of ionicons just containing the needed icons.

Add Nopelist

Add to Global:
https://github.com/genediazjr/nopelist

Alexa or other SEM tool rating

Lists with more backlinks/"buzz" (using Alexa or other similar SEM tool) should receive a boost in FilScore.

Weight: Medium

Add TopCashback Exception List

Add to Global:
https://www.topcashback.com/Help/164

Related to #34

Replace json-schema with sample list object in json-data files

For simplicity, replace the json-schema directory with a sample json list object at the top of each json-data file. These sample objects will not be displayed on filterlists.com, but will make it easy for contributors to quickly see all fields (and their purpose) that can be defined for each list.

GitHub stars

Use the list's repo star count to calculate the FilScore.

This would only be factored in lists maintained on GitHub. This would be loosely correlated to #45 , but far easier to calculate.

Software support

Use the list's software support to calculate the FilScore.

Lists that can be subscribed to in larger numbers of software tools should be rated slightly better.

Requires #11

Add better support for accessing all lists

Either add an "all lists" page or add a search box that searches all lists rather than just the lists on any given list type page.

Add property for list author

It'd be nice to have a property to display each list's author/maintainer. This could be a person's name, a person'a alias, or an organization. The render of the author's name should link to their profile/homepage/twitter/etc if available.