mrrfv/cloudflare-gateway-pihole-scripts

Use Cloudflare Gateway DNS/VPN to block ads, malware and tracking domains - free alternative to NextDNS, Pi-hole and Adguard

Cloudflare Gateway Pi-hole Scripts (CGPS)

Cloudflare Gateway allows you to create custom rules to filter HTTP, DNS, and network traffic based on your firewall policies. This is a collection of scripts that can be used to get a similar experience as if you were using Pi-hole, but with Cloudflare Gateway - so no servers to maintain or need to buy a Raspberry Pi!

About the individual scripts

• cf_list_delete.js - Deletes all lists created by CGPS from Cloudflare Gateway. This is useful for subsequent runs.
• cf_list_create.js - Takes a blocklist.txt file containing domains and creates lists in Cloudflare Gateway
• cf_gateway_rule_create.js - Creates a Cloudflare Gateway rule to block all traffic if it matches the lists created by CGPS.
• cf_gateway_rule_delete.js - Deletes the Cloudflare Gateway rule created by CGPS. Useful for subsequent runs.
• download_lists.js - Initiates blocklist and whitelist download.

Features

• Support for basic hosts files
• Full support for domain lists
• Automatically cleans up filter lists: removes duplicates, invalid domains, comments and more
• Works fully unattended
• Allowlist support, allowing you to prevent false positives and breakage by forcing trusted domains to always be unblocked.
• Experimental SNI-based filtering support that works independently of DNS settings, preventing unauthorized or malicious DNS changes from bypassing the filter.
• Optional health check: Sends a ping request ensuring continuous monitoring and alerting for the workflow execution, or messages a Discord webhook with progress.

Usage

Prerequisites

1. Node.js installed on your machine
2. Cloudflare Zero Trust account - the Free plan is enough. Use the Cloudflare documentation for details.
3. Cloudflare email, API token with Zero Trust read and edit permissions, and account ID. See here for more information about how to create the token.
4. A file containing the domains you want to block - max 300,000 domains for the free plan - in the working directory named blocklist.txt. Mullvad provides awesome DNS blocklists that work well with this project. A script that downloads recommended blocklists, download_lists.js, is included.
5. Optional: You can whitelist domains by putting them in a file allowlist.txt. You can also use the get_recomended_whitelist.sh Bash script to get the recommended whitelists.
6. Optional: A Discord (or similar) webhook URL to send notifications to.

Running locally

1. Clone this repository.
2. Run npm install to install dependencies.
3. Copy .env.example to .env and fill in the values.
4. If this is a subsequent run, execute node cf_gateway_rule_delete.js and node cf_list_delete.js (in order) to delete old data.
5. If you haven't downloaded any filters yourself, run the node download_lists.js command to download recommended filter lists (about 250 000 domains).
6. Run node cf_list_create.js to create the lists in Cloudflare Gateway. This will take a while.
7. Run node cf_gateway_rule_create.js to create the firewall rule in Cloudflare Gateway.
8. Profit!

Running in GitHub Actions

These scripts can be run using GitHub Actions so your filters will be automatically updated and pushed to Cloudflare Gateway. This is useful if you are using a frequently updated malware blocklist.

Please note that the GitHub Action downloads the recommended blocklists and whitelist by default. You can change this behavior by setting Actions variables.

1. Create a new empty, private repository. Forking or public repositories are discouraged, but supported - although the script never leaks your API keys and GitHub Actions secrets are automatically redacted from the logs, it's better to be safe than sorry.
2. Create the following GitHub Actions secrets in your repository settings:
   • CLOUDFLARE_API_TOKEN: Your Cloudflare API Token with Zero Trust read and edit permissions
   • CLOUDFLARE_ACCOUNT_ID: Your Cloudflare account ID
   • CLOUDFLARE_LIST_ITEM_LIMIT: The maximum number of blocked domains allowed for your Cloudflare Zero Trust plan. Default to 300,000. Optional if you are using the free plan.
   • PING_URL: /Optional/ The HTTP(S) URL to ping (using curl) after the GitHub Action has successfully updated your filters. Useful for monitoring.
   • DISCORD_WEBHOOK_URL: /Optional/ The Discord (or similar) webhook URL to send notifications to. Good for monitoring as well.
3. Create the following GitHub Actions variables in your repository settings if you desire:
   • FAST_MODE: Enable the scripts to send the requests simultaneously. Beware that there's a rate limit of 1200 requests per five minutes (https://developers.cloudflare.com/fundamentals/api/reference/limits/) so make sure you know what you are doing.
   • ALLOWLIST_URLS: Uses your own allowlists. One URL per line. Recommended allowlists will be used if this variable is not provided.
   • BLOCKLIST_URLS: Uses your own blocklists. One URL per line. Recommended blocklists will be used if this variable is not provided.
   • BLOCK_PAGE_ENABLED: Enable showing block page if host is blocked.
4. Create a new file in the repository named .github/workflows/main.yml with the contents of auto_update_github_action.yml found in this repository. The default settings will update your filters every week at 3 AM UTC. You can change this by editing the schedule property.
5. Enable GitHub Actions in your repository settings.

DNS setup for Cloudflare Gateway

1. Go to your Cloudflare Zero Trust dashboard, and navigate to Gateway -> DNS Locations.
2. Click on the default location or create one if it doesn't exist.
3. Configure your router or device based on the provided DNS addresses.

Alternatively, you can install the Cloudflare WARP client and log in to Zero Trust. This method proxies your traffic over Cloudflare servers, meaning it works similarly to a commercial VPN. You need to do this if you want to use the SNI-based filtering feature, as it requires Cloudflare to inspect your raw traffic (HTTPS remains encrypted if "TLS decryption" is disabled).

Dry runs

To see if e.g. your filter lists are valid without actually changing anything in your Cloudflare account, you can set the DRY_RUN environment variable to 1, either in .env or the regular way. This will only print info such as the lists that would be created or the amount of duplicate domains to the console.

Warning: This currently only works for cf_list_create.js.

Why not...

Pi-hole or Adguard Home?

• Complex setup to get it working outside your home
• Requires a Raspberry Pi

NextDNS?

• DNS filtering is disabled after 300,000 queries per month on the free plan

Cloudflare Gateway?

• Requires a valid payment card or PayPal account
• Limit of 300k domains on the free plan

a hosts file?

• Potential performance issues, especially on Windows
• No filter updates
• Doesn't work for your mobile device
• No statistics on how many domains you've blocked

License

MIT License. See LICENSE for more information.

cloudflare-gateway-pihole-scripts's Issues

gateway locations

Hi,
I need advice how to connect my mobile devices to cf gateway.
Should we create new dns location for evry device to get the unique dot/doh adress ?
Are there resctrictions on the zero trust free how many device/location we can have?

[Feature Request] Discord (or similar) webhook notifications

CGPS is more of a "set-and-forget" script, meaning that if something happens to go wrong, it usually isn't noticed until it's too late and ads start showing up. It'd be nice to have a feature allowing the use of Discord (as well as Matrix, maybe with the use of Matrix-Webhooks) for error notifications and such.

Regarding 300000 free limit

I see that you wrote that there's a limit of 300000 blocked domains on the free account. I can't any info regarding this on cloudflare.

Also, according to this documentation https://developers.cloudflare.com/cloudflare-one/account-limits/#gateway there's a limit of 100 on Lists. And each list can only contain 1000 items. So shouldn't there be a limit of 100000 on free accounts?

Although, i tried your script via github on my cloudflare account with OISD list. It created around 187 lists without any error. I am worried if my cloudflare account will get banned for bypassing 100 limit on Lists. Is this correct?

[Feature Request] SNI-based blocking

An optional feature that creates a network firewall policy that denies access to blocked domains based on their SNI, making it near impossible to bypass the filter by apps using hardcoded DNS settings.

Getting 404 at CF Steps

Hope you can help me find what I'm doing wrong.
I forked the repo, setup my secrets and variables. I'm getting a 404 error. This looks like a CF/Credential error, but I cannot figure it out :( I run a curl test to verify the token is valid. "This API Token is valid and active"

Pics of my repo action secrets:

Pic of Variables:

Log at Delete old rule and list (If I comment this step out, fails at Create).
Run npm run cloudflare-delete

cloudflare-delete
npm run cloudflare-delete:rule && npm run cloudflare-delete:list

cloudflare-delete:rule
node cf_gateway_rule_delete.js

file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/lib/helpers.js:106
throw new Error(HTTP error! Status: ${response.status});
^

Error: HTTP error! Status: 404
at request (file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/lib/helpers.js:106:11)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/cf_gateway_rule_delete.js:4:27

Node.js v20.9.0
Error: Process completed with exit code 1.

"response.body?.pipeTo is not a function" while downloading filter lists using Node.js v16

@hlqviet Tagging you because this might be related to #25 (though I'll try to fix this on my own in my free time).

This is most likely caused by the old Node.js version, that we have to support because that's what the original GitHub Actions workflow uses.

Environment

  node: v16.20.2
  npm: 8.19.4
  yarn: 1.22.19

Error

> Run bash ./get_recommended_whitelist.sh
file:///home/runner/work/***/***/cloudflare-gateway-pihole-scripts/lib/utils.js:69
    await response.body?.pipeTo(writableStream, { preventClose: true });
                         ^

TypeError: response.body?.pipeTo is not a function
    at downloadFiles (file:///home/runner/work/***/***/cloudflare-gateway-pihole-scripts/lib/utils.js:69:26)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async downloadLists (file:///home/runner/work/***/***/cloudflare-gateway-pihole-scripts/download_lists.js:18:3)
    at async file:///home/runner/work/***/***/cloudflare-gateway-pihole-scripts/download_lists.js:32:5
Error: Process completed with exit code 1.

[Feature Request] Docker support

A Docker image with cron built-in that works in a similar fashion as the GitHub Action, with the security benefit of being able to run it on your own hardware.

Se7envn1981

Error: HTTP error! Status: 502

Hi,

I just synced with the latest,and now getting these errors.

Could not create "CGPS List - Chunk 140" - Error: HTTP error! Status: 502
file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/lib/helpers.js:106
throw new Error(HTTP error! Status: ${response.status});
^

Error: HTTP error! Status: 502
at request (file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/lib/helpers.js:106:11)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async createZeroTrustListsOneByOne (file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/lib/api.js:48:7)
at async file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/cf_list_create.js:140:5

Node.js v20.9.0
Error: Process completed with exit code 1.

tutorial for github

Hi,
If i understand good,this script can be installed on github?

If yes, can you please share the step by step procedure to install your script on my github account?
And also how to link cloudlfare gateway with github for the blocklist.

Error: Process completed with exit code 1.

Error: HTTP error! Status: 400
at request (file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/lib/helpers.js:106:11)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/cf_gateway_rule_delete.js:4:27

Node.js v20.11.0
Error: Process completed with exit code 1.

clean up oisd li

Hello,

Thank you for your work. I am trying to use your scripts with oisd list.
Is there a good way to clean up the oisd big list? https://big.oisd.nl/

It should be done in cf_list_create.js but not sure how.

Latest code merge

Hi @mrrfv, I just noticed that the recent changes haven't merged into v1 😅

google.com blocked

I see that google.com is blocked now, but we cant see which blocklist is doing this.
Can you check please and fix it.

Create or delete lists script error

Hello,

I I run create lists script (and also delete lists script) I run into this error:

Creating 169 lists for 168793 domains...
Could not create "CGPS List - Chunk 1" - Error: HTTP error! Status: 400
file:///srv/cloudflare-gateway-pihole-scripts/lib/helpers.js:106
    throw new Error(`HTTP error! Status: ${response.status}`);
          ^

Error: HTTP error! Status: 400
    at request (file:///srv/cloudflare-gateway-pihole-scripts/lib/helpers.js:106:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async createZeroTrustListsOneByOne (file:///srv/cloudflare-gateway-pihole-scripts/lib/api.js:48:7)
    at async file:///srv/cloudflare-gateway-pihole-scripts/cf_list_create.js:140:5

The error is the same as above for the delete lists script.

while joining a server in CS2 server cuts the connection

while joining a server in CS2 everyone doesnt move, can only hear weapon sounds, disabling zero trust fixes the problem

[OFF-TOPIC] [Need Help] How to use it in router which only supports IPv4 DNS?

I am using Xiaomi Router 3c & Router 4c, both of which only support IPv4 DNS. Cause when I entered the IPv6 one, it gave an error saying the DNS needs to of a value 255.255.255.255 max. The script is working perfectly fine. This is first time I used node.js, npm, etc. but surprisingly managed to do it in one go. Tried the output in two of my android devices and chrome browser using the dns over tls / https DNS and all are working fine.

I just couldn't manage to make it work in my router and windows 10 PC.

My ISP is using PPPoE connection type, which assigns random IP address to my network.

If someone can guide me how can I use this DNS with my router I'll be very grateful 🙏

update issues when forked

Hi,
These days i see there are some great new updates, however with forked version we cant sync to the latest automatically anymore.
Please solve this issue,so when can sync the the latest version with no problem.

whitelist

Hi,

the whitelist you added,is it automatically added when updating workflow?
and can we select which whitelist we want to be used?
I would prefer the hagezi's whitelist.

It doesnt block twitch ads

It doesnt block twitch ads, can you add ads from twitch to this script?

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

github-actions

.github/workflows/dry-run.yml

• actions/checkout v4
• actions/setup-node v4

.github/workflows/sync-branch.yml

• actions/checkout v4
• connor-baer/action-sync-branch v1.1.0

auto_update_github_action.yml

• actions/checkout v4
• actions/setup-node v4
• actions/checkout v4
• gautamkrishnar/keepalive-workflow v2

nodenv

.node-version

• node 20.12.2

npm

package.json

• dotenv 16.4.5
• node-fetch 3.3.2
• node 20.12.2

---

• Check this box to trigger a request for Renovate to run again on this repository

HTTP error! Status: 504

> cloudflare-delete:list
> node cf_list_delete.js

file:///home/runner/work/cloudflare-gateway-adblock/cloudflare-gateway-adblock/lib/helpers.js:40
    throw new Error(`HTTP error! Status: ${response.status}`);
          ^

Error: HTTP error! Status: 504
    at request (file:///home/runner/work/cloudflare-gateway-adblock/cloudflare-gateway-adblock/lib/helpers.js:40:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async file:///home/runner/work/cloudflare-gateway-adblock/cloudflare-gateway-adblock/cf_list_delete.js:9:29

Hi, my github action run failed and show up this HTTP error message, it works great before.

Is there any fixable problem? What info should I provide for maintainers?

cgps Process completed with exit code 1.

Hi,

I have the latest version forked, and synced yesterday the new update.
Now when i want to update my filter list, i get this error.

at new URL (node:internal/url:637:13)
at new Request (node:internal/deps/undici/undici:7132:25)
at fetch2 (node:internal/deps/undici/undici:10715:25)
at Object.fetch (node:internal/deps/undici/undici:11574:18)
at fetch (node:internal/process/pre_execution:242:25)
at file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/lib/utils.js:60:57
at Array.map ()
at downloadFiles (file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/lib/utils.js:60:44)
at downloadLists (file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/download_lists.js:18:9) {
input: '',
code: 'ERR_INVALID_URL'
}
}

Node.js v18.18.0
Error: Process completed with exit code 1.

block lists

Hi,
I see mullavad already added adguard dns list.
Also cgps script adds adguard dns, so its double now.

I hope you can add some of hagezi's list.

hagezi blocklist

Hi,

Can you please add hagezi Pro blocklist to the cgps script.
It is very well maintained block list.

Please make this easier to understand

Hello, a youtube video with every step would be a godsend with this - as ive never used node.js and have no idea wth im doing.
Would grately appreciate it.