cokebar/gfwlist2dnsmasq

A shell script which convert gfwlist into dnsmasq rules. Python version: https://github.com/cokebar/gfwlist2dnsmasq_python

gfwlist2dnsmasq's Issues

%2F not recognized as "/" correctly

For example:

server=/bbc.com/127.0.0.1#5353
server=/bbc.com%2Fzhongwen/127.0.0.1#5353
...
server=/bbs.sina.com/127.0.0.1#5353
server=/bbs.sina.com%2F/127.0.0.1#5353

The "%2F" and characters after should be removed.

Use short option for base64 for compatibility of Busybox base64

if [ $SYS_KERNEL = "Darwin"  -o $SYS_KERNEL = "FreeBSD" ]; then
    BASE64_ERES='base64 -D'
else
    BASE64_ERES='base64 -d'
fi

This way the script can be embedded in OpenWrt/Lede for updating GFWList regularly.

3 warnings

WARNING:
The following lines in GfwList contain regex, and might be ignored:

459:/^https?://[^\/]+blogspot.(.*)/
537:/^https?://([^\/]+.)google.(ac|ad|ae|al|am|as|at|az|ba|be|bf|bg|bi|bj|bs|bt|by|ca|cat|cd|cf|cg|ch|ci|cl|cm|co.ao|co.bw|co.ck|co.cr|co.id|co.il|co.in|co.jp|co.ke|co.kr|co.ls|co.ma|com|com.af|com.ag|com.ai|com.ar|com.au|com.bd|com.bh|com.bn|com.bo|com.br|com.bz|com.co|com.cu|com.cy|com.do|com.ec|com.eg|com.et|com.fj|com.gh|com.gi|com.gt|com.hk|com.jm|com.kh|com.kw|com.lb|com.ly|com.mm|com.mt|com.mx|com.my|com.na|com.nf|com.ng|com.ni|com.np|com.om|com.pa|com.pe|com.pg|com.ph|com.pk|com.pr|com.py|com.qa|com.sa|com.sb|com.sg|com.sl|com.sv|com.tj|com.tr|com.tw|com.ua|com.uy|com.vc|com.vn|co.mz|co.nz|co.th|co.tz|co.ug|co.uk|co.uz|co.ve|co.vi|co.za|co.zm|co.zw|cv|cz|de|dj|dk|dm|dz|ee|es|fi|fm|fr|ga|ge|gg|gl|gm|gp|gr|gy|hk|hn|hr|ht|hu|ie|im|iq|is|it|je|jo|kg|ki|kz|la|li|lk|lt|lu|lv|md|me|mg|mk|ml|mn|ms|mu|mv|mw|mx|ne|nl|no|nr|nu|org|pl|pn|ps|pt|ro|rs|ru|rw|sc|se|sh|si|sk|sm|sn|so|sr|st|td|tg|tk|tl|tm|tn|to|tt|us|vg|vn|vu|ws)/./
1917:/twimg.edgesuite.net//?appledaily/

support the new gfwlist-plus?

gfwlist stopped updating, therefore a new gfwlist-plus project appears. do you hope to support it?

Doesn't work on entware-arm@Asuswrt-merlin

gfwlist2dnsmasq.sh -f /jffs/configs/dnsmasq.d/dnsmasq_gfw.conf

/jffs/scripts/gfwlist2dnsmasq.sh: line 50: getopts: not found
Please enter full path to the file.( Use: -f /path/to/output_filename)

When EXCLUDE_DOMAIN_FILE has no content all gfwlists are excluded

I'm managing multiple systems so they all make use of --exclude-domain-file whitelist.conf option. I was setting up a new system so I just touch whitelist.conf instead of putting any value there. This excluded all domains from the output file.

google owned domains are ignored

Met the following warnings when run on openwrt 18.06.4:

WARNING:
The following lines in GfwList contain regex, and might be ignored:

1256:/^https?:\/\/[^\/]+blogspot\.(.*)/
1352:/^https?:\/\/([^\/]+\.)*google\.(ac|ad|ae|af|al|am|as|at|az|ba|be|bf|bg|bi|bj|bs|bt|by|ca|cat|cd|cf|cg|ch|ci|cl|cm|co.ao|co.bw|co.ck|co.cr|co.id|co.il|co.in|co.jp|co.ke|co.kr|co.ls|co.ma|com|com.af|com.ag|com.ai|com.ar|com.au|com.bd|com.bh|com.bn|com.bo|com.br|com.bz|com.co|com.cu|com.cy|com.do|com.ec|com.eg|com.et|com.fj|com.gh|com.gi|com.gt|com.hk|com.jm|com.kh|com.kw|com.lb|com.ly|com.mm|com.mt|com.mx|com.my|com.na|com.nf|com.ng|com.ni|com.np|com.om|com.pa|com.pe|com.pg|com.ph|com.pk|com.pr|com.py|com.qa|com.sa|com.sb|com.sg|com.sl|com.sv|com.tj|com.tr|com.tw|com.ua|com.uy|com.vc|com.vn|co.mz|co.nz|co.th|co.tz|co.ug|co.uk|co.uz|co.ve|co.vi|co.za|co.zm|co.zw|cv|cz|de|dj|dk|dm|dz|ee|es|eu|fi|fm|fr|ga|ge|gg|gl|gm|gp|gr|gy|hk|hn|hr|ht|hu|ie|im|iq|is|it|it.ao|je|jo|kg|ki|kz|la|li|lk|lt|lu|lv|md|me|mg|mk|ml|mn|ms|mu|mv|mw|mx|ne|nl|no|nr|nu|org|pl|pn|ps|pt|ro|rs|ru|rw|sc|se|sh|si|sk|sm|sn|so|sr|st|td|tg|tk|tl|tm|tn|to|tt|us|vg|vn|vu|ws)\/.*/
3095:/twimg\.edgesuite\.net\/\/?appledaily/

not able to include ipset

sh gfwlist2dnsmasq.sh -d 127.0.0.1 -p 5353 -s gfwlist -o gfwlist.conf
but not able to include ipset
ubuntu x86 with sed, base64, curl ready

This script will try to convert some of the regex rules. But you should know this may not be a equivalent conversion.
If there's regex rules which this script do not deal with, you should add the domain manually to the list.

Fetching Google search domain list... Done

Google search domains... Added.
Blogspot domains... Added.
twimg.edgesuit.net... Added.
gfwlist2dnsmasq.sh: 196: [: 1: unexpected operator
Ipset rules not included.

Converting GfwList to dnsmasq rules... Done.

Cleaning up... Done.

multiple dns

the usage of multiple dns is allowed for each domain, so is it possible to add the feature?

How to obtain Blogspot's country-specific domain (ccTLD) list.

Google search uses ccTLDs, like google.com.hk/google.co.jp. And the whole list can be found here:
https://www.google.com/supported_domains

For blogspot, it has same situation. But I don't know how to obtain the whole domain list.

Anyone can help?

ash: 1: unknown operand

Job Started.

Fetching GfwList... ash: 1: unknown operand
Done.

if [ $USE_WGET != 1 ]; then -> if [ "$USE_WGET" != 1 ]; then

帮助信息有错误，默认生成的端口是5353的，说明里写的是5300

The script should start with #!

The script's first line miss the ! . It should be #!/bin/sh

openwrt_shadowsocks 环境下，telegram客户端 无法联网

openwrt_shadowsocks 环境下，安卓手机版telegram 无法联网，我使用如下教程搞定的，除了telegram客户端（win10 版和安卓版，linux未试）不能联网，其它一切正常。

Rules beginning with "|" in GFWList are not processed

For example:

|http://*.1mobile.tw
|https://ss*.4sqi.net
|http://85st.com

Ability to customize (include/exclude) domains

It will be helpful to be able to add extra domains that are not in the output. Maybe they are in the original gfwlist in a regex form and not transformed in the output dnsmasq config. Or maybe they are blocked or very slow to visit from a specific ISP.

Similarly, it is also helpful to be able to remove domains from the gfwlist.

生成列表的正则表达式有问题？

gfwlist 的原文为：

...
.hkheadline.com*blog
.hkheadline.com/instantnews
...

生成的结果：

...
server=/hkheadline.com/127.0.0.1#5353
server=/hkheadline.comblog/127.0.0.1#5353
...

hkheadline.comblog 这个应该要被过滤掉？

Read Gfwlist from local file / custom URL

Now the script can only fetch gfwlist from github repo ( https://github.com/gfwlist/gfwlist ).
It is planned to add this feature in the comming v0.9.0 version.

SwitchyOmega如何使用？

gfwlist_domain如何被SwitchyOmega使用？

I used ipv6 ,can this script accept ::1?

I used ipv6 ,can this script accept ::1?

代码健壮性问题

gfwlist2dnsmasq/gfwlist2dnsmasq.sh

Line 75 in 00e2200

which sed base64 mktemp >/dev/null

当缺少which 命令时，脚本执行出现异常。

Is it possible to add nftable sets support?

dnsmasq 2.87 would be able to add nftable sets, Is it possible to add nftable sets support?

Doesn't work on BSD based system

./gfwlist2dnsmasq.sh -f ~/Desktop/1.txt
./gfwlist2dnsmasq.sh: line 106: md5sum: command not found
-e Fetching GfwList...base64: invalid option -- d
Usage: base64 [-hvD] [-b num] [-i in_file] [-o out_file]
-h, --help display this message
-D, --decode decodes input
-b, --break break encoded string into num character lines
-i, --input input file (default: "-" for stdin)
-o, --output output file (default: "-" for stdout)
-e Done.

Converting GfwList to dnsmasq rules...
-e
WARNING:
The following lines in GfwList contain regex, and might be ignored.

-e
This script will try to convert some of the regex rules. But you should know this may not be a equivalent conversion.
If there's regex rules which this script do not deal with, you should add the domain manually to the list.

sed: illegal option -- r
usage: sed script [-Ealn] [-i extension] [file ...]
sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...]
sed: illegal option -- r
usage: sed script [-Ealn] [-i extension] [file ...]
sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...]
sed: illegal option -- r
usage: sed script [-Ealn] [-i extension] [file ...]
sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...]
-e Fetching Google search domain list...-e Done

Google search domains... Added.
Blogspot domains... Added.
twimg.edgesuit.net... Added.
Ipset rules not included.
-e
Converting GfwList to dnsmasq rules... Done.

-e Generating dnsmasq configuration file...-e Done.

-e Cleaning up...-e Done.

Finished!

can gfwlist 2 unbound with ipset?

the unbound supports ipset in 21.02.1, can you modify this script to realize gfwlist to unbound with ipset conversion?
thanks.

Read extra/exclude domain file from custom URL

Add domains resolved to Google China servers

Some domains (e.g., dl.google.com) can be resolved to Google China servers when using a Chinese DNS. In most conditions this will yield better page load time and download speed. The list can be found from https://raw.githubusercontent.com/felixonmars/dnsmasq-china-list/master/google.china.conf
The list seems to be maintained quite actively.

Support IPv6 dns server

like 2001:470:20::2

curl -k

line 176, sometimes -k is necessary for downloading

Are you planning to support nftset?

Reference this: https://www.right.com.cn/FORUM/thread-8313005-1-1.html

Using Travis CI to auto-generate configuration files.