Giter Site home page Giter Site logo

cokebar / gfwlist2dnsmasq Goto Github PK

View Code? Open in Web Editor NEW
887.0 887.0 248.0 605 KB

A shell script which convert gfwlist into dnsmasq rules. Python version: https://github.com/cokebar/gfwlist2dnsmasq_python

License: GNU General Public License v3.0

Shell 100.00%
dnsmasq gfwlist

gfwlist2dnsmasq's People

Contributors

billsq avatar cokebar avatar honwen avatar mitchelljo avatar moetayuko avatar wyf88 avatar yuchting avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gfwlist2dnsmasq's Issues

google owned domains are ignored

Met the following warnings when run on openwrt 18.06.4:

WARNING:
The following lines in GfwList contain regex, and might be ignored:

1256:/^https?:\/\/[^\/]+blogspot\.(.*)/
1352:/^https?:\/\/([^\/]+\.)*google\.(ac|ad|ae|af|al|am|as|at|az|ba|be|bf|bg|bi|bj|bs|bt|by|ca|cat|cd|cf|cg|ch|ci|cl|cm|co.ao|co.bw|co.ck|co.cr|co.id|co.il|co.in|co.jp|co.ke|co.kr|co.ls|co.ma|com|com.af|com.ag|com.ai|com.ar|com.au|com.bd|com.bh|com.bn|com.bo|com.br|com.bz|com.co|com.cu|com.cy|com.do|com.ec|com.eg|com.et|com.fj|com.gh|com.gi|com.gt|com.hk|com.jm|com.kh|com.kw|com.lb|com.ly|com.mm|com.mt|com.mx|com.my|com.na|com.nf|com.ng|com.ni|com.np|com.om|com.pa|com.pe|com.pg|com.ph|com.pk|com.pr|com.py|com.qa|com.sa|com.sb|com.sg|com.sl|com.sv|com.tj|com.tr|com.tw|com.ua|com.uy|com.vc|com.vn|co.mz|co.nz|co.th|co.tz|co.ug|co.uk|co.uz|co.ve|co.vi|co.za|co.zm|co.zw|cv|cz|de|dj|dk|dm|dz|ee|es|eu|fi|fm|fr|ga|ge|gg|gl|gm|gp|gr|gy|hk|hn|hr|ht|hu|ie|im|iq|is|it|it.ao|je|jo|kg|ki|kz|la|li|lk|lt|lu|lv|md|me|mg|mk|ml|mn|ms|mu|mv|mw|mx|ne|nl|no|nr|nu|org|pl|pn|ps|pt|ro|rs|ru|rw|sc|se|sh|si|sk|sm|sn|so|sr|st|td|tg|tk|tl|tm|tn|to|tt|us|vg|vn|vu|ws)\/.*/
3095:/twimg\.edgesuite\.net\/\/?appledaily/

Doesn't work on BSD based system

./gfwlist2dnsmasq.sh -f ~/Desktop/1.txt
./gfwlist2dnsmasq.sh: line 106: md5sum: command not found
-e Fetching GfwList...base64: invalid option -- d
Usage: base64 [-hvD] [-b num] [-i in_file] [-o out_file]
-h, --help display this message
-D, --decode decodes input
-b, --break break encoded string into num character lines
-i, --input input file (default: "-" for stdin)
-o, --output output file (default: "-" for stdout)
-e Done.

Converting GfwList to dnsmasq rules...
-e
WARNING:
The following lines in GfwList contain regex, and might be ignored.

-e
This script will try to convert some of the regex rules. But you should know this may not be a equivalent conversion.
If there's regex rules which this script do not deal with, you should add the domain manually to the list.

sed: illegal option -- r
usage: sed script [-Ealn] [-i extension] [file ...]
sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...]
sed: illegal option -- r
usage: sed script [-Ealn] [-i extension] [file ...]
sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...]
sed: illegal option -- r
usage: sed script [-Ealn] [-i extension] [file ...]
sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...]
-e Fetching Google search domain list...-e Done

Google search domains... Added.
Blogspot domains... Added.
twimg.edgesuit.net... Added.
Ipset rules not included.
-e
Converting GfwList to dnsmasq rules... Done.

-e Generating dnsmasq configuration file...-e Done.

-e Cleaning up...-e Done.

Finished!

Ability to customize (include/exclude) domains

It will be helpful to be able to add extra domains that are not in the output. Maybe they are in the original gfwlist in a regex form and not transformed in the output dnsmasq config. Or maybe they are blocked or very slow to visit from a specific ISP.

Similarly, it is also helpful to be able to remove domains from the gfwlist.

not able to include ipset

sh gfwlist2dnsmasq.sh -d 127.0.0.1 -p 5353 -s gfwlist -o gfwlist.conf
but not able to include ipset
ubuntu x86 with sed, base64, curl ready

This script will try to convert some of the regex rules. But you should know this may not be a equivalent conversion.
If there's regex rules which this script do not deal with, you should add the domain manually to the list.

Fetching Google search domain list... Done

Google search domains... Added.
Blogspot domains... Added.
twimg.edgesuit.net... Added.
gfwlist2dnsmasq.sh: 196: [: 1: unexpected operator
Ipset rules not included.

Converting GfwList to dnsmasq rules... Done.

Cleaning up... Done.

%2F not recognized as "/" correctly

For example:

server=/bbc.com/127.0.0.1#5353
server=/bbc.com%2Fzhongwen/127.0.0.1#5353
...
server=/bbs.sina.com/127.0.0.1#5353
server=/bbs.sina.com%2F/127.0.0.1#5353

The "%2F" and characters after should be removed.

ash: 1: unknown operand

Job Started.

Fetching GfwList... ash: 1: unknown operand
Done.

if [ $USE_WGET != 1 ]; then -> if [ "$USE_WGET" != 1 ]; then

Doesn't work on entware-arm@Asuswrt-merlin

gfwlist2dnsmasq.sh -f /jffs/configs/dnsmasq.d/dnsmasq_gfw.conf

/jffs/scripts/gfwlist2dnsmasq.sh: line 50: getopts: not found
Please enter full path to the file.( Use: -f /path/to/output_filename)

curl -k

line 176, sometimes -k is necessary for downloading

生成列表的正则表达式有问题?

gfwlist 的原文为:

...
.hkheadline.com*blog
.hkheadline.com/instantnews
...

生成的结果:

...
server=/hkheadline.com/127.0.0.1#5353
server=/hkheadline.comblog/127.0.0.1#5353
...

hkheadline.comblog 这个应该要被过滤掉?

multiple dns

the usage of multiple dns is allowed for each domain, so is it possible to add the feature?

3 warnings

WARNING:
The following lines in GfwList contain regex, and might be ignored:

459:/^https?://[^\/]+blogspot.(.*)/
537:/^https?://([^\/]+.)google.(ac|ad|ae|al|am|as|at|az|ba|be|bf|bg|bi|bj|bs|bt|by|ca|cat|cd|cf|cg|ch|ci|cl|cm|co.ao|co.bw|co.ck|co.cr|co.id|co.il|co.in|co.jp|co.ke|co.kr|co.ls|co.ma|com|com.af|com.ag|com.ai|com.ar|com.au|com.bd|com.bh|com.bn|com.bo|com.br|com.bz|com.co|com.cu|com.cy|com.do|com.ec|com.eg|com.et|com.fj|com.gh|com.gi|com.gt|com.hk|com.jm|com.kh|com.kw|com.lb|com.ly|com.mm|com.mt|com.mx|com.my|com.na|com.nf|com.ng|com.ni|com.np|com.om|com.pa|com.pe|com.pg|com.ph|com.pk|com.pr|com.py|com.qa|com.sa|com.sb|com.sg|com.sl|com.sv|com.tj|com.tr|com.tw|com.ua|com.uy|com.vc|com.vn|co.mz|co.nz|co.th|co.tz|co.ug|co.uk|co.uz|co.ve|co.vi|co.za|co.zm|co.zw|cv|cz|de|dj|dk|dm|dz|ee|es|fi|fm|fr|ga|ge|gg|gl|gm|gp|gr|gy|hk|hn|hr|ht|hu|ie|im|iq|is|it|je|jo|kg|ki|kz|la|li|lk|lt|lu|lv|md|me|mg|mk|ml|mn|ms|mu|mv|mw|mx|ne|nl|no|nr|nu|org|pl|pn|ps|pt|ro|rs|ru|rw|sc|se|sh|si|sk|sm|sn|so|sr|st|td|tg|tk|tl|tm|tn|to|tt|us|vg|vn|vu|ws)/./
1917:/twimg.edgesuite.net//?appledaily/

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.