max197616 / nfqfilter Goto Github PK
View Code? Open in Web Editor NEWLicense: GNU General Public License v3.0
License: GNU General Public License v3.0
ВНИМАНИЕ! Данный проект больше не поддерживается. Рекомендуется использовать extFilter https://github.com/max197616/extfilter . Программа основана на разработке https://github.com/ircop/nfq_filter. Программа компилируется и работает под CentOS 7. Для сборки программы необходимы следующие программы и библиотеки: libnetfilter_queue libnfnetlink Poco >= 1.6 nDPI = 1.7-stable (ставится автоматически с github с наложением необходимых патчей) git patch Сборка программы: 1 ./autogen.sh 2 ./configure 3 make Для запуска программы необходимо указать путь к конфигурационному .ini файлу (-c в командой строке). Для запуска в режиме daemon необходимо указать ключи --daemon и --pidfile=/path/to/file.pid Для перенаправления трафика на фильтр необходимо использовать следующую конструкцию iptables: iptables -t mangle -A PREROUTING -s x.x.x.x/y -p tcp -m tcp -j NFQUEUE --queue-num 0 --queue-bypass где, x.x.x.x/y сеть, которую необходимо проверять. Для блокировки SSL/ip:port трафика необходимо использовать следующее правило: iptables -A FORWARD -m mark --mark 17 -p tcp -j REJECT --reject-with tcp-reset где 17, это значение из конфига nfqfilter. Так же вместо маркировки трафика можно сразу посылать TCP RST клиенту и серверу, если указать в конфиге send_rst = true . В каталоге contrib находятся примеры файлов для работы фильтра. Чтобы заставить фильтр перечитать конфиги, необходимо послать ему сигнал HUP.
https://eais.rkn.gov.ru/docs/Recomendation.pdf
п. 12
Блокировка домена по маске *.domain.com
Если в списке url есть что типа osago-blank.co./index.htm
то nfqfilter пропускает его.
Это проблему лучше решать на этапе формирования списков ?
Имееv url:
http://spys.ru/
Если у клиента маленькое мту, то url будет таким:
http://spys.r
u/
Короче, как идея реализовать добавление url в фильтр с учетом фрагментации?
В конфиге nfqfilter.ini:
; время вывода статистики по использованию памяти, минут
statistic_interval = 10
В логах:
2016-08-15 04:20:16.363 [30753] Information Application - nDPI memory (once): *** KB
2016-08-15 04:20:16.363 [30753] Information Application - nDPI memory per flow: *** KB
2016-08-15 04:20:16.363 [30753] Information Application - nDPI current memory usage: * MB
2016-08-15 04:20:16.363 [30753] Information Application - nDPI maximum memory usage: * GB
2016-08-15 04:20:16.363 [30753] Information Application - Total seen packets: _, Total seen bytes: *, Average packet size: *_* bytes, Traffic throughput: * pps
2016-08-15 04:20:16.363 [30753] Information Application - Total matched by ip/port: _, Total matched by ssl: *, Total matched by ssl/ip: _
2016-08-15 04:20:16.364 [30753] Information Application - Total redirected domains _, Total redirected urls: *, Total marked ssl: *, Total marked hosts: *, Total rst sended: _
2016-08-15 04:20:26.364 [30753] Information Application - nDPI memory (once): *** KB
2016-08-15 04:20:26.364 [30753] Information Application - nDPI memory per flow: *** KB
2016-08-15 04:20:26.364 [30753] Information Application - nDPI current memory usage: *** MB
2016-08-15 04:20:26.364 [30753] Information Application - nDPI maximum memory usage: * GB
2016-08-15 04:20:26.364 [30753] Information Application - Total seen packets: _, Total seen bytes: *, Average packet size: * bytes, Traffic throughput: * pps
2016-08-15 04:20:26.364 [30753] Information Application - Total matched by ip/port: _, Total matched by ssl: *__, Total matched by ssl/ip: *`
То есть на самом деле это не 10 минут а 10 секунд.
Наверно стоит поправить описание в nfqfilter.ini, или в коде. Не знаю где правильнее.
Небольшой баг
При наличии в urllist корневой страницы сайта domain/ блокируются все страницы.
При блокировке url вида domain/123 блокируюся все url вида domain/123*
Есть комп:
uname -a
Linux nas 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
при попытки собрать, получил:
# make
Making all in src
make[1]: Entering directory '/home/nfqfilter/src'
g++ -DHAVE_CONFIG_H -I. -I../include -I../nDPI/src/include -std=c++0x -O2 -pthread -MT main.o -MD -MP -MF .deps/main.Tp o -c -o main.o main.cpp
In file included from /usr/include/Poco/Util/ServerApplication.h:44:0,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Util/Application.h:358:35: error: ‘typedef class std::vector<std::__cxx11::basic_string<char> > Poco::Ut il::Application::ArgVec’ is private
typedef std::vector<std::string> ArgVec;
^
In file included from main.cpp:23:0:
../include/main.h:150:17: error: within this context
int main(const ArgVec& args);
^
main.cpp: In member function ‘virtual void nfqFilter::initialize(Poco::Util::Application&)’:
main.cpp:125:60: error: no matching function for call to ‘Poco::Logger::debug(const char [30], std::__cxx11::string&)’
logger().debug("URL additional info set to %s", add_p_type);
^
In file included from /usr/include/Poco/Util/Application.h:48:0,
from /usr/include/Poco/Util/ServerApplication.h:44,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
inline void Logger::debug(const std::string& msg)
^
/usr/include/Poco/Logger.h:415:13: note: candidate expects 1 argument, 2 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
inline bool Logger::debug() const
^
/usr/include/Poco/Logger.h:469:13: note: candidate expects 0 arguments, 2 provided
main.cpp:132:50: error: no matching function for call to ‘Poco::Logger::debug(const char [20], std::__cxx11::string&)’
logger().debug("HTTP code set to %s", http_code);
^
In file included from /usr/include/Poco/Util/Application.h:48:0,
from /usr/include/Poco/Util/ServerApplication.h:44,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
inline void Logger::debug(const std::string& msg)
^
/usr/include/Poco/Logger.h:415:13: note: candidate expects 1 argument, 2 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
inline bool Logger::debug() const
^
/usr/include/Poco/Logger.h:469:13: note: candidate expects 0 arguments, 2 provided
main.cpp:147:69: error: no matching function for call to ‘Poco::Logger::information(const char [25], int&)’
logger().information("Starting up on queue: %d",_config.queueNumber);
^
In file included from /usr/include/Poco/Util/Application.h:48:0,
from /usr/include/Poco/Util/ServerApplication.h:44,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Logger.h:409:13: note: candidate: void Poco::Logger::information(const string&)
inline void Logger::information(const std::string& msg)
^
/usr/include/Poco/Logger.h:409:13: note: candidate expects 1 argument, 2 provided
/usr/include/Poco/Logger.h:463:13: note: candidate: bool Poco::Logger::information() const
inline bool Logger::information() const
^
/usr/include/Poco/Logger.h:463:13: note: candidate expects 0 arguments, 2 provided
In file included from /usr/include/Poco/Util/ServerApplication.h:44:0,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Util/Application.h: In member function ‘virtual int nfqFilter::main(const ArgVec&)’:
/usr/include/Poco/Util/Application.h:358:35: error: ‘typedef class std::vector<std::__cxx11::basic_string<char> > Poco::Ut il::Application::ArgVec’ is private
typedef std::vector<std::string> ArgVec;
^
main.cpp:261:27: error: within this context
int nfqFilter::main(const ArgVec& args)
^
main.cpp: In member function ‘void nfqFilter::loadDomains(std::__cxx11::string&, AhoCorasickPlus*, DomainsMatchType*)’:
main.cpp:312:86: error: no matching function for call to ‘Poco::Logger::warning(const char [51], std::__cxx11::string&, st d::__cxx11::string&)’
logger().warning("Pattern '%s' already present database from file %s",insert,fn);
^
In file included from /usr/include/Poco/Util/Application.h:48:0,
from /usr/include/Poco/Util/ServerApplication.h:44,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Logger.h:397:13: note: candidate: void Poco::Logger::warning(const string&)
inline void Logger::warning(const std::string& msg)
^
/usr/include/Poco/Logger.h:397:13: note: candidate expects 1 argument, 3 provided
/usr/include/Poco/Logger.h:451:13: note: candidate: bool Poco::Logger::warning() const
inline bool Logger::warning() const
^
/usr/include/Poco/Logger.h:451:13: note: candidate expects 0 arguments, 3 provided
main.cpp:314:85: error: no matching function for call to ‘Poco::Logger::error(const char [45], std::__cxx11::string&, int& , std::__cxx11::string&)’
logger().error("Failed to add '%s' from line %d from file %s",insert,lineno,fn);
^
In file included from /usr/include/Poco/Util/Application.h:48:0,
from /usr/include/Poco/Util/ServerApplication.h:44,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Logger.h:391:13: note: candidate: void Poco::Logger::error(const string&)
inline void Logger::error(const std::string& msg)
^
/usr/include/Poco/Logger.h:391:13: note: candidate expects 1 argument, 4 provided
/usr/include/Poco/Logger.h:445:13: note: candidate: bool Poco::Logger::error() const
inline bool Logger::error() const
^
/usr/include/Poco/Logger.h:445:13: note: candidate expects 0 arguments, 4 provided
main.cpp:320:85: error: no matching function for call to ‘Poco::Logger::debug(const char [48], std::__cxx11::string&, int& , std::__cxx11::string&)’
logger().debug("Inserted domain: '%s' from line %d from file %s",str,lineno,fn);
^
In file included from /usr/include/Poco/Util/Application.h:48:0,
from /usr/include/Poco/Util/ServerApplication.h:44,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
inline void Logger::debug(const std::string& msg)
^
/usr/include/Poco/Logger.h:415:13: note: candidate expects 1 argument, 4 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
inline bool Logger::debug() const
^
/usr/include/Poco/Logger.h:469:13: note: candidate expects 0 arguments, 4 provided
main.cpp:322:84: error: no matching function for call to ‘Poco::Logger::debug(const char [47], std::__cxx11::string&, int& , std::__cxx11::string&)’
logger().debug("Updated domain: '%s' from line %d from file %s",str,lineno,fn);
^
In file included from /usr/include/Poco/Util/Application.h:48:0,
from /usr/include/Poco/Util/ServerApplication.h:44,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
inline void Logger::debug(const std::string& msg)
^
/usr/include/Poco/Logger.h:415:13: note: candidate expects 1 argument, 4 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
inline bool Logger::debug() const
^
/usr/include/Poco/Logger.h:469:13: note: candidate expects 0 arguments, 4 provided
main.cpp: In member function ‘void nfqFilter::loadURLs(std::__cxx11::string&, AhoCorasickPlus*)’:
main.cpp:360:94: error: no matching function for call to ‘Poco::Logger::warning(const char [62], std::__cxx11::string&, st d::__cxx11::string&)’
logger().warning("Pattern '%s' already present in the URL database from file %s",str,fn);
^
In file included from /usr/include/Poco/Util/Application.h:48:0,
from /usr/include/Poco/Util/ServerApplication.h:44,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Logger.h:397:13: note: candidate: void Poco::Logger::warning(const string&)
inline void Logger::warning(const std::string& msg)
^
/usr/include/Poco/Logger.h:397:13: note: candidate expects 1 argument, 3 provided
/usr/include/Poco/Logger.h:451:13: note: candidate: bool Poco::Logger::warning() const
inline bool Logger::warning() const
^
/usr/include/Poco/Logger.h:451:13: note: candidate expects 0 arguments, 3 provided
main.cpp:362:82: error: no matching function for call to ‘Poco::Logger::error(const char [45], std::__cxx11::string&, int& , std::__cxx11::string&)’
logger().error("Failed to add '%s' from line %d from file %s",str,lineno,fn);
^
In file included from /usr/include/Poco/Util/Application.h:48:0,
from /usr/include/Poco/Util/ServerApplication.h:44,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Logger.h:391:13: note: candidate: void Poco::Logger::error(const string&)
inline void Logger::error(const std::string& msg)
^
/usr/include/Poco/Logger.h:391:13: note: candidate expects 1 argument, 4 provided
/usr/include/Poco/Logger.h:445:13: note: candidate: bool Poco::Logger::error() const
inline bool Logger::error() const
^
/usr/include/Poco/Logger.h:445:13: note: candidate expects 0 arguments, 4 provided
main.cpp: In member function ‘void nfqFilter::loadHosts(std::__cxx11::string&, IPPortMap*)’:
main.cpp:394:47: error: no matching function for call to ‘Poco::Logger::debug(const char [17], std::__cxx11::string&, std: :__cxx11::string&)’
logger().debug("IP is %s port %s",ip,port);
^
In file included from /usr/include/Poco/Util/Application.h:48:0,
from /usr/include/Poco/Util/ServerApplication.h:44,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
inline void Logger::debug(const std::string& msg)
^
/usr/include/Poco/Logger.h:415:13: note: candidate expects 1 argument, 3 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
inline bool Logger::debug() const
^
/usr/include/Poco/Logger.h:469:13: note: candidate expects 0 arguments, 3 provided
main.cpp:397:45: error: no matching function for call to ‘Poco::Logger::debug(const char [19], std::__cxx11::string&)’
logger().debug("IP %s without port", ip);
^
In file included from /usr/include/Poco/Util/Application.h:48:0,
from /usr/include/Poco/Util/ServerApplication.h:44,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
inline void Logger::debug(const std::string& msg)
^
/usr/include/Poco/Logger.h:415:13: note: candidate expects 1 argument, 2 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
inline bool Logger::debug() const
^
/usr/include/Poco/Logger.h:469:13: note: candidate expects 0 arguments, 2 provided
main.cpp:406:57: error: no matching function for call to ‘Poco::Logger::debug(const char [24], std::__cxx11::string&, std: :__cxx11::string&)’
logger().debug("Adding port %s to ip %s", port, ip);
^
In file included from /usr/include/Poco/Util/Application.h:48:0,
from /usr/include/Poco/Util/ServerApplication.h:44,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
inline void Logger::debug(const std::string& msg)
^
/usr/include/Poco/Logger.h:415:13: note: candidate expects 1 argument, 3 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
inline bool Logger::debug() const
^
/usr/include/Poco/Logger.h:469:13: note: candidate expects 0 arguments, 3 provided
main.cpp:410:63: error: no matching function for call to ‘Poco::Logger::debug(const char [29], std::__cxx11::string&, int& )’
logger().debug("Inserted ip: %s from line %d", ip, lineno);
^
In file included from /usr/include/Poco/Util/Application.h:48:0,
from /usr/include/Poco/Util/ServerApplication.h:44,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
inline void Logger::debug(const std::string& msg)
^
/usr/include/Poco/Logger.h:415:13: note: candidate expects 1 argument, 3 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
inline bool Logger::debug() const
^
/usr/include/Poco/Logger.h:469:13: note: candidate expects 0 arguments, 3 provided
main.cpp:412:75: error: no matching function for call to ‘Poco::Logger::debug(const char [37], std::__cxx11::string&, int& , std::__cxx11::string&)’
logger().debug("Adding port %s from line %d to ip %s", port,lineno,ip);
^
In file included from /usr/include/Poco/Util/Application.h:48:0,
from /usr/include/Poco/Util/ServerApplication.h:44,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
inline void Logger::debug(const std::string& msg)
^
/usr/include/Poco/Logger.h:415:13: note: candidate expects 1 argument, 4 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
inline bool Logger::debug() const
^
/usr/include/Poco/Logger.h:469:13: note: candidate expects 0 arguments, 4 provided
main.cpp: In member function ‘void nfqFilter::loadSSLIP(const string&, Patricia*)’:
main.cpp:440:102: error: no matching function for call to ‘Poco::Logger::information(const char [61], std::__cxx11::string &, int&)’
logger().information("Unable to add IP address %s from line %d to the SSL IPs list", str, lineno);
^
In file included from /usr/include/Poco/Util/Application.h:48:0,
from /usr/include/Poco/Util/ServerApplication.h:44,
from ../include/main.h:23,
from main.cpp:23:
/usr/include/Poco/Logger.h:409:13: note: candidate: void Poco::Logger::information(const string&)
inline void Logger::information(const std::string& msg)
^
/usr/include/Poco/Logger.h:409:13: note: candidate expects 1 argument, 3 provided
/usr/include/Poco/Logger.h:463:13: note: candidate: bool Poco::Logger::information() const
inline bool Logger::information() const
^
/usr/include/Poco/Logger.h:463:13: note: candidate expects 0 arguments, 3 provided
Makefile:390: recipe for target 'main.o' failed
make[1]: *** [main.o] Error 1
make[1]: Leaving directory '/home/nfqfilter/src'
Makefile:340: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
Если в файл domains добавить доменное имя длиннее чем 191 символ (определили опытным путем), то он не фильтруется.
В тоже время, если этот же домен добавить в ulrs, то он удачно блокируется.
В данный момент времени в реестре есть одна запись длинной 206 символов:
www.legal-pills.net.kupit-tabletki-kadillak-adam-boby-yasnost-eyv-lyubov-ulybka.snezhok-vitaminki-diski-tably-kolosa-bubliki-kalachi-muzyka.ekstazi-kruglye-yeshki-eksta-eks-ti-si-tapki-bubly-pilsy-eyfor.net
Имеем сообщения (все) такого вида):
Host poker.123365-sb.com present in domain (file line [ERRFMT]) list from ip ХХ.ХХ.ХХ.ХХ
Вместо [ERRFMT] явно должен быть номер строки...
Добрый день. Маркированные пакеты не попадают в NAT таблицу.
Счетчики на этом правиле растут
iptables -A FORWARD -m mark --mark 17 -p tcp -j ACCEPT
А на этом нет
iptables -t nat -A PREROUTING -m mark --mark 17 -p tcp -j ACCEPT
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.