powerdns / weakforced Goto Github PK
View Code? Open in Web Editor NEWAnti-Abuse for servers at authentication time
License: GNU General Public License v3.0
Anti-Abuse for servers at authentication time
License: GNU General Public License v3.0
Currently wforce config file is placed to /etc/ root but it would probably make sense to have an own directory for it.
This repository does not contain a changelog, it would be nice to have one.
The provided systemd file does not work in debian/jessie because weakforced fails to notify properly.
Currently itβs hard coded
checking for curl-config... no
checking whether libcurl is usable... no
leads to
CXX wforce.o
In file included from wforce.hh:35:0,
from wforce.cc:24:
webhook.hh:25:23: fatal error: curl/curl.h: No such file or directory
compilation terminated.
When my runtime Lua is bad (and it often is), I get an error like:
Exception thrown by a callback function called by Lua
There's no identifying information about where the error occurred. I don't know if it's possible for wforce to capture (or if wforce even has visibility into the underlying exception), but if it is, it'd be extremely helpful if wforce reported the traceback for that exception (or even just the offending line). Or is there an option to enable that somewhere that I've missed?
ipcipher is described here:
https://powerdns.org/ipcipher/
This feature would be a global configuration option that causes all IP addresses (v4 and v6) to be encrypted using ipcipher before logging. Note that this would be for all "built-in" logging, and 'default' policy logging, but would not prevent people from using the custom logging functions to log unencrypted IP addresses. The encrypted IP address should be easily available for all functions, and/or a Lua function to encrypt the IP address.
A tool should be provided to enable the IP addresses in the (built-in) logs to be decrypted.
Currently the report api and the regressions tests use Python 2. This will be out of support by 2020.
Move all python to python 3.
Commit 948216e makes sure we only try to use IPV6_PKTINFO when it is available, fixing compilation on OSX. However, that file was taken from PowerDNS which compiles fine on OSX without the ifdefs! It turns out one of the compiler flags that the pdns configure sets, makes the define available. We should port that part of configure to weakforced and dnsdist. We should also port the ifdefs to pdns.
Currently GeoIP support is limited to GeoIP legacy databases.
Currently if an IP/Login is blacklisted, the allow call doesn't return the reason.
Also the getDBStats call doesn't return the blacklist reason, it would be good it if did.
The logging subsystem should only log to stderr when running under systemd.
It would be very useful for the existing stats call to also provide counters for the return codes, not just for the function calls.
IE, as well as counts on number of allow() and report() calls made, also counts on the number of allow() calls split by return code 0, -1, or >0.
Even better would be to have a generic count, so in the Lua code you could use something like
addstat(key) followed by increment(key) in an allow or report function, and then the normal stats() call would also return "counter.key: 1" or similar. I know we could implement this in lua using a database, but it would be slow.
Counters like this could then be fed into your favourite monitoring package (MRTG, Zabbix, Nagio, Cacti...) to obtain graphs of weakforced performance and behaviour.
Lots of questions (but I didn't want to explode your "Issues" count in github)
Does the usual advice to use 'local' for all locally-scoped variables apply here as usual? Or is there any complication with 'local'? There aren't any 'local' instances in the wforce.conf or wforce.conf.example, so I wondered if there was a reason.
Is there a way to dump a database? Either in lua or via the CLI
Assuming there's a way to dump the db in lua or the cli (and if in lua, presumably I'd write a custom function to dump the database), will that lock anything? I.e. will traversing the db cause twGet/twAdd to block until the traversal is done?
If I'm collecting a number of stats, is it better to use fewer databases (with correspondingly larger # of entries), or more databases (more sharded)? Just wondering about local contention.
Is the RBL lookup a blocking function? That is, should I use them sparingly or be liberal with them? My RBLs will be served local to the box via unbound, so relatively quick.
Is there any way to do non-blocking calls to redis, a la https://github.com/openresty/lua-resty-redis ? I'd love to be able to track whitelists there, but not if calling out to redis is a blocking operation and ties down an entire thread.
I want to be able to set some masks on lt.remote addresses (but not set the whole database to use that mask, though I'm blanking on what that directive is). Is there a function to apply a netmask to a copy of lt.remote, or do I just need to do the usual conversion to Int and bitshift and convert back? The context is that I was to be able to add a stat to the db for both the full lt.remote IP as well as the /24 version of lt.remote
Sorry for so many questions. Thanks!
So that you can exclude IPs from WL Ranges (e.g. exclude a /24 from a /16 WL).
In an ISP environment, features are best architected with a view towards minimal support desk load.
Log inspection has shown that a significant percentage of policy hits are from subscriber IPs. Outright blocking those (either via IP, or login+ip) is problematic.
When the 1hour policy rules are set to tarpit, and the 24hour rules set to blacklist (via custom lua to update the reject counts based on tarpit hits), it's advantageous to have a second 'list of IPs' to consult, and to bypass the 24hour rules if a match is found.
A custom message to be returned in the response to the allow() command, set from Lua workflow.
Please sync json11 to latest upstream version. The current one fails with clang 5.0 with
json11.cpp:153:24: error: invalid operands to binary expression ('nullptr_t' and 'nullptr_t')
return m_value < static_cast<const Value<tag, T> *>(other)->m_value;
~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
json11.cpp:209:5: note: in instantiation of member function 'json11::Value<json11::Json::Type::NUL, nullptr_t>::less' requested here
JsonNull() : Value(nullptr) {}
^
1 error generated.
With GeoIPv2, a new attribute 'traits' is exposed, which includes is_anonymous_proxy which is, clearly, very useful for certain weakforced rules.
Can we have the 'is_anonymous_proxy' trait exposed by the weakforced as part of the loginrecord structure? The is_in_european_union flag might be useful to some people as well. Ideally, all attributes would be accessible from the loginrecord structure.
e.g.
$ mmdblookup --file /usr/share/GeoIP/GeoIP2-City.mmdb --ip 185.121.168.254
{
"registered_country":
{
"geoname_id":
3202326 <uint32>
"is_in_european_union":
true <boolean>
"iso_code":
"HR" <utf8_string>
"names":
{
...
}
}
"traits":
{
"is_anonymous_proxy":
true <boolean>
}
}
Use case
redis DB with a Set of allowed countries per login. e.g.:
redis-cli:
sadd darix DE
from the wforce.conf i could do something like: (pseudo code)
cur_ct = lookupCountry(lt.remote)
if (redis-call('EXISTS ' .. lt.login) == 1) then
if (redis-call("SISMEMBER " .. lt.login .. " " .. cur_ct) == 0) then
-- list present but the current country is not in the list
return -1, "country blocked", "country blocked", { remoteCountry=cur_ct }
end
else
-- case for all/no countries (depending on how you interpret no list)
end
Or would you prefer people use e.g. gh:nrk/redis-lua
Currently blacklisting is done in C++ before Lua is called. A global config key could disable this, so that when configured, blacklist checking is instead done by calling a lua function. This Lua function would check the blacklists, but then would also allow overriding with a custom lua function to do different stuff (like check attrs for per-user overrides or whatever).
Just did an installation on Ubuntu 16.04 but when I invoke ./wforce
I am getting an error:
Fatal error: cannot find regexes.yaml at /usr/local/etc/regexes.yaml
Any ideas?
Feature request -
It would be useful to have support for LDAP lookups integrated into Weakforced.
Although we can currently make LDAP queries by using the Lua LDAP module, this can be relatively slow, particularly when you're trying to make a fast allow() function call. An integrated LDAP lookup (similar to the integrated DNS lookup) could have caching (both positive and negative) to help speed things up, and better timeout handling; this would potentially allow people to configure per-account thresholds in their rules without performance problems.
I have a question rather than a bug.
In a 3-node wforced cluster, if a report
call causes an RRD item to be updated, how long (on average) would it take until this is replicated to all available siblings?
I am getting some unexpected behaviour in updates and need to rule this out.
checking for LIBHIREDIS... no
checking for LIBHIREDIS... no
leads to
blacklist.hh:39:29: fatal error: hiredis/hiredis.h: No such file or directory
Currently the main reporting commands are "report" and "allow", both of which are based on the LoginTuple structure and are based on the concept of reporting and allowing/rejecting login attempts. This is fine, but it would be good to be able to report on other types of actions, e.g. mail submissions. While we could add new commands to handle mail submissions, it might be better to allow "custom" commands, which allowed arbitrary data to be sent and returned via json key-value pairs. These could be configured from wforce.conf.
For example:
function reportMailSubmit(attrs)
if (attrs.expectedAttribute == "...")
then
...
end
return { return_value, message_for_client, log_message, return_attrs }
end
addCustomCommand("reportMailSubmit", reportMailSubmit)
New resetEntity() command in addition to existing allow() and report() commands over the web interface.
Should allow reset of tine window stats related to a particular key: username or IP address.
For example:
...?cmd=resetEntity/
{ "ip" = "10.3.4.5" }
or
{"ip" = "::1" }
or
{ "username" = "neilcook" }
The result should be that all entries of the supplied type with the supplied value as a key are deleted.
Support HTTP 1.1 persistent connections and the HTTP 1.0 "Connection: keep-alive".
Also timeout connections that have not been used for N seconds (configurable).
Currently the report_api app is included as a non-production-ready web app.
It should be packaged separately as a deployable web app ready for production use.
Preferably by someone who knows how to do this (i.e. not @neilcook!)
Currently Lua can include files from anywhere in the filesystem, which is a possible attack vector e.g. Lua reading an auto-generated file in /tmp (although since we set private tmp that particular attack is actually not feasible).
This requests that we use the system ReadWritePaths=, ReadOnlyPaths=, InaccessiblePaths= to protect pdns from these kinds of attacks. This should work so that we whitelist specific directories and all other directories cannot be "seen" by the process (assuming the above systems.exec commands can enable this).
This doesn't stop an admin creating a world-writable directory underneath the whitelisted directories, but as lieter says, you can't guard against that kind of stupid. It does however stop admins from including Lua files from "anywhere" in the filesystem,.
Weakforced fails with following HTTP request:
POST /?command=allow HTTP/1.1
Host: localhost:8084
Connection: close
Content-Type: application/json
Authorization: Basic d2ZvcmNlOnN1cGVy
Content-Transfer-Encoding: chunked
0037
{"login":"cmouse","pwhash":"1234","remote":"127.0.0.1"}
0
I'd expect this to be parsed and processed.
Currently replication is expected to be a full mesh network, i.e. we accept replication events from all configured siblings and also send replication events to all configured siblings. This does mean that as the number of servers increases, each server has to send every replication event to an ever increasing number of siblings. In multi-site environments, each server in each site has to know about and send events to every server in every site.
Thus it would also be useful (e.g. multi-site environment) to have the ability to configure a server that received replication events from all configured siblings, but only sent them on to a specific (separate) list of servers. This would allow point-to-point links for replication traffic between sites for example.
There are some challenges with this approach - e.g. how to handle redundancy. For example if I send all replication events from Site A to a server at Site B, which then forwarded those events to a single server at Site B, what happens if that server goes down. If the forwarder sends to multiple servers at Site B, how to prevent multiple replication events from being sent to the wforce servers in Site B.
Due to the above, the existing UDP transport for replication may not be the best transport for this problem. If we used HTTP for example, we could make use of load-balancing/HA solutions such as HAProxy.
Currently:
2017-12-14T14:37:53.571193+00:00 dovauth-ch2g-03o wforce[28622]: stats last 300 secs: WTW_0_1=20194 WTW_1_10=1 WTW_10_100=0 WTW_100_1000=0 WTW_Slow=0 WTR_0_1=20175 WTR_1_10=19 WTR_10_100=0 WTR_100_1000=0 WTR_Slow=0
Proposed - something like:
2017-12-14T14:37:53.571193+00:00 dovauth-ch2g-03o wforce[28622]: stats last 300 secs: WTW_0_1=20194 WTW_1_10=1 WTW_10_100=0 WTW_100_1000=0 WTW_Slow=0 WTR_0_1=20175 WTR_1_10=19 WTR_10_100=0 WTR_100_1000=0 WTR_Slow=0 allow=30293 report=29283 reset=2345
Need to think about whether custom endpoints should be included.
Product: weakforced
Version: 1:1.2.0-2
~# wforce -c
Read configuration from '/etc/wforce.conf'
Connecting to 0.0.0.0:4004
> status
Fatal error: EOF while writing message
And logs show
Jan 11 15:44:19 dev02 wforce[29825]: Got control connection from 127.0.0.1:58726
Jan 11 15:44:22 dev02 wforce[29825]: Got an exception in client connection from 127.0.0.1:58726: [string "status"]:1: '=' expected near '<eof>'
Jan 11 15:44:22 dev02 wforce[29855]: Fatal error: EOF while writing message
Created a custom function getstats
to let me query the current database content. When calling this via the 8084 API, it returns the content correctly, but has an HTTP status 404.
The function returns a true
status, and the calling IP is in the ACL subnet.
function getstats (args)
sdb = getStringStatsDB("OneHourDB")
if ( args.attrs.remote and args.attrs.remote ~= "" )
then
ip = newCA(args.attrs.remote)
fpbyip = sdb:twGet(ip, "failedPasswords")
end
fpbylogin = sdb:twGet(args.attrs.login, "failedPasswords")
return true, { login=args.attrs.login, ip=args.attrs.remote, failedpasswordsbylogin=fpbylogin, failedpasswordsbyip=fpbyip }
end
setCustomEndpoint("getstats",false,getstats)
This seems to be an error in the wforced handler?
Currently a blacklist entry cannot be deleted from Lua, this should be fixed.
Currently the console commands are somewhat inconsistent. Some will return different strings for success/error, others will return nothing at all for either case, while errors appear in the logs. In future this should be consistent, and errors should not be restricted to the logs, but visible to the user (or application) that invoked the command.
There are times when I'd like to reset a key but only for a certain field name. For that case, I'm currently doing a db:twSub( .. , twGet( ... ) ) (i.e. subtracting the current value to get back to 0. Might be useful to have a second arg to twReset that takes the field name to reset.
wforce v2.0.0-8
The getDBStats() call appears to always return a blacklisted status of true, regardless of the actual fact.
The bl_reason and bl_expiry return empty strings when the item is in reality not listed; possibly these should be absent?
$ curl -s -X GET -H Authorization: Basic XXXX -H Content-Type: application/json http://localhost:8084/?command=getBL
{ "bl_entries": [ { "expiration": "2019-Feb-20 05:23:45", "ip": "1.1.1.1/32", "reason": "Manual" }]}
These items are not in the blacklist but return the wrong value:
$ curl -s -X POST -H Authorization: Basic XXXX -H Content-Type: application/json --data {"login":"[email protected]"} http://localhost:8084/?command=getDBStats
{"bl_expire": "", "bl_reason": "", "blacklisted": true, "login": "[email protected]", "stats": {}}
$ curl -s -X POST -H Authorization: Basic XXXX -H Content-Type: application/json --data {"ip":"1.2.3.4"} http://localhost:8084/?command=getDBStats
{"bl_expire": "", "bl_reason": "", "blacklisted": true, "ip": "1.2.3.4", "stats": {}}
This item is in the blacklist and returns correctly:
$ curl -s -X POST -H Authorization: Basic XXXX -H Content-Type: application/json --data {"ip":"1.1.1.1"} http://localhost:8084/?command=getDBStats
{"bl_expire": "2019-Feb-20 05:23:45", "bl_reason": "Manual", "blacklisted": true, "ip": "1.1.1.1", "stats": {}}
All of the stats which are currently logged could also be available via prometheus, on the same web server that currently provides the REST API (e.g. on a different endpoint).
Should be able to use the web hook framework to send custom events to configured webhooks
This would work in exactly the same way as the native blacklist.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.