Giter Site home page Giter Site logo

powerdns / weakforced Goto Github PK

View Code? Open in Web Editor NEW
123.0 22.0 32.0 7.27 MB

Anti-Abuse for servers at authentication time

License: GNU General Public License v3.0

C++ 74.46% C 0.55% Shell 1.77% Python 6.94% Makefile 1.10% M4 13.52% Lua 0.64% Dockerfile 0.74% Jinja 0.28%
linux macos security intrusion-prevention intrusion-detection gplv3 c-plus-plus anti-bot attack-prevention hacktoberfest

weakforced's Issues

curl check does not fail but curl is still used 

checking for curl-config... no
checking whether libcurl is usable... no

leads to

  CXX      wforce.o
In file included from wforce.hh:35:0,
                 from wforce.cc:24:
webhook.hh:25:23: fatal error: curl/curl.h: No such file or directory
compilation terminated.

Debugging Lua scripts

When my runtime Lua is bad (and it often is), I get an error like:

Exception thrown by a callback function called by Lua

There's no identifying information about where the error occurred. I don't know if it's possible for wforce to capture (or if wforce even has visibility into the underlying exception), but if it is, it'd be extremely helpful if wforce reported the traceback for that exception (or even just the offending line). Or is there an option to enable that somewhere that I've missed?

Add support for ipcipher for logged IP addresses

ipcipher is described here:
https://powerdns.org/ipcipher/

This feature would be a global configuration option that causes all IP addresses (v4 and v6) to be encrypted using ipcipher before logging. Note that this would be for all "built-in" logging, and 'default' policy logging, but would not prevent people from using the custom logging functions to log unencrypted IP addresses. The encrypted IP address should be easily available for all functions, and/or a Lua function to encrypt the IP address.

A tool should be provided to enable the IP addresses in the (built-in) logs to be decrypted.

Migrate to Python 3

Currently the report api and the regressions tests use Python 2. This will be out of support by 2020.

Move all python to python 3.

make IPV6_PKTINFO available on osx

Commit 948216e makes sure we only try to use IPV6_PKTINFO when it is available, fixing compilation on OSX. However, that file was taken from PowerDNS which compiles fine on OSX without the ifdefs! It turns out one of the compiler flags that the pdns configure sets, makes the define available. We should port that part of configure to weakforced and dnsdist. We should also port the ifdefs to pdns.

Add stats counters for return codes and errors (Feature request)

It would be very useful for the existing stats call to also provide counters for the return codes, not just for the function calls.

IE, as well as counts on number of allow() and report() calls made, also counts on the number of allow() calls split by return code 0, -1, or >0.

Even better would be to have a generic count, so in the Lua code you could use something like
addstat(key) followed by increment(key) in an allow or report function, and then the normal stats() call would also return "counter.key: 1" or similar. I know we could implement this in lua using a database, but it would be slow.

Counters like this could then be fed into your favourite monitoring package (MRTG, Zabbix, Nagio, Cacti...) to obtain graphs of weakforced performance and behaviour.

Misc Questions

Lots of questions (but I didn't want to explode your "Issues" count in github)

  • Does the usual advice to use 'local' for all locally-scoped variables apply here as usual? Or is there any complication with 'local'? There aren't any 'local' instances in the wforce.conf or wforce.conf.example, so I wondered if there was a reason.

  • Is there a way to dump a database? Either in lua or via the CLI

  • Assuming there's a way to dump the db in lua or the cli (and if in lua, presumably I'd write a custom function to dump the database), will that lock anything? I.e. will traversing the db cause twGet/twAdd to block until the traversal is done?

  • If I'm collecting a number of stats, is it better to use fewer databases (with correspondingly larger # of entries), or more databases (more sharded)? Just wondering about local contention.

  • Is the RBL lookup a blocking function? That is, should I use them sparingly or be liberal with them? My RBLs will be served local to the box via unbound, so relatively quick.

  • Is there any way to do non-blocking calls to redis, a la https://github.com/openresty/lua-resty-redis ? I'd love to be able to track whitelists there, but not if calling out to redis is a blocking operation and ties down an entire thread.

  • I want to be able to set some masks on lt.remote addresses (but not set the whole database to use that mask, though I'm blanking on what that directive is). Is there a function to apply a netmask to a copy of lt.remote, or do I just need to do the usual conversion to Int and bitshift and convert back? The context is that I was to be able to add a stat to the db for both the full lt.remote IP as well as the /24 version of lt.remote

Sorry for so many questions. Thanks!

Add second class of service for no-blacklisting

In an ISP environment, features are best architected with a view towards minimal support desk load.

Log inspection has shown that a significant percentage of policy hits are from subscriber IPs. Outright blocking those (either via IP, or login+ip) is problematic.

When the 1hour policy rules are set to tarpit, and the 24hour rules set to blacklist (via custom lua to update the reject counts based on tarpit hits), it's advantageous to have a second 'list of IPs' to consult, and to bypass the 24hour rules if a match is found.

Update json11

Please sync json11 to latest upstream version. The current one fails with clang 5.0 with

json11.cpp:153:24: error: invalid operands to binary expression ('nullptr_t' and 'nullptr_t')
        return m_value < static_cast<const Value<tag, T> *>(other)->m_value;
               ~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
json11.cpp:209:5: note: in instantiation of member function 'json11::Value<json11::Json::Type::NUL, nullptr_t>::less' requested here
    JsonNull() : Value(nullptr) {}
    ^
1 error generated.

Add GeoIPv2 traits to attributes exposed by interface

With GeoIPv2, a new attribute 'traits' is exposed, which includes is_anonymous_proxy which is, clearly, very useful for certain weakforced rules.
Can we have the 'is_anonymous_proxy' trait exposed by the weakforced as part of the loginrecord structure? The is_in_european_union flag might be useful to some people as well. Ideally, all attributes would be accessible from the loginrecord structure.

e.g.

$ mmdblookup --file /usr/share/GeoIP/GeoIP2-City.mmdb --ip 185.121.168.254
  {
    "registered_country": 
      {
        "geoname_id": 
          3202326 <uint32>
        "is_in_european_union": 
          true <boolean>
        "iso_code": 
          "HR" <utf8_string>
        "names": 
          {
               ...
          }
      }
    "traits": 
      {
        "is_anonymous_proxy": 
          true <boolean>
      }
  }

expose generic redis API into wforce.conf

Use case

redis DB with a Set of allowed countries per login. e.g.:

redis-cli:

sadd darix DE

from the wforce.conf i could do something like: (pseudo code)

cur_ct = lookupCountry(lt.remote)
if (redis-call('EXISTS ' .. lt.login) == 1) then
   if (redis-call("SISMEMBER " .. lt.login .. " " .. cur_ct) == 0) then
     -- list present but the current country is not in the list
     return -1, "country blocked", "country blocked", { remoteCountry=cur_ct }
  end
else
-- case for all/no countries (depending on how you interpret no list)
end

Or would you prefer people use e.g. gh:nrk/redis-lua

Delegate blacklisting to Lua

Currently blacklisting is done in C++ before Lua is called. A global config key could disable this, so that when configured, blacklist checking is instead done by calling a lua function. This Lua function would check the blacklists, but then would also allow overriding with a custom lua function to do different stuff (like check attrs for per-user overrides or whatever).

Integrate LDAP support

Feature request -
It would be useful to have support for LDAP lookups integrated into Weakforced.
Although we can currently make LDAP queries by using the Lua LDAP module, this can be relatively slow, particularly when you're trying to make a fast allow() function call. An integrated LDAP lookup (similar to the integrated DNS lookup) could have caching (both positive and negative) to help speed things up, and better timeout handling; this would potentially allow people to configure per-account thresholds in their rules without performance problems.

Q: what is the average latency in a wforced cluster?

I have a question rather than a bug.
In a 3-node wforced cluster, if a report call causes an RRD item to be updated, how long (on average) would it take until this is replicated to all available siblings?
I am getting some unexpected behaviour in updates and need to rule this out.

Add ability to create "custom" REST API commands and hooks

Currently the main reporting commands are "report" and "allow", both of which are based on the LoginTuple structure and are based on the concept of reporting and allowing/rejecting login attempts. This is fine, but it would be good to be able to report on other types of actions, e.g. mail submissions. While we could add new commands to handle mail submissions, it might be better to allow "custom" commands, which allowed arbitrary data to be sent and returned via json key-value pairs. These could be configured from wforce.conf.

For example:

function reportMailSubmit(attrs)
if (attrs.expectedAttribute == "...")
then
...
end
return { return_value, message_for_client, log_message, return_attrs }
end

addCustomCommand("reportMailSubmit", reportMailSubmit)

resetEntity() command to reset time window stats for a particular key

New resetEntity() command in addition to existing allow() and report() commands over the web interface.

Should allow reset of tine window stats related to a particular key: username or IP address.

For example:

...?cmd=resetEntity/

{ "ip" = "10.3.4.5" }
or
{"ip" = "::1" }
or
{ "username" = "neilcook" }

The result should be that all entries of the supplied type with the supplied value as a key are deleted.

Support HTTP persistent connections

Support HTTP 1.1 persistent connections and the HTTP 1.0 "Connection: keep-alive".

Also timeout connections that have not been used for N seconds (configurable).

Package report_api as a deployable webapp

Currently the report_api app is included as a non-production-ready web app.

It should be packaged separately as a deployable web app ready for production use.

Preferably by someone who knows how to do this (i.e. not @neilcook!)

Restrict Lua includes to a whitelisted set of directories

  • Issue type: Security Feature request

Short description

Currently Lua can include files from anywhere in the filesystem, which is a possible attack vector e.g. Lua reading an auto-generated file in /tmp (although since we set private tmp that particular attack is actually not feasible).
This requests that we use the system ReadWritePaths=, ReadOnlyPaths=, InaccessiblePaths= to protect pdns from these kinds of attacks. This should work so that we whitelist specific directories and all other directories cannot be "seen" by the process (assuming the above systems.exec commands can enable this).
This doesn't stop an admin creating a world-writable directory underneath the whitelisted directories, but as lieter says, you can't guard against that kind of stupid. It does however stop admins from including Lua files from "anywhere" in the filesystem,.

wforce daemon cannot parse chunked request

Weakforced fails with following HTTP request:

POST /?command=allow HTTP/1.1
Host: localhost:8084
Connection: close
Content-Type: application/json
Authorization: Basic d2ZvcmNlOnN1cGVy
Content-Transfer-Encoding: chunked

0037
{"login":"cmouse","pwhash":"1234","remote":"127.0.0.1"}
0

I'd expect this to be parsed and processed.

Replication Forwarders

Currently replication is expected to be a full mesh network, i.e. we accept replication events from all configured siblings and also send replication events to all configured siblings. This does mean that as the number of servers increases, each server has to send every replication event to an ever increasing number of siblings. In multi-site environments, each server in each site has to know about and send events to every server in every site.

Thus it would also be useful (e.g. multi-site environment) to have the ability to configure a server that received replication events from all configured siblings, but only sent them on to a specific (separate) list of servers. This would allow point-to-point links for replication traffic between sites for example.

There are some challenges with this approach - e.g. how to handle redundancy. For example if I send all replication events from Site A to a server at Site B, which then forwarded those events to a single server at Site B, what happens if that server goes down. If the forwarder sends to multiple servers at Site B, how to prevent multiple replication events from being sent to the wforce servers in Site B.

Due to the above, the existing UDP transport for replication may not be the best transport for this problem. If we used HTTP for example, we could make use of load-balancing/HA solutions such as HAProxy.

Add allow/report/reset etc. stats to the 5 min stats logging

Currently:
2017-12-14T14:37:53.571193+00:00 dovauth-ch2g-03o wforce[28622]: stats last 300 secs: WTW_0_1=20194 WTW_1_10=1 WTW_10_100=0 WTW_100_1000=0 WTW_Slow=0 WTR_0_1=20175 WTR_1_10=19 WTR_10_100=0 WTR_100_1000=0 WTR_Slow=0

Proposed - something like:
2017-12-14T14:37:53.571193+00:00 dovauth-ch2g-03o wforce[28622]: stats last 300 secs: WTW_0_1=20194 WTW_1_10=1 WTW_10_100=0 WTW_100_1000=0 WTW_Slow=0 WTR_0_1=20175 WTR_1_10=19 WTR_10_100=0 WTR_100_1000=0 WTR_Slow=0 allow=30293 report=29283 reset=2345

Need to think about whether custom endpoints should be included.

Client gets disconnected on bad command

Product: weakforced
Version: 1:1.2.0-2

~# wforce  -c
Read configuration from '/etc/wforce.conf'
Connecting to 0.0.0.0:4004
> status
Fatal error: EOF while writing message

And logs show

Jan 11 15:44:19 dev02 wforce[29825]: Got control connection from 127.0.0.1:58726
Jan 11 15:44:22 dev02 wforce[29825]: Got an exception in client connection from 127.0.0.1:58726: [string "status"]:1: '=' expected near '<eof>'
Jan 11 15:44:22 dev02 wforce[29855]: Fatal error: EOF while writing message

Custom functions appear to return HTTP status 404

Created a custom function getstats to let me query the current database content. When calling this via the 8084 API, it returns the content correctly, but has an HTTP status 404.
The function returns a true status, and the calling IP is in the ACL subnet.

function getstats (args)
sdb = getStringStatsDB("OneHourDB")
if ( args.attrs.remote and args.attrs.remote ~= "" )
then
  ip = newCA(args.attrs.remote)
  fpbyip = sdb:twGet(ip, "failedPasswords")
end
fpbylogin = sdb:twGet(args.attrs.login, "failedPasswords")
return true, { login=args.attrs.login, ip=args.attrs.remote, failedpasswordsbylogin=fpbylogin, failedpasswordsbyip=fpbyip }
end
setCustomEndpoint("getstats",false,getstats)

This seems to be an error in the wforced handler?

Make console command returns consistent

Currently the console commands are somewhat inconsistent. Some will return different strings for success/error, others will return nothing at all for either case, while errors appear in the logs. In future this should be consistent, and errors should not be restricted to the logs, but visible to the user (or application) that invoked the command.

Suggestion for twReset

There are times when I'd like to reset a key but only for a certain field name. For that case, I'm currently doing a db:twSub( .. , twGet( ... ) ) (i.e. subtracting the current value to get back to 0. Might be useful to have a second arg to twReset that takes the field name to reset.

getDBStats() returning wrong status

wforce v2.0.0-8

The getDBStats() call appears to always return a blacklisted status of true, regardless of the actual fact.

The bl_reason and bl_expiry return empty strings when the item is in reality not listed; possibly these should be absent?

$ curl -s -X GET -H Authorization: Basic XXXX -H Content-Type: application/json http://localhost:8084/?command=getBL
{  "bl_entries": [ { "expiration": "2019-Feb-20 05:23:45", "ip": "1.1.1.1/32", "reason": "Manual" }]}

These items are not in the blacklist but return the wrong value:

$ curl -s -X POST -H Authorization: Basic XXXX -H Content-Type: application/json --data {"login":"[email protected]"} http://localhost:8084/?command=getDBStats
{"bl_expire": "", "bl_reason": "", "blacklisted": true, "login": "[email protected]", "stats": {}}

$ curl -s -X POST -H Authorization: Basic XXXX -H Content-Type: application/json --data {"ip":"1.2.3.4"} http://localhost:8084/?command=getDBStats
{"bl_expire": "", "bl_reason": "", "blacklisted": true, "ip": "1.2.3.4", "stats": {}}

This item is in the blacklist and returns correctly:

$ curl -s -X POST -H Authorization: Basic XXXX -H Content-Type: application/json --data {"ip":"1.1.1.1"} http://localhost:8084/?command=getDBStats
{"bl_expire": "2019-Feb-20 05:23:45", "bl_reason": "Manual", "blacklisted": true, "ip": "1.1.1.1", "stats": {}}

Add prometheus metrics

All of the stats which are currently logged could also be available via prometheus, on the same web server that currently provides the REST API (e.g. on a different endpoint).

Debian Files

  • The debian/control references a package called libprotobuf-def, but the package is actually libprotobuf-dev
  • The debian/postinst file is a little intrusive. It really shouldn't be updating the existing wforce.conf file unless it's creating it (IMO).
  • I updated my debian/postinst's definition of where to put wforce.conf to make it inline with other places that wforce seems to look for wforce.conf: WFORCECONF=/etc/wforce/wforce.conf
  • The version in debian/changelog is 0.9 and there's an issue with a newline in it which causes build scripts to abort:
    dpkg-buildpackage: warning: debian/changelog(l2): found end of file where expected first heading
  • The debian/changelog file's says 'wforced', where debian/control has 'wforce'. For me at least, I have to fix the name in debian/changelog to 'wforce' to get packaging to work.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.