Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/powerdns/weakforced

Anti-Abuse for servers at authentication time
https://github.com/powerdns/weakforced

anti-bot attack-prevention c-plus-plus gplv3 hacktoberfest intrusion-detection intrusion-prevention linux macos security

Last synced: 6 days ago
JSON representation

Anti-Abuse for servers at authentication time

Awesome Lists containing this project

README 

        
Weakforced

----------



The goal of 'wforce' is to detect brute forcing of passwords across many

servers, services and instances.  In order to support the real world, brute

force detection policy can be tailored to deal with "bulk, but legitimate"

users of your service, as well as botnet-wide slowscans of passwords.



The aim is to support the largest of installations, providing services to

hundreds of millions of users.  The current version of weakforced is not

quite there yet, although it certainly scales to support up to ten

million users, if not more. The limiting factor is number of logins

per second at peak.



Wforce is a project by Dovecot, PowerDNS and Open-Xchange. For historical

reasons, it lives in the PowerDNS github organization. If you have any questions, email

[email protected].



For detailed technical documentation, please go to [https://powerdns.github.io/weakforced/](https://powerdns.github.io/weakforced/).



Here is how it works:

 * Report successful logins via JSON http-api

 * Report unsuccessful logins via JSON http-api

 * Query if a login should be allowed to proceed, should be delayed, or ignored via http-api

 * API for querying the status of logins, IP addresses etc.

 * Runtime console for server introspection



wforce is aimed to receive message from services like:



 * IMAP

 * POP3

 * Webmail logins

 * FTP logins

 * Authenticated SMTP

 * Self-service logins

 * Password recovery services



By gathering failed and successful login attempts from as many services as

possible, brute forcing attacks as well as other suspicious behaviour

can be detected and prevented more effectively.



Inspiration:

http://www.techspot.com/news/58199-developer-reported-icloud-brute-force-password-hack-to-apple-nearly-six-month-ago.html



Installing

----------



Docker:



There is a docker image hosted on docker hub, see [https://powerdns.github.io/weakforced/](https://powerdns.github.io/weakforced/) for more details.



From GitHub:



The easy way:



```

$ git clone https://github.com/PowerDNS/weakforced.git

$ cd weakforced

$ git submodule init

$ git submodule update

$ builder/build.sh debian-bullseye | debian-bookworm | el-7 | el-8  | el-9 | amazon-2

```

This will build packages (`wforce`,`wforce-trackalert` and `wforce-debuginfo`) for the appropriate OS. You will need docker for the builder to work.



Note that since the 2.12 release, the built packages include an openresty luajit fork (`wforce-lua-dist`); this is because that fork

fixes issues in the main luajit library that wforce runs into under conditions of high load. The `wforce-lua-dist` package

also contains some lua modules that have proved useful to wforce deployments over the years as well as luarocks to install

new modules.



The hard way:



```

$ git clone https://github.com/PowerDNS/weakforced.git

$ cd weakforced

$ autoreconf -i

$ ./configure

$ make

```



This requires recent versions of libtool, automake and autoconf to be

installed.



It also requires:

 * A compiler supporting C++ 17

 * Lua 5.1+ development libraries (or LuaJIT if you configure --with-luajit)

 * Boost 1.61+

 * Protobuf compiler and protobuf development libraries

 * Getdns development libraries (if you want to use the DNS lookup functionality)

 * libsodium

 * python + virtualenv for regression testing

 * libgeoip-dev for GeoIP support

 * libsystemd-dev for systemd support

 * pandoc for building the manpages

 * libcurl-dev (OpenSSL version)

 * libhiredis-dev

 * libssl-dev

 * libprometheus-cpp (https://github.com/jupp0r/prometheus-cpp)

 * libmaxminddb-dev

 * libyaml-cpp-dev

 * libdrogon (https://github.com/drogonframework/drogon) - Used for the HTTP server

 * libjsoncpp-dev

 * libuuid-dev

 * libz-dev

 * docker for regression testing

 * python3 rather than python2

 * python-bottle for regression testing of webhooks



To build on OS X, `brew install readline` and use

`./configure PKG_CONFIG_PATH= LDFLAGS=-L/usr/local/opt/readline/lib CPPFLAGS=-I/usr/local/opt/readline/include`



Add --with-luajit to the end of the configure line if you want to use LuaJIT.



Policies

--------



There is a sensible, if very simple, default policy in wforce.conf (running without

this means *no* policy), and extensive support for crafting your own policies using

the insanely great Lua scripting language.



Note that although there is

a single Lua configuration file, the canonicalize, reset, report and allow functions run in

different lua states from the rest of the configuration. This mostly

"just works", but may lead to unexpected behaviour such as running Lua

commands at the server Lua prompt, and getting multiple answers

(because Lua commands are passed to all Lua states).



Sample:



```lua

-- set up the things we want to track

field_map = {}

-- use hyperloglog to track cardinality of (failed) password attempts

field_map["diffFailedPasswords"] = "hll"

-- track those things over 6x10 minute windows

newStringStatsDB("OneHourDB", 600, 6, field_map)



-- this function counts interesting things when "report" is invoked

function twreport(lt)

	sdb = getStringStatsDB("OneHourDB")

	if (not lt.success)

	then

	   sdb:twAdd(lt.remote, "diffFailedPasswords", lt.pwhash)

	   addrlogin = lt.remote:tostring() .. lt.login

	   sdb:twAdd(addrlogin, "diffFailedPasswords", lt.pwhash)

	end

end



function allow(lt)

	sdb = getStringStatsDB("OneHourDB")

	if(sdb:twGet(lt.remote, "diffFailedPasswords") > 50)

	then

		return -1, "", "", {} -- BLOCK!

	end

	// concatenate the IP address and login string

	addrlogin = lt.remote:tostring() .. lt.login	

	if(sdb:twGet(addrlogin, "diffFailedPasswords") > 3)

	then

		return 3, "tarpitted", "diffFailedPasswords", {} -- must wait for 3 seconds

	end



	return 0, "", "", {} -- OK!

end

```



Many more metrics are available to base decisions on. Some example

code is in [wforce.conf](wforce/wforce.conf), and more extensive examples are

in [wforce.conf.example](wforce/wforce.conf.example). For full

[documentation](docs/manpages), use "man wforce.conf".



To report (if you configured with 'webserver("127.0.0.1:8084", "secret")'):



```bash

$ for a in {1..101}

  do

    curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1", "pwhash":"1234'$a'", "success":"false"}' \

    http://127.0.0.1:8084/?command=report -u wforce:secret

  done

```



This reports 101 failed logins for one user, but with different password hashes.



Now to look up if we're still allowed in:



```bash

$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1", "pwhash":"1234"}' \

  http://127.0.0.1:8084/?command=allow -u wforce:super

{"status": -1, "msg": "diffFailedPasswords"}

```



It appears we are not!



You can also provide additional information for use by weakforce using

the optional "attrs" object. An example:



```bash

$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1",

"pwhash":"1234", "attrs":{"attr1":"val1", "attr2":"val2"}}' \

  http://127.0.0.1:8084/?command=allow -u wforce:super

{"status": 0, "msg": ""}

```



An example using the optional attrs object using multi-valued

attributes:



```bash

$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1",

"pwhash":"1234", "attrs":{"attr1":"val1", "attr2":["val2","val3"]}}' \

  http://127.0.0.1:8084/?command=allow -u wforce:super

{"status": 0, "msg": ""}

```



There is also a command to reset the stats for a given login and/or IP

Address, using the 'reset' command, the logic for which is also

implemented in Lua. The default configuration for reset is as follows:



```lua

function reset(type, login, ip)

	 sdb = getStringStatsDB("OneHourDB")

	 if (string.find(type, "ip"))

	 then

		sdb:twReset(ip)

	 end

	 if (string.find(type, "login"))

	 then

		sdb:twReset(login)

	 end

	 if (string.find(type, "ip") and string.find(type, "login"))

	 then

		iplogin = ip:tostring() .. login

		sdb:twReset(iplogin)

	 end

	 return true

end

```



To test it out, try the following to reset the login 'ahu':



```bash

$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu"}'\

  http://127.0.0.1:8084/?command=reset -u wforce:super

{"status": "ok"}

```



You can reset IP addresses also:



```bash

$ curl -X POST -H "Content-Type: application/json" --data '{"ip":"128.243.21.16"}'\

  http://127.0.0.1:8084/?command=reset -u wforce:super

{"status": "ok"}

```



Or both in the same command (this helps if you are tracking stats using compound keys

combining both IP address and login):



```bash

$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "ip":"FE80::0202:B3FF:FE1E:8329"}'\

  http://127.0.0.1:8084/?command=reset -u wforce:super

{"status": "ok"}

```



Finally there is a "ping" command, to check the server is up and

answering requests:



```bash

$ curl -X GET http://127.0.0.1:8084/?command=ping -u wforce:super

{"status": "ok"}

```



Console

-------

Available over TCP/IP, like this:

```lua

setKey("Ay9KXgU3g4ygK+qWT0Ut4gH8PPz02gbtPeXWPdjD0HE=")

controlSocket("0.0.0.0:4004")

```



Launch wforce as a daemon (`wforce --daemon`), to connect, run `wforce -c`.

Comes with autocomplete and command history. If you put an actual IP address

in place of 0.0.0.0, you can use the same config to listen and connect

remotely.



To get some stats, try:

```lua

> stats()

40 reports, 8 allow-queries, 40 entries in database

```



The [wforce manpage](docs/manpages/wforce.1.md) describes the

command-line options and all the possible console commands in more detail.



Spec

----

Wforce accepts reports with 4 mandatory fields plus multiple optional

fields.



Mandatory:

 * login (string): the user name or number or whatever

 * remote (ip address): the address the user arrived on

 * pwhash (string): a highly truncated hash of the password used

 * success (boolean): was the login a success or not?



Optional:

 * policy_reject (boolean) -  If the login was not successful only because of a policy-based reject from wforce (i.e. the username and password were correct).

 * attrs (json object): additional information about the login. For

 example, attributes from a user database.

 * device_id (string) - A string that represents the device that the user

   logged in from. For HTTP this would typically be the User-Agent

   string, and for IMAP it would be the IMAP client ID command string.

 * protocol (string) - A string representing the protocol used to login,

 e.g. "http", "imap", "pop3".

 * tls (boolean) - Whether or not the login was secured with TLS.



The entire HTTP API is [documented](docs/swagger/wforce_api.7.yml)

using the excellent OpenAPI (swagger) specification. 



The pwhash field deserves some clarification. In order to

distinguish actual brute forcing of a password, and repeated incorrect but

identical login attempts, we need some marker that tells us if passwords are

different.



Naively, we could hash the password, but this would spread knowledge of

secret credentials beyond where it should reasonably be. Even if we salt and

iterate the hash, or use a specific 'slow' hash, we're still spreading

knowledge.



However, if we take any kind of hash and truncate it severely, for example

to 12 bits, the hash tells us very little about the password itself - since

one in 4096 random strings will match it anyhow. But for detecting multiple

identical logins, it is good enough.



For additional security, hash the login name together with the password - this

prevents detecting different logins that might have the same password.



NOTE: wforce does not require any specific kind of hashing scheme, but it

is important that all services reporting successful/failed logins use the

same scheme!



When in doubt, try:



```SQL

TRUNCATE(SHA256(SECRET + LOGIN + '\x00' + PASSWORD), 12)

```



Which denotes to take the first 12 bits of the hash of the concatenation of

a secret, the login, a 0 byte and the password.  Prepend 4 0 bits to get

something that can be expressed as two bytes.



API Calls

---------

We can call 'report', and 'allow' commands. The optional 'attrs' field

enables the client to send additional data to weakforced.



To report, POST to /?command=report a JSON object with fields from the

LoginTuple as described above.



To request if a login should be allowed, POST to /?command=allow, again with

the LoginTuple. The result is a JSON object with a "status" field. If this is -1, do

not perform login validation (i.e. provide no clue to the client if the password

was correct or not, or even if the account exists).



If 0, allow login validation to proceed. If a positive number, sleep this

many seconds until allowing login validation to proceed.



Custom API Endpoints

--------------------



You can create custom API commands (REST Endpoints) using the

following configuration:



```lua

setCustomEndpoint("custom", customfunc)

```



which will create a new API command "custom", which calls the Lua

function "customfunc" whenever that command is invoked. Parameters to

custom commands are always in the same form, which is key-value pairs

wrapped in an 'attrs' object. For example, the following parameters

sents as json in the message body would be valid:



```json

{ "attrs" : { "key" : "value" }}

```



Custom functions return values are also key-value pairs, this time

wrapped in an 'r_attrs' object, along with a boolean success field,

for example:



```json

{ "r_attrs" : { "key" : "value" }, "success" : true}

```



An example configuration for a custom API endpoint would look like:



```lua

function custom(args)

	for k,v in pairs(args.attrs) do

		infoLog("custom func argument attrs", { key=k, value=v });

	end

	-- return consists of a boolean, followed by { key-value pairs }

	return true, { key=value }

end

setCustomEndpoint("custom", custom)

```



An example curl command would be:



```lua

% curl -v -X POST -H "Content-Type: application/json" --data

  '{"attrs":{"login1":"ahu", "remote": "127.0.0.1",  "pwhash":"1234"}}'

  http://127.0.0.1:8084/?command=custom -u wforce:super

{"r_attrs": {}, "success": true}

```



WebHooks

---------

It is possible to configure webhooks, which get called whenever

specific events occur. To do this, use the "addWebHook" configuration

command. For example:



```lua

config_keys={}

config_keys["url"] = "http://webhooks.example.com:8080/webhook/"

config_keys["secret"] = "verysecretcode"

events = { "report", "allow" }

addWebHook(events, config_keys)

```



The above will call the webhook at the specified url, for every report

and allow command received, with the body of the POST containing the

original json data sent to wforce. For more information use ["man

wforce.conf"](docs/manpages/wforce.conf.5.md) and

["man wforce_webhook"](docs/manpages/wforce_webhook.5.md).



Custom WebHooks

----------------



Custom webhooks can also be defined, which are not invoked based on

specific events, but instead from Lua. Configuration is similar to

normal webhooks:



```lua

config_keys={}

config_keys["url"] = "http://webhooks.example.com:8080/webhook/"

config_keys["secret"] = "verysecretcode"

config_keys["content-type"] = "application/json"

addCustomWebHook("mycustomhook", config_keys)

```



However, the webhook will only be invoked via the Lua

"runCustomWebHook" command, for example:



```lua

runCustomWebHook(mycustomhook", "{ \"foo\":\"bar\" }")

```



The above command will invoke the custom webhook "mycustomhook" with

the data contained in the second argument, which is simply a Lua

string. No parsing of the data is performed, however the Content-Type

of the webhook, which defaults to application/json can be customized

as shown above.



Blacklists

----------



Blacklisting capability is provided via either REST endpoints or Lua

commands, to add/delete IP addresses, logins or IP:login tuples from

the Blacklist. Blacklist information can be replicated (see below),

and also optionally persisted in a Redis DB. Use "man wforce.conf" to

learn more about the blacklist commands.



Load balancing: siblings

------------------------

For high-availability or performance reasons it may be desireable to run

multiple instances of wforce. To present a unified view of status however,

these instances then need to share data. To do so, wforce

implements a simple knowledge-sharing system.



The original version of wforce simply broadcast all received report

tuples (best effort, UDP) to all siblings. However the latest version

only broadcasts incremental changes to the underlying state databases,

namely the stats dbs and the blacklist.



The sibling list is parsed such that we don't broadcast messages to ourselves

accidentally, and can thus be identical across all servers.



Even if you configure siblings, stats db data is not replicated by default. To do

this, use the "twEnableReplication()" command on each

stats db for which you wish to enable replication. Blacklist

information is automatically replicated if you have configured siblings.



To define siblings, use:



```lua

setKey("Ay9KXgU3g4ygK+qWT0Ut4gH8PPz02gbtPeXWPdjD0HE=")

addSibling("192.168.1.79")

addSibling("192.168.1.30")

addSibling("192.168.1.54")

siblingListener("0.0.0.0")

```



The first line sets the authentication and encryption key for our sibling

communications. To make your own key (recommended), run `makeKey()` on the

console and paste the output in all your configuration files.



This last line configures that we also listen to our other siblings (which

is nice).  The default port is 4001, the protocol is UDP.



To view sibling stats:



```lua

> siblings()

Address                             Send Successes  Send Failures  Rcv Successes   Rcv Failures     Note

192.168.1.79:4001                   18              7              0               0

192.168.1.30:4001                   25              0              0               0

192.168.1.54:4001                   0               0              0               0                Self

```



With this setup, several wforces are all kept in sync, and can be load

balanced behind (for example) haproxy, which incidentally can also offer SSL.



GeoIP2 Support

-------------



GeoIP support is provided using the GeoIP2 Maxmind APIs DBs

(i.e. DBs ending in .mmdb). This is the preferred integration to use,

as support for GeoIP Legacy DBs will be discontinued by Maxmind

in 2019.



GeoIP2 DBs are represented by a Lua object that is created with the

following call:



```lua

newGeoIP2DB("Name", "/path/to/file.mmdb")

```



The Lua object is retrieved with the following call:



```lua

local mygeodb = getGeoIP2DB("Name")

```



You can then lookup information using the following calls:



* lookupCountry() - Returns the 2 letter country code associated with

  the IP address

* lookupISP() - Returns the name of the ISP associated with the IP

  address (requires the Maxmind ISP DB, which is only available on

  subscription)

* lookupCity - Rather than only returning a city name, this call

returns a Lua table which includes the following information:

    * country_code

    * country_name

    * region

    * city

    * postal_code

    * continent_code

    * latitude

    * longitude



For example:



```lua

local geoip_data = mygeodp:lookupCity(newCA("128.243.21.16"))

print(geoip_data.city)

print(geoip_data.longitude)

print(geoip_data.latitude)

```



Legacy GeoIP Support

-------------



Support for legacy GeoIP databases (i.e. ending in .dat) is

deprecated, since Maxmind will be discontinuing support for them

in 2019.



Three types of GeoIP lookup are supported:



* Country lookups - Initialized with initGeoIPDB() and looked up

    with lookupCountry()

* ISP Lookups - Initialized with initGeoIPISPDB() and looked up with

  lookupISP()

* City Lookup - Initialized with initGeoIPCityDB() and looked up

  with lookupCity()



The Country and ISP lookups return a string, while lookupCity()

returns a Lua map consisting of the following keys:

* country_code

* country_name

* region

* city

* postal_code

* continent_code

* latitude

* longitude



For example:

```lua

local geoip_data = lookupCity(newCA("128.243.21.16"))

print(geoip_data.city)

```



When a DB is initialized, wforce attempts to open both v4 and v6

versions of the database. If either is not found an error is thrown,

so make sure both ipv4 and v6 versions of each DB are

installed.



Additionally, when using the free/lite versions of the

databases, you may see errors such as "initGeoIPCityDB(): Error

initialising GeoIP (No geoip v6 city db available)". This is usually

because the filenames for the "lite" DBs are not the same as the

expected filenames for the full DBs, specifically all files must

start with GeoIP rather than GeoLite. Creating symbolic links to the

expected filenames will fix this problem, for example:



```

ln -s GeoLiteCityv6.dat GeoIPCityv6.dat

```