Blocking IPs in HAProxy

Question

A client's website is currently under attack, and I've been called in to fix it.

A huge number of IPs (easily over 5,000) is constantly hitting /login, presumably trying to bruteforce their way in.

I've changed the site so the page returns a 500 error, but they're not giving up. Obviously this is no good for the real users, who now can't log in.

Load balancing is done via HAProxy, which I have very little knowledge of (although I have a lot more than I did a few hours ago). I've tried lots of sensible things that I've found online but none seem to have helped, probably because there is such a big block of IPs performing the attack.

Onto the question, then: How can I reject an IP if it hits /login more than X times in Y seconds?

And, subpoint - how can I see a log of rejections so I know it's actually working?

Here is a sample of the haproxy.log:

Jun  3 14:24:50 hap-server haproxy[11831]: 46.161.62.79:15290 [03/Jun/2017:14:24:49.505] www-https-test~ www-backend/www-03 751/0/202/38/991 500 220 - - ---- 428/428/120/38/0 0/0 "GET /login HTTP/1.1"
Jun  3 14:24:50 hap-server haproxy[11831]: 46.161.63.132:47804 [03/Jun/2017:14:24:49.505] www-https-test~ www-backend/www-04 751/0/202/38/991 500 220 - - ---- 428/428/119/42/0 0/0 "GET /login HTTP/1.1"
Jun  3 14:24:50 hap-server haproxy[11831]: 46.161.62.43:53176 [03/Jun/2017:14:24:49.505] www-https-test~ www-backend/www-02 751/0/202/38/991 500 220 - - ---- 428/428/118/38/0 0/0 "GET /login HTTP/1.1"

Answer 1

Thanks to a comment by EEAA, I was able to solve this using fail2ban.

There's very little documentation about how to use fail2ban with HAProxy, however - so little in fact that this page is already nearing the top of a Google search for "haproxy fail2ban", so I'll detail how I did it.

First of all, install fail2ban. If you can't do that bit you probably shouldn't carry on without some more help.

fail2ban works by scanning your access logs, looking for a pattern you set up. If it finds that pattern X times in Y seconds, it will automatically ban that IP for Z seconds.

Your HAProxy log should be at /var/log/haproxy.log. If you're under heavy load it might be too big to open with Vim or Nano, so just look at the last few lines by using tail: tail -n50 /var/log/haproxy.log

A sample of a "bad" entry in my log looked like this:

Jun  3 16:48:03 hap-server haproxy[21751]: 178.159.100.29:48806 [03/Jun/2017:16:48:03.735] www-https-test~ www-backend/www-04 172/0/2/3/177 200 339 - - ---- 36/36/0/0/0 0/0 "GET /login HTTP/1.1"

The important bits we want fail2ban to get hold of are the IP of the attacker and the page they are hitting.

To tell fail2ban how to do this, first we want to create a filter. Create a file in the folder /etc/fail2ban/filter.d. I called mine applogin.conf but you can call it what you like, as long as it ends in .conf.

The contents were as follows:

[Definition]

failregex = ^.*haproxy\[[0-9]+\]: <HOST>:.* "(GET |POST )/login HTTP/1.1"$
ignoreregex =

<HOST> is the point in the line from your log where the IP address appears. If you want to use the same regex as me, replace /login with the address the attackers are targeting on your server.

Now you have to tell fail2ban that you want it to look for that filter. Make a copy of /etc/fail2ban/jail.conf (in case you mess up and need to start again), then open jail.conf and add the following lines at the bottom.

[app-login]
enabled  = true
bantime  = 1200
findtime = 120
maxretry = 6
filter   = applogin
logpath  = /var/log/haproxy.log
port     = http,https

The part in the square brackets is just a name - use what you like. bantime is the number of seconds a user is banned for. maxretry is the amount of times a user can hit the page in findtime seconds. So, in my example, if someone tries to get to www.mydomain.com/login more than six times (or maybe exactly six times, I'm not sure) in 2 minutes then they will be banned for 20 minutes. filter is the name of the file you created in /etc/fail2ban/filter.d (but without the .conf). logpath is the path to the log file you want it to search in.

Save that file, then restart fail2ban: service fail2ban restart Tail the fail2ban logfile: tail -f /var/log/fail2ban.log and hopefully, after a minute or two, you'll start to see something like

2017-06-03 17:04:32,040 fail2ban.actions: WARNING [app-login] Ban 146.185.200.122

That's it! You're safe from baddies (the ones that try to get to that specific page, anyway).

Note - you might also see some SSH bans going on in your fail2ban log. It creates some rules for SSH automatically when it's installed, so don't panic.

Answer 2

I'm glad that you seem to have solved the immediate problem with fail2ban, and it does make sense to block at the iptables level, but you can do the exact same thing in your HAProxy config: You can use acl's with src_http_req_rate() or even src_http_err_rate(Abuse): I've used them in some examples of haproxy configs for DDOS mitigation here.

Longer term: It sounds like you might want to implement a double login for that page, if the users will accept it then you could just put an extra htaccess password in from of the login page, so that they need to separate logins (pretty hard to crack with a script). Or you could use mod_security to do the same thing or some kind of honey trap.