7

A client's website is currently under attack, and I've been called in to fix it.

A huge number of IPs (easily over 5,000) is constantly hitting /login, presumably trying to bruteforce their way in.

I've changed the site so the page returns a 500 error, but they're not giving up. Obviously this is no good for the real users, who now can't log in.

Load balancing is done via HAProxy, which I have very little knowledge of (although I have a lot more than I did a few hours ago). I've tried lots of sensible things that I've found online but none seem to have helped, probably because there is such a big block of IPs performing the attack.

Onto the question, then: How can I reject an IP if it hits /login more than X times in Y seconds?

And, subpoint - how can I see a log of rejections so I know it's actually working?

Here is a sample of the haproxy.log:

Jun  3 14:24:50 hap-server haproxy[11831]: 46.161.62.79:15290 [03/Jun/2017:14:24:49.505] www-https-test~ www-backend/www-03 751/0/202/38/991 500 220 - - ---- 428/428/120/38/0 0/0 "GET /login HTTP/1.1"
Jun  3 14:24:50 hap-server haproxy[11831]: 46.161.63.132:47804 [03/Jun/2017:14:24:49.505] www-https-test~ www-backend/www-04 751/0/202/38/991 500 220 - - ---- 428/428/119/42/0 0/0 "GET /login HTTP/1.1"
Jun  3 14:24:50 hap-server haproxy[11831]: 46.161.62.43:53176 [03/Jun/2017:14:24:49.505] www-https-test~ www-backend/www-02 751/0/202/38/991 500 220 - - ---- 428/428/118/38/0 0/0 "GET /login HTTP/1.1"
Thomas
  • 4,155
  • 5
  • 21
  • 28
Grim...
  • 351
  • 1
  • 2
  • 9
  • fail2ban likely has an haproxy module that you can use. – EEAA Jun 03 '17 at 14:45
  • That's cool, but I don't have the first clue how to use fail2ban either :/ – Grim... Jun 03 '17 at 15:00
  • That's alright. It's a great chance for you to dig in and learn. We don't do tutorials here, if that's what you were looking for. – EEAA Jun 03 '17 at 15:01
  • I mean, I had other plans for my Saturday. This started five hours ago. – Grim... Jun 03 '17 at 15:09
  • 1
    I can sympathize with you. Truly. If you aren't able to dig into things like this, perhaps it's time to consider hiring an expert sysadmin that can look after things. – EEAA Jun 03 '17 at 15:48
  • fail2ban, sadly, has no haproxy module out of the box. You must be able to do it somehow because of questions like this https://stackoverflow.com/questions/28912845/fail2ban-with-haproxy-logs-how-to-to-block but fruitless searching for any sort of less detailed guidance basically gets you "RTFM" or descriptions of how to block Wordpress logins. Or this question, now ;-) – Grim... Jun 03 '17 at 16:00
  • 1
    This article has some great info about tracking hits to specific pages and putting them into a tracking table ready for denying. https://www.haproxy.com/blog/introduction-to-haproxy-stick-tables/ But you must make sure that your haproxy can handle the botnet traffic. If its struggling, then the banning of IPs in the firewall is a better solution using Fail2Ban – blissweb Jan 25 '22 at 07:22

2 Answers2

17

Thanks to a comment by EEAA, I was able to solve this using fail2ban.

There's very little documentation about how to use fail2ban with HAProxy, however - so little in fact that this page is already nearing the top of a Google search for "haproxy fail2ban", so I'll detail how I did it.

First of all, install fail2ban. If you can't do that bit you probably shouldn't carry on without some more help.

fail2ban works by scanning your access logs, looking for a pattern you set up. If it finds that pattern X times in Y seconds, it will automatically ban that IP for Z seconds.

Your HAProxy log should be at /var/log/haproxy.log. If you're under heavy load it might be too big to open with Vim or Nano, so just look at the last few lines by using tail: tail -n50 /var/log/haproxy.log

A sample of a "bad" entry in my log looked like this:

Jun  3 16:48:03 hap-server haproxy[21751]: 178.159.100.29:48806 [03/Jun/2017:16:48:03.735] www-https-test~ www-backend/www-04 172/0/2/3/177 200 339 - - ---- 36/36/0/0/0 0/0 "GET /login HTTP/1.1"

The important bits we want fail2ban to get hold of are the IP of the attacker and the page they are hitting.

To tell fail2ban how to do this, first we want to create a filter. Create a file in the folder /etc/fail2ban/filter.d. I called mine applogin.conf but you can call it what you like, as long as it ends in .conf.

The contents were as follows:

[Definition]

failregex = ^.*haproxy\[[0-9]+\]: <HOST>:.* "(GET |POST )/login HTTP/1.1"$
ignoreregex =

<HOST> is the point in the line from your log where the IP address appears. If you want to use the same regex as me, replace /login with the address the attackers are targeting on your server.

Now you have to tell fail2ban that you want it to look for that filter. Make a copy of /etc/fail2ban/jail.conf (in case you mess up and need to start again), then open jail.conf and add the following lines at the bottom.

[app-login]
enabled  = true
bantime  = 1200
findtime = 120
maxretry = 6
filter   = applogin
logpath  = /var/log/haproxy.log
port     = http,https

The part in the square brackets is just a name - use what you like. bantime is the number of seconds a user is banned for. maxretry is the amount of times a user can hit the page in findtime seconds. So, in my example, if someone tries to get to www.mydomain.com/login more than six times (or maybe exactly six times, I'm not sure) in 2 minutes then they will be banned for 20 minutes. filter is the name of the file you created in /etc/fail2ban/filter.d (but without the .conf). logpath is the path to the log file you want it to search in.

Save that file, then restart fail2ban: service fail2ban restart Tail the fail2ban logfile: tail -f /var/log/fail2ban.log and hopefully, after a minute or two, you'll start to see something like

2017-06-03 17:04:32,040 fail2ban.actions: WARNING [app-login] Ban 146.185.200.122

That's it! You're safe from baddies (the ones that try to get to that specific page, anyway).

Note - you might also see some SSH bans going on in your fail2ban log. It creates some rules for SSH automatically when it's installed, so don't panic.

Grim...
  • 351
  • 1
  • 2
  • 9
7

I'm glad that you seem to have solved the immediate problem with fail2ban, and it does make sense to block at the iptables level, but you can do the exact same thing in your HAProxy config: You can use acl's with src_http_req_rate() or even src_http_err_rate(Abuse): I've used them in some examples of haproxy configs for DDOS mitigation here.

Longer term: It sounds like you might want to implement a double login for that page, if the users will accept it then you could just put an extra htaccess password in from of the login page, so that they need to separate logins (pretty hard to crack with a script). Or you could use mod_security to do the same thing or some kind of honey trap.

  • I like the idea, and have already setup quite a few rules in haproxy but denying the http request does not stop the traffic to Haproxy. Today our haproxy was ground to a halt by attackers, so I think I will try the Fail2Ban linked to FirewallD blocking technique next. – blissweb Jan 25 '22 at 07:18