Blocking slowloris using fail2ban, what are the correct parameters?

Question

I have a fail2ban rule for Apache2 logs that which looks like this:

[Definition]
failregex = ^[^ ]+ <HOST> .* \[\] "[^\"]*" 408 \d+

This will detect the 408 errors which happen when a TCP connection times out.

The following is the Jail definition that uses the definition above:

[snap-apache-timeout]
enabled  = true
filter   = snap-apache-timeout-filter
action   = snap-firewall-action[scheme=all,period=year,reason=fail2ban: apache timeout]
logpath  = %(apache_access_log)s
maxretry = 35
findtime = 3600

As we can see, I set this up as a max-retry of 35 within one hour. In other words, if I get 35+ HTTP 408 errors within one hour, I run the corresponding action which is to block the user's IP address.

Only, I get some of my customers blocked once in a while. I am thinking this in part happens because some closes their browser while some connections were still open in the background (i.e. the user clicks to go to another page and does not wait for it to be loaded but instead just closes the page.)

What do you do against Slowloris? Nothing?

The more I look at it and the more I'm thinking this is not very doable with just the Apache2 408 error (i.e. that's not the same as detecting a really slow connection! Just that the connection is lost once in a while...)

There are logs between two 408 for a given IP address which causes problems. Unfortunately Apache does not show which connection times out. Maybe it's possible to add that to the logs?

exdox.com:443 a.b.c.d - - [07/Apr/2017:15:08:54 +0000] "-" 408 5845 "-" "-"
exdox.com:443 a.b.c.d - - [07/Apr/2017:15:08:39 +0000] "GET /finball/location/exdox_snap5-204/data HTTP/1.1" 200 2569 "exdox.com" "https://exdox.com/dashboard" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" TLSv1.2 ECDHE-RSA-AES256-SHA384
exdox.com:443 a.b.c.d - - [07/Apr/2017:15:09:03 +0000] "GET /finball/location/exdox_snap5-204/logo.png/icon-77x77.png?fallback=ok HTTP/1.1" 200 10883 "exdox.com" "https://exdox.com/finball/location/exdox_snap5-204/data" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" TLSv1.2 ECDHE-RSA-AES256-SHA384
exdox.com:443 a.b.c.d - - [07/Apr/2017:15:09:05 +0000] "GET /favicon.ico HTTP/1.1" 200 1406 "exdox.com" "-" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" TLSv1.2 ECDHE-RSA-AES256-SHA384
exdox.com:443 a.b.c.d - - [07/Apr/2017:15:09:06 +0000] "GET /finball/location/exdox_snap5-204/data/create/20170406 HTTP/1.1" 303 238 "exdox.com" "https://exdox.com/finball/location/exdox_snap5-204/data" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" TLSv1.2 ECDHE-RSA-AES256-SHA384
exdox.com:443 a.b.c.d - - [07/Apr/2017:15:09:06 +0000] "GET /finball/location/exdox_snap5-204/data/create/20170406 HTTP/1.1" 303 238 "exdox.com" "https://exdox.com/finball/location/exdox_snap5-204/data" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" TLSv1.2 ECDHE-RSA-AES256-SHA384
exdox.com:443 a.b.c.d - - [07/Apr/2017:15:09:11 +0000] "GET /finball/location/exdox_snap5-204/data/create/20170405 HTTP/1.1" 303 237 "exdox.com" "https://exdox.com/finball/location/exdox_snap5-204/data" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" TLSv1.2 ECDHE-RSA-AES256-SHA384
exdox.com:443 a.b.c.d - - [07/Apr/2017:15:09:11 +0000] "GET /finball/location/exdox_snap5-204/data/create/20170405 HTTP/1.1" 303 237 "exdox.com" "https://exdox.com/finball/location/exdox_snap5-204/data" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" TLSv1.2 ECDHE-RSA-AES256-SHA384
exdox.com:443 a.b.c.d - - [07/Apr/2017:15:09:06 +0000] "GET /finball/location/exdox_snap5-204/data/20170406?a=edit HTTP/1.1" 200 3521 "exdox.com" "https://exdox.com/finball/location/exdox_snap5-204/data" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" TLSv1.2 ECDHE-RSA-AES256-SHA384
exdox.com:443 a.b.c.d - - [07/Apr/2017:15:09:12 +0000] "GET /finball/location/exdox_snap5-204/data/20170405?a=edit HTTP/1.1" 200 6516 "exdox.com" "https://exdox.com/finball/location/exdox_snap5-204/data" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" TLSv1.2 ECDHE-RSA-AES256-SHA384
exdox.com:443 a.b.c.d - - [07/Apr/2017:15:09:16 +0000] "GET /finball/location/exdox_snap5-204/logo.png/icon-77x77.png?fallback=ok HTTP/1.1" 200 10883 "exdox.com" "https://exdox.com/finball/location/exdox_snap5-204/data/20170405?a=edit" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" TLSv1.2 ECDHE-RSA-AES256-SHA384
exdox.com:443 a.b.c.d - - [07/Apr/2017:15:09:17 +0000] "GET /favicon.ico HTTP/1.1" 200 1406 "exdox.com" "-" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" TLSv1.2 ECDHE-RSA-AES256-SHA384
exdox.com:443 a.b.c.d - - [07/Apr/2017:15:09:56 +0000] "GET /finball/location/exdox_snap5-204/data HTTP/1.1" 200 2569 "exdox.com" "https://exdox.com/finball/location/exdox_snap5-204/data/20170405?a=edit" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" TLSv1.2 ECDHE-RSA-AES256-SHA384
exdox.com:443 a.b.c.d - - [07/Apr/2017:15:10:01 +0000] "GET /finball/location/exdox_snap5-204/logo.png/icon-77x77.png?fallback=ok HTTP/1.1" 200 10883 "exdox.com" "https://exdox.com/finball/location/exdox_snap5-204/data" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" TLSv1.2 ECDHE-RSA-AES256-SHA384
exdox.com:443 a.b.c.d - - [07/Apr/2017:15:10:01 +0000] "GET /favicon.ico HTTP/1.1" 200 1406 "exdox.com" "-" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" TLSv1.2 ECDHE-RSA-AES256-SHA384
exdox.com:443 a.b.c.d - - [07/Apr/2017:15:10:21 +0000] "-" 408 193 "-" "-"

As per @symcbean's answer below, the page mentions the use of the reqtimeout Apache2 module. I have that installed with the default settings:

RequestReadTimeout header=20-40,minrate=500
RequestReadTimeout body=10,minrate=500

I do not use the qos or security modules as mentioned on that page. Instead, I have fail2ban to check the logs for 408 errors as shown above. I don't think that's any different from the security setup shown on that page except that one says "if more than 5 in the last minute".

nginx is what everyone does to mitigate against this attack. slowloris is just an apache vuln, and for some reason no one has written a patch. — rook, Apr 07 '17 at 22:51
Do you have anything, like a link to some page, to back this up? (not that I'm questioning about the Apache vulnerability, I can see it's there...) — Alexis Wilke, Apr 07 '17 at 23:12
@AlexisWilke nginx is still technically vulnerable to slowloris (or at least slowloris-type attacks), but it handles it much better than Apache does. [This](http://serverfault.com/a/281595/240570) ServerFault answer explains it pretty well. Both Apache and nginx can be configured to handle these attacks better, but out of the box, both are vulnerable. — mattrick, Apr 08 '17 at 04:36
@mattrick nginx will not see any degradation of service when running the slowloris attack, I encourage you to test this first hand before making blanket statements. Just run the old perl script on a laptop, and you'll be able to take down any Apache system on the net. — rook, Apr 10 '17 at 21:47
@rook see: https://github.com/valyala/goloris. Its specifically designed for nginx. Like I said, "slowloris-type attacks". — mattrick, Apr 12 '17 at 01:57
@mattrick Goloris seems to say that Nginx is not vulnerable since 1.5.10. Are you saying that the server was updated to at least partially prevent the problem? On my end, I'm also wondering whether a similar slowness could be detected from regular customers and thus I could end up blocking such instead of just real slowloris connections. — Alexis Wilke, Apr 12 '17 at 16:42

symcbean · Answer 1 · 2017-04-07T23:35:30.293

1

Your config looks fine as long as you have taken appropriate steps elsewhere to drop the connection and return a 408 error when the webserver receives something which looks like a slowloris attack.

(Note that just reducing the timeout from the default 60 seconds to, say, 3, will also increase the ability to detect attacks without installing additional modules)

The scenario of someone closing a browser before the response is unlikely to cause a 408 error unless there's something very odd on your local network.

Really you should be looking at the logs leading up to a response to see if your server really is returning a 408 error; a better pattern might be " 408

edited Apr 07 '17 at 23:35

answered Apr 07 '17 at 23:29

symcbean

18,278
39
73

I have the 408 errors in the logs and I know it is caught properly by my `fail2ban` setup. The block uses the `snapfirewall` tool, though, instead of the default `fail2ban` action. The result is very much the same, the IP appears in the firewall and the clients is 100% locked out of all our sever within 1 or 2 seconds. (my `snapfirewall` is capable of propagating the block to all servers, `fail2ban` is very local by default!) – Alexis Wilke Apr 08 '17 at 04:19
I think that my problem is that I get a 408 in many different cases, such as while I receive one character a second instead of 64kb/s or whether the connection somehow dies in the middle of it and not because the sender is slow. – Alexis Wilke Apr 08 '17 at 05:04

Blocking slowloris using fail2ban, what are the correct parameters?

1 Answers1