2

In summary, I had a case where all Apache threads were hung because all of them were waiting for a TCP ACK from clients after having sent the HTTP page, and because of that, the Apache threads were waiting 300s (Timeout value of conf) before going to the next request. And then the same thing happened.

The things happened as is: there was a sudden peak of traffic, apache & db servers load goes up as expected, and at some point, every apache thread goes into this state, and the apache & db load goes to 0. And it stayed this way for hours. After a apache restart, pages are processed normally again, load goes up, and again this happen and load goes to 0. Now, if this was an attack or a consequence of some bad soft/hardware is yet to be known.

To the details: When everything is hung, you go see the server-status page of apaches, and all threads are marked as "W" (working), and you see the timer SS (time the request is being processed) going up up to 300s, and then go next to the next HTTP request and start again to 300s.

In the socket part via netstat, we see all the sockets from these Apache thread hung in CLOSE_WAIT with a high Send-Q value (packets sent not acknowledged). Using strace, we indeed see Apache doing a poll() of 300s on the socket, waiting for the packets to be acknowleged.

Now, whether it is an attack or some bad network configuration that lost the packets, my question is: how do we prevent this? It seems to be a particular nasty kind of attack.

I am aware about the slow loris attack, when you make a HTTP request very slowly, and this can be mitigated if you have a CDN, a reverse proxy, ... But for this particular case, I am not seeing something that can prevent that?

How would you prevent this to happen?

Thanks!

pauska
  • 19,532
  • 4
  • 55
  • 75
nand
  • 21
  • 3
  • Were you able to figure out what was causing this? I have a server intermittently doing the exact same thing, with many connections in CLOSE_WAIT and a high Send-Q value. – Eric Mason Feb 07 '15 at 09:09

1 Answers1

0

put a litespeed infront of apache. That can solve your problem. That is exactly what we do for slow request, reflection and regular ddos problems. Solve means at least at some scale i meant.

Edited Section:

Sure,

Litespeed is a web server/load balancer a type of reverse proxy also rate base limiting aviable ddos attack mitigation software. We had the same type of attacks you meantioned and much more worse close to 400.000 http request per second based attacks. We install a litespeed server with 24 core 128 gb ram litespeeds runs in mounted ram and operation system runs in raid0 striped ssd disks. And that solved and stop this attack.

Also via configuration and setup you can stop the slow loris attack. We still have a load balancer, but the incoming traffic passes through first the litespeed then proxeyed to our hw loadbalancer.

x0r
  • 31
  • 3
  • Can you elaborate on your answer? The wording is a bit confusing. – Bart Silverstrim Dec 05 '11 at 11:59
  • Almost any proxy/firewall can prevent slow loris because it will wait for the whole HTTP request before forwarding it to Apache. In this case, when Apache send back the HTTP answer, it may be forwarded by the proxy/firewall, but the proxy/firewall won't "simulate" an ACK for Apache. Apache has to wait for the actual ACK from the client. And if it never comes, 300s timeout. So I'm not sure your solution is adressing this specific point? – nand Dec 05 '11 at 16:56
  • I meant Load balancer working as l7 proxy. Apache's client is only the load balancer. Cause of the proxy only setup can be done on the load balancer no on apaches. You can play around all timeout or protocol specified stuff from load balancer. Load balancer can simulate ack can be work as proxy arp / similar like junipers mib. – x0r Dec 08 '11 at 09:25