I'm pushing a moderate amount of traffic (about 20 req/s) to a simple PHP API running on an Apache server. The Apache server slumbers (mod_status barely registers one active request), however, I'm seeing "Connection Refused" errors happening periodically despite this. I tried winding up uname -n
and MinSpareServers
, but to no effect.
I ran a diagnostic script like this while under load, and I'd see about 1 in 10 requests failing with "Connection refused"
while [ 1 ]; do echo "" | telnet <server> 80; sleep 1; done
However, when running a similar request on the local machine, it was fine.
while [ 1 ]; do echo "" | telnet localhost 80; sleep 1; done
Initially I thought it might be that all the requests are coming from the same client, and the server was being throttled. However, while under load, all other hosts that try and connect will see the same intermittent problem.
The fact that localhost behaves differently suggests to me that it's a networking, rather than apache, problem, but I'm not sure where to diagnose this. Is there logging that I can inspect/enable to see if the connection refusal is happening at a O/S (or possibly iptables) level.
Relevant software:
- Ubuntu Lucid, 2.6.33 kernel
- Apache 2.2.14
UPDATE:
In our iptables config I can see the following rule under "Chain TCPACCEPT":
655K 34M ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 12/sec burst 24
It seems like this rate limit could definitely be causing some these "Connection Refused" errors. I can disable the firewall for a short time to see if this resolves the issue, but I assume this rule is there to help cope with DOS attacks. The current limit seems too small for both avg and burst. What are more sensible limits (I can adjust these in our iptables config)?