Slumbering apache server refusing connections, but not via localhost

Question

I'm pushing a moderate amount of traffic (about 20 req/s) to a simple PHP API running on an Apache server. The Apache server slumbers (mod_status barely registers one active request), however, I'm seeing "Connection Refused" errors happening periodically despite this. I tried winding up uname -n and MinSpareServers, but to no effect.

I ran a diagnostic script like this while under load, and I'd see about 1 in 10 requests failing with "Connection refused"

while [ 1 ]; do echo "" | telnet <server> 80; sleep 1; done

However, when running a similar request on the local machine, it was fine.

while [ 1 ]; do echo "" | telnet localhost 80; sleep 1; done

Initially I thought it might be that all the requests are coming from the same client, and the server was being throttled. However, while under load, all other hosts that try and connect will see the same intermittent problem.

The fact that localhost behaves differently suggests to me that it's a networking, rather than apache, problem, but I'm not sure where to diagnose this. Is there logging that I can inspect/enable to see if the connection refusal is happening at a O/S (or possibly iptables) level.

Relevant software:

• Ubuntu Lucid, 2.6.33 kernel
• Apache 2.2.14

---

UPDATE:

In our iptables config I can see the following rule under "Chain TCPACCEPT":

655K 34M ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 12/sec burst 24

It seems like this rate limit could definitely be causing some these "Connection Refused" errors. I can disable the firewall for a short time to see if this resolves the issue, but I assume this rule is there to help cope with DOS attacks. The current limit seems too small for both avg and burst. What are more sensible limits (I can adjust these in our iptables config)?

Answer 1

If it's happening periodically, it's most likely not iptables but to view your iptables rules, run

sudo iptables -L -n -v

Your iptables logs will be under /var/log but it can vary depending on your configuration. Check under the /etc/rsyslog.d directory to see if there's a configuration file for your iptables log location.

You could also use netstat or lsof to see what's happening on the port also.

netstat -an | grep "80.*LIST"

lsof -i :80

Finally, you may want to take a network capture to see if there are TCP RSTs being sent back or there's simply no response coming back - use tcpdump for that.

Update to Question Update:

What happens if you disable that iptables rule? It does a couple of things but one of them appears to be restricting tcp segments that only have the syn packet set to 12/sec average with a burst limit. If you're sending an average of 20/sec, I believe that you're exceeding the limits set in the iptables rule.

Therefore, you could either turn off this rule on the firewall to test (I don't recommend turning off your firewall completely) or you could modify it to say 22/sec with a burst of 36.

I would strongly think about modifying configuration that you're not familiar with or are unsure as to why it's there, before modifying.