1

I see tons of request errors on one of my frontend proxies (on the order of a few per second), but I can't figure out what's causing them. I've tried using the "show errors" command on the stats socket...

echo "show errors" | socat unix-connect:/var/run/haproxy.stat stdio

But this returns nothing. Debug logging doesn't give me any hints either. Is there some other place I should be looking?

Edit: Just to clarify, there is no "error message" per se (though it would sure help to have one). I'm just looking at the counter labeled "request errors" in the web interface and the socat output, and I'm trying to figure out what's incrementing it.

Mike Conigliaro
  • 3,105
  • 2
  • 24
  • 24
  • What is the exact error??? i might have debugged this already – Arenstar Nov 16 '10 at 03:21
  • I'd be curious to know what version you are using. The server I haduse haproxy on had a hard drive failure yesterday, so I just installed the latest version and I am starting to see these req errors and gateway timeout errors that I never had before with an older version. At any given point I only see a max of 400 to 600 active connections. –  Nov 16 '10 at 05:25
  • This is the latest Ubuntu package on Lucid (1.3.22-1) – Mike Conigliaro Nov 16 '10 at 22:22

2 Answers2

1

Have you looked at dmesg? A common problem with Proxy servers is to hit the max Linux connection tracking since each request is using to connections. If this is the case you will see ip_conntrack: table full, dropping packet. in dmesg. You can see the current count and raise it via sysctl or proc:

[kbrandt@lb01: ~] cat /proc/sys/net/netfilter/nf_conntrack_max
131072
[kbrandt@lb01: ~] cat /proc/sys/net/netfilter/nf_conntrack_count
185

You can also bypass connection tracking with the NOTRACK target, i.e.:

sudo /sbin/iptables -t raw -A PREROUTING -p tcp --sport 80 -j NOTRACK

Keep in mind that it is a security risk to disable tracking though, you don't want to do it unless you are already behind a stateful firewall.

Can you post the errors you are seeing?

Kyle Brandt
  • 82,107
  • 71
  • 302
  • 444
  • The only error I see in dmesg is: "possible SYN flooding on port 80. Sending cookies." Also the only thing listed under /proc/sys/net/netfilter/ is nf_log (no conntrack stuff). – Mike Conigliaro Nov 16 '10 at 22:24
1

I suspect Michael is only seeing them on the stats web page. Most likely those errors are simply due to port scans. If your haproxy configuration has "option dontlognull", you won't get any log for them, but still they are correctly accounted for. You'd see them in the "show errors" if they were invalid HTTP request and this does not seem to be the case. Don't worry for a few per second, it's almost nothing once you've figured that they are just port scans. Some heavy sites are getting between hundreds and thousands a second !

@Tim: recent versions have fixed an issue that older ones had which caused some of these errors not to be reported (they were incidently accounted in the backend where they're not retrieved). I'm sure it's just that.

  • I'm actually seeing these errors in both the stats page and the socat "show errors" output. Also, I didn't have "option dontlognull" enabled originally, but I did enable it thinking that it might make the error counter stop increasing. I only posted the original question here once I saw that this option had no affect on the error counter. – Mike Conigliaro Nov 16 '10 at 22:29