I'm in the process of setting up a series of servers as proxies to our application server, primarily to obscure the application server's IP as part of a suite of anti-DDOS measures.
My goal is to have HAProxy accept short-lived HTTP/1.0 connections and forward them to the backend over persistent connections, so as to significantly cut down on the overhead lost to the connection process, slow start, etc.
I have this configuration partly working, but am still seeing a much higher connection rate to the backend than expected - on investigation, it appears that each backend connection is serving between 1 and 5 requests, then being reset by the HAProxy side.
These reset packets always happen after a response is fully received from the server, before any further request is sent, and they happen after a seemingly random delay - sometimes as short as 0.1s, sometimes as long as 20s. Changing HAProxy's timeout settings seems to have no effect.
Does anyone know what would cause HAProxy to exhibit this behaviour, or how I might better go about debugging it?
Edit: Forgot to mention, I was unable to get the backends to generate any meaningful logs, even using tcplog
Configuration (redacted):
global
log /dev/log local0
log /dev/log local1 notice
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
maxconn 16384
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
option http-keep-alive
timeout connect 5s
timeout client 50s
timeout server 50s
timeout http-keep-alive 50s
timeout http-request 30s
maxconn 4000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
# Temporary setting, egress is fragile w/ high rate of connections
default-server maxconn 200
frontend fe-web
bind 0.0.0.0:80
acl cloudflare src -f /etc/haproxy/cloudflare_ips
block if !cloudflare
use_backend be-web
frontend fe-legacy
bind 0.0.0.0:29080
option forwardfor header CF-Connecting-IP
use_backend be-legacy
backend be-web
server srv-web 46.105.16.9:34020 check
option httpchk HEAD / HTTP/1.1\r\nHost:\ legacy-domain
backend be-legacy
server srv-legacy 46.105.16.9:34020 check
option httpchk HEAD / HTTP/1.1\r\nHost:\ legacy-domain
