I have an API that sends messages to RabbitMQ.
I have a high availability RabbitMQ cluster behind HAProxy.
When I load test my API I start seeing lots of this:
Recovering from a network failure...
Exception in the reader loop: AMQ::Protocol::EmptyResponseError: Empty response received from the server.
In my unicorn log.
If I connect directly to RabbitMQ as a posed to via haproxy I do not. Where am I going wrong, my haproxy config looks like this:
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
#log loghost local0 info
maxconn 4096
#debug
#quiet
user haproxy
group haproxy
defaults
log global
mode http
retries 3
timeout client 50s
timeout connect 10s
timeout server 50s
option dontlognull
option forwardfor
option httplog
option redispatch
balance roundrobin
# Set up application listeners here.
listen http_frontend
bind *:80
mode http
default_backend http_backend
option httpclose
reqadd X-Forwarded-Proto:\ http
listen https_frontend
bind *:443 ssl crt /etc/haproxy.pem
mode http
default_backend http_backend
reqadd X-Forwarded-Proto:\ https
listen http_bucky_frontend
bind *:1880
mode http
default_backend http_bucky_backend
option httpclose
reqadd X-Forwarded-Proto:\ http
listen https_bucky_frontend
bind *:1443 ssl crt /etc/haproxy.pem
mode http
default_backend http_bucky_backend
reqadd X-Forwarded-Proto:\ https
listen rabbitmq_frontend
bind *:5672
mode tcp
default_backend rabbitmq_backend
option tcplog
listen admin
bind 127.0.0.1:22002
mode http
stats uri /
backend http_backend
mode http
server 0-http_backend x.x.x.x:9000 maxconn 100 check
server 1-http_backend x.x.x.x:9000 maxconn 100 check
backend http_bucky_backend
mode http
option httpchk GET /status
http-check expect string up
server 0-http_bucky_backend x.x.x.x:9000 maxconn 100 check
server 1-http_bucky_backend x.x.x.x:9000 maxconn 100 check
backend rabbitmq_backend
balance roundrobin
mode tcp
server 0-rabbitmq_backend x.x.x.x:5672 maxconn 4000 check
server 1-rabbitmq_backend x.x.x.x:5672 maxconn 4000 check
When under load the load balancer is normally 20-30% cpu