HAProxy returns Bad Request (Invalid Host) for seemingly no reason

Question

I've been trying to test a setup that looks like this:

Website: Http GET Request -> Nginx -> HAProxy -> .Net application

I've placed Nginx and HAProxy on the same Debian machine. However, HAProxy constantly returns "Bad Request (Invalid Host)". I've determined that HAProxy is at fault by first sending the request directly from the website to the .Net application, which works. Having Nginx link to the .Net application also works. However, when I try to test this with the HAProxy in place, I start getting the error. It doesn't matter whether the .Net application is actually running behind HAProxy, I always get the error.

An example of the message I'm trying to send is http://192.168.7.119:81/a:diff1. The expected reply is a JPG image. This seems to work fine when sent to anything other than HAProxy (Apache, Nginx, the application) but HAProxy just says "Bad Request". Strangely enough, I'm not seeing any errors in my log file.

Here's an example from the log file:

Feb  2 15:10:08 companyDebian haproxy[5566]: 192.168.7.114:51105 [02/Feb/2015:15:10:08.415] renderfarm renderfarm/renderA 0/0/0/1/1 400 212 - - ---- 1/1/0/1/0 0/0 "GET /a:diff1 HTTP/1.1"
Feb  2 15:10:08 companyDebian haproxy[5566]: 192.168.7.114:51105 [02/Feb/2015:15:10:08.417] renderfarm renderfarm/renderA 73/0/0/4/77 400 212 - - ---- 1/1/0/1/0 0/0 "GET /favicon.ico HTTP/1.1"

My config file looks like this:

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL).
    ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
    ssl-default-bind-options no-sslv3

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

listen renderfarm
    bind 192.168.7.119:81
    mode http
    balance roundrobin     
    server renderA 192.168.7.114:40000 maxconn 1

I have no prior experience with HAProxy and am using it because it was recommended to me. As such, I don't know what other steps I can take to fix this problem. The config manual mentions various options you can set, such as tune.bufsize and option accept-invalid-http-request but these have no effect.

Note: The idea is to add more servers running the application once this setup works. Each server really can only process 1 request at a time.

Answer 1

I faced the same issue and took me a while to figure out why this was happening.

My environment is formed by 1 HAproxy and 2 nginx as a backend with nodejs as a CGI-like.

The root cause is based on how HAproxy builds the HTTP request. By default HAproxy would not include host header on the request, so you need to added manually, otherwise nginx will return 400 as default and HAproxy will mark it as unhealthy.

Example Below:

• HAproxy health check conf:

  option httpchk HEAD / HTTP/1.1
  

• Equivalent HTTP request:

  tony@calderona:~ $ telnet A.B.C.D 80
  Trying A.B.C.D...
  Connected to A.B.C.D.
  Escape character is '^]'.
  HEAD / HTTP/1.1
  
  HTTP/1.1 400 Bad Request
  Server: nginx/1.10.3
  Date: Sun, 10 Dec 2017 09:52:12 GMT
  Content-Type: text/html
  Content-Length: 173
  Connection: close 
  

• HAproxy health check conf:

  option httpchk HEAD / HTTP/1.1\r\nHost:\ www.temporaryworkaround.org
  

• Equivalent HTTP request:

  tony@calderona:~ $ telnet A.B.C.D 80
  Trying A.B.C.D...
  Connected to A.B.C.D.
  Escape character is '^]'.
  HEAD / HTTP/1.1
  host: www.temporaryworkaround.org
  
  HTTP/1.1 200 OK
  Server: nginx/1.10.3
  Date: Sun, 10 Dec 2017 09:52:24 GMT
  Content-Type: text/html; charset=utf-8
  Content-Length: 10282
  Connection: keep-alive
  

Cheers

Answer 2

If you want to see the exact error HAproxy runs into you can do so using socat to connect to the admin socket. Install socat via apt-get install socat, then run the following:

echo "show errors" | socat unix-connect:/run/haproxy/admin.sock stdio

If you run this right after getting a "Bad Request" error it should show you exactly what HAproxy didn't like about the HTTP request the client made.

NOTE: The above works only when you have enabled Unix Socket commands for HAProxy. You will have to add a one line configuration under global section to enable this.

global
     stats socket /var/run/haproxy.sock mode 600 level admin

Official documentation

Answer 3

It might be worthwhile trying to change your HAProxy configuration to TCP instead of HTTP. You lose some advanced load balancing options that HTTP provides (setting cookies for persistence, etc), but it doesn't look like you're using them right now anyway.

Here's an example config that I use with HAProxy for MySQL load balancing:

listen mysql-failover
    bind 10.10.10.210:3306
    mode tcp
    option  tcplog
    option mysql-check user haproxy
    server db1 10.10.10.13:3306 check fall 2 inter 1000
    server backup 10.10.10.11:3306 check

Note that if you change to TCP, you'll lose access to the built-in HTTP stats page, which you can define in a separate listen statement. For example:

listen stats
    bind 10.10.10.200:80
    mode http
    stats enable
    stats hide-version
    stats uri /
    option httpclose
    stats auth username:password

Answer 4

Putting HAProxy behind Nginx seems frivolous as both Nginx or HAProxy can singlehandedly provide the functionality you seem to require.

Consider splitting the listen section into frontend and backend sections with host or path_beg acls and simplify the configuration until you get a response. Also consider either solely using forwardfor with one server or adding another server for roundrobin. Some examples-

frontend http-in
     bind 0.0.0.0:80
     acl host_1 hdr(host) -i firstsite.com
     acl host_nginx hdr(host) -i secondsite.com

Pairing front and backends-

use_backend firstsite_redirect if host_1
use_backend nginx if host_nginx

Backend configuration-

backend firstsite_redirect
    mode http
    option forwardfor
    server firstserver 192.168.0.2:8080

backend nginx
    mode http
    balance roundrobin
    option httpclose
    option forwardfor
    cookie SRVNAME insert
    server nginx01 192.168.0.3:8080 cookie s1 check
    server nginx02 192.168.0.4:8080 cookie s2 check

If you want to redirect based on URL, consider using path_beg in the frontend config instead-

frontend http-in 0.0.0.0:80
    acl docs path_beg /docs
    acl phpmyadmin path_beg /phpmyadmin