20

I have the following setup:

(internet) ---> [  pfSense Box  ]    /-> [ Apache / PHP server ]
                [running HAproxy] --+--> [ Apache / PHP server ]
                                    +--> [ Apache / PHP server ]
                                     \-> [ Apache / PHP server ]

For HTTP requests this works great, requests are distributed to my Apache servers just fine. For SSL requests, I had HAproxy distributing the requests using TCP load balancing, and it worked however since HAproxy didn't act as a proxy, it didn't add the X-Forwarded-For HTTP header, and the Apache / PHP servers didn't know the client's real IP address.

So, I added stunnel in front of HAproxy, reading that stunnel could add the X-Forwarded-For HTTP header. However, the package which I could install into pfSense does not add this header... also, this apparently kills my ability to use KeepAlive requests, which I would really like to keep. But the biggest issue which killed that idea was that stunnel converted the HTTPS requests into plain HTTP requests, so PHP didn't know that SSL was enabled and tried to redirect to the SSL site.

How can I use HAproxy to load balance across a number of SSL servers, allowing those servers to both know the client's IP address and know that SSL is in use? And if possible, how can I do it on my pfSense server?

Or should I drop all this and just use nginx?

Community
  • 1
Josh
  • 9,001
  • 27
  • 78
  • 124
  • 3
    Re: stunnel and `X-Forwarded-For`, see [here](http://serverfault.com/questions/301221/haproxy-stunnel-keep-alive). – Shane Madden Aug 17 '11 at 21:40
  • @Shane: Thanks. That's exactly where I read that I lose KeepAlive :-) – Josh Aug 17 '11 at 21:55
  • 2
    +1 for excellent ASCII diagramming. :-) – KyleFarris Jul 17 '13 at 19:25
  • @AlanHamlett, your [link](http://wakatime.com/blog/34-how-to-scale-ssl-with-haproxy-and-nginx) is 404. – luckydonald Nov 11 '16 at 20:30
  • @luckydonald thanks, here is an updated link. You can use Proxy Protocol by adding the send-proxy keyword to your haproxy config. I wrote a blog post with examples here: https://wakatime.com/blog/23-how-to-scale-ssl-with-haproxy-and-nginx – Alan Hamlett Nov 13 '16 at 12:59

6 Answers6

40

Just for the record, as this thread is often referred to concerning HAProxy + SSL, HAProxy does support native SSL on both sides since 1.5-dev12. So having X-Forwarded-For, HTTP keep-alive as well as a header telling the server that the connection was made over SSL is as simple as the following :

listen front
    bind :80
    bind :443 ssl crt /etc/haproxy/haproxy.pem
    mode http
    option http-server-close
    option forwardfor
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    server srv1 1.1.1.1:80 check ...
    ...

I'm sure that by the time you came up with something different, but at least new visitors will get the easy solution now :-)

TheCloudlessSky
  • 215
  • 3
  • 10
Willy Tarreau
  • 3,894
  • 1
  • 19
  • 12
  • Thanks, this is good general info... my question was about HAproxy *running on pfSense* so for now I need to use nginx in front of HAproxy still, as pfSense doesn't support this version of HAProxy (yet) – Josh Sep 16 '12 at 12:44
  • Sorry Josh, I don't know enough about pfSense to know if you can update components on it or not, and since you were speaking about installing a package, I believed it was the case. Last time I tried it was around 5 years ago so I don't remember all the details. – Willy Tarreau Sep 26 '12 at 06:39
  • 1
    I don't understand much about haproxy configuration for now, but with the latest version, I had to add an acl : `acl is-ssl dst_port 443` and rewrite a line : `reqadd X-Forwarded-Proto:\ https if is-ssl` Nginx seems to handle this header fairly well – greg0ire Jan 02 '13 at 17:30
  • This worked like a charm. No nginx required. – Jay Taylor Jul 14 '13 at 22:08
  • 1
    @greg0ire that's because with latest haproxy there is no is_ssl but ssl_fc instead – josch Mar 05 '14 at 09:41
  • 1
    this should be the accepted answer – sgohl Apr 28 '20 at 22:39
15

You dont need to drop it all, you could just use nginx in front of haproxy for SSL support, keeping all your load balancing config. You dont even need to use nginx for HTTP if you don't want to. Nginx can pass both X-Forwarded-For and a custom header indicating SSL is in use (and client cert information if you want). Nginx config snippet that sends required information:

proxy_set_header SCHEME $scheme;      # http/https
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header CLIENT_CERT $ssl_client_raw_cert;
Ochoto
  • 1,174
  • 7
  • 12
12

For anyone else who finds this question, I followed Ochoto's advice and used nginx. Here's the specific steps I used to make this work on my pfSense router:

  1. Using the pfsense web interface, I installed the pfsense PfJailctl package and the "jail_template" package under System > Packages so I could create a FreeBSD jail under which to compile and install nginx on the pfsense system.

  2. I configured a jail for my nginx server under Services > Jails, giving the new jail the same hostname and IP address of the virtual IP alias which I had HAproxy running on. I bound the jail to the WAN interface. I used the default jail template and enabled unionfs rather than nullfs.

  3. Once the jail had been started, I SSHed in to the pfsense box and ran jls to find the jail's number. I then ran jexec 1 sh to get a shell inside the jail. From there I set up BSD ports and installed nginx using:

    portsnap extract
portsnap fetch update
cd /usr/ports/www/nginx
make install clean

  4. I then configured nginx to listen on port 443, and pass all requests to HAproxy on port 80, including the real IP and the SSL status inside HTTP headers. My usr/local/etc/nginx/nginx.conf looks like:

    worker_processes  1;

events {
    worker_connections  2048;
}

http {
    upstream haproxy {
        server 209.59.186.35:80;
    }

    server {
        listen       443;
        server_name  my.host.name default_server;
        ssl                  on;
        ssl_certificate      my.crt;
        ssl_certificate_key  my.key;
        ssl_session_timeout  5m;

        ssl_protocols  SSLv3 TLSv1;
        ssl_ciphers  HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers   on;

        location / {
            proxy_pass http://haproxy;

            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            proxy_set_header X-Forwarded-Proto https;
        }
    }

}

  5. I then modified my PHP application to detect the X-Forwarded-Proto HTTP Header:

    function usingSSL()
{
    return (
       (isset($_SERVER['HTTPS']) && strtolower($_SERVER['HTTPS']) == 'on' )
        || (isset($_SERVER['HTTP_X_FORWARDED_PROTO'])
               && strtolower($_SERVER['HTTP_X_FORWARDED_PROTO']) == 'https' ));
}

So the final setup is:

(internet) ---> [ -> nginx -> haproxy -]--> (pool of apache servers)
                [    (pfSense server)  ]
Josh
  • 9,001
  • 27
  • 78
  • 124
  • 2
    You should disable SSLv2 unless you really need it. http://www.gnu.org/software/gnutls/manual/html_node/On-SSL-2-and-older-protocols.html I don't know why Nginx still supports it in its default config. – Ochoto Aug 18 '11 at 20:20
  • Also realise that with 1024 worker connections you will support at most 512 concurrent clients. – Ochoto Aug 18 '11 at 20:27
  • @Ochoto: Thanks for both of those tips! I'm new to HAproxy but even less familiar with nignx... – Josh Aug 19 '11 at 11:55
7

My configuration for a 1.5-dev-17 version of haproxy:

global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 notice
        #log loghost    local0 info
        maxconn 4096
        #chroot /usr/share/haproxy
        user haproxy
        group haproxy
        daemon
        #debug
        #quiet

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option  http-server-close
        retries 3
        option redispatch
        fullconn 1000        
        maxconn 1000
        timeout queue 600s
        timeout connect 5s
        timeout client 600s
        timeout server 600s

frontend http-in
        bind *:80
        bind *:443 ssl crt /usr/local/etc/ssl/certs
        reqadd X-Forwarded-Proto:\ https if { ssl_fc }
        default_backend varnish-ha
        option forwardfor
backend varnish-ha
  server hafront1 10.1.69.1:6081  minconn 100 maxqueue 10000

It uses the ssl_fc ACL. Please note that the option http-server-close part is very important.

greg0ire
  • 316
  • 1
  • 6
  • 26
5

HAProxy can't hit an SSL backend without using raw TCP mode, losing X-Forwarded-For, but, you could potentially re-encrypt the traffic with a listening stunnel for the backend transit. Ugly, though.

I like Ochoto's approach better, with a caveat: nginx is a perfectly capable load balancer; if you're using it, I'd say use it for everything. Proxy your incoming HTTPS to load balanced HTTPS backends - and that way, there's no need for custom headers for SSL information (unless you do need the client certificate).

Shane Madden
  • 112,982
  • 12
  • 174
  • 248
  • I'm not sure why I'm clinging on to HAproxy. I think it's because pfSense has a package for it, and SOIS uses it. Neither one is a great reason. :-) – Josh Aug 17 '11 at 21:54
  • I digress from nginx beeing a capable load balancer, unless you use the non standard module upstream_fair it does a simple round robin (or client ip hash) without taking into account if destination backend is already busy with requests and thus growing queue in that backend when there are other backends free and waiting for a job. HAProxy also nicely monitor backends and displays stats about them. – Ochoto Aug 18 '11 at 07:18
  • If only one of the following would become true a) Nginx gets decent state tracking and fair load balancing b) HAProxy gets decent SSL support One can only hope – Yavor Shahpasov Aug 19 '11 at 09:29
  • I just deployed a setup using nginx -> haproxy -> nginx -> backend for SSL, this is due to the lack of HTTPS support in haproxy as discussed here, but also because nginx does not support http health check scripts. – Geoffrey Mar 28 '12 at 23:13
2

I implemented a solution last year to integrate HAProxy with pfSense in a way that it harnesses all features of HAProxy and maintains a good isolation with pfSense. So that it is a viable option for production environments. SSL is terminated on HAProxy. I installed HAProxy inside a jail in pfSense using ezjail and Ports Collection. That way it is very easy to maintain both components independently. And you may install whatever version you want. I started with 1.5-dev13. And since then it is working perfectly for me. I have documented the whole thing here.

Installing HAProxy on pfSense

BTW Willy, thanks a lot for such an excellent product.

Dinesh Sharma
  • 21
  • 2