2

I'm having the darndest time trying to make a simple reverse proxy select the backend based on the URL.

I keep getting a 503 Service Unavailable. No server is available to handle this request. error when I try this.

Any help would be appreciated!

I'd like to enter nc.mydomain.com and forward my request to my nextcloud server (I'll add other subdomains and domains later). Here is my config:

haproxy.cfg:

global
    log /dev/log    local0
    log /dev/log    local1 notice
    daemon
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3 no-tlsv10

defaults
    mode http
    log global
    option  httplog
    option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http
    option forwardfor if-none

frontend for-letsencrypt
    bind :80
    acl isletsencrypt path_beg /.well-known/acme-challenge/
    use_backend letsencrypt-backend if isletsencrypt

backend letsencrypt-backend
    server letsencrypt 127.0.0.1:8888

frontend public-web-in
    mode http
    bind :80
    bind :443 ssl crt /etc/letsencrypt/live/nc.mydomain.com/nc-haproxy.pem alpn h2,http/1.1
    redirect scheme https if !{ ssl_fc }
    acl isnextcloud req.hdr(Host) -i -m str nc.mydomain.com
    use_backend ncserver if isnextcloud
    default_backend myrouter

backend ncserver
    description My Nextcloud server
    server nc1 192.168.1.28:80 send-proxy ssl verify none

backend myrouter
    description My default backend server
    server router 192.168.1.1:80 send-proxy

I set the logging level to debug and this is from /var/log/haproxy.log:

Jan 31 10:36:58 haproxy haproxy[3706]: [WARNING] 030/103658 (3706) : Exiting Master process...
Jan 31 10:36:58 haproxy haproxy[3706]: [ALERT] 030/103658 (3706) : Current worker #1 (3707) exited with code 143 (Terminated)
Jan 31 10:36:58 haproxy haproxy[3706]: [WARNING] 030/103658 (3706) : All workers exited. Exiting... (143)
Jan 31 10:36:58 haproxy haproxy[4964]: [WARNING] 030/103658 (4964) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.
Jan 31 10:36:58 haproxy haproxy[4964]: Proxy for-letsencrypt started.
Jan 31 10:36:58 haproxy haproxy[4964]: Proxy for-letsencrypt started.
Jan 31 10:36:58 haproxy haproxy[4964]: [NOTICE] 030/103658 (4964) : New worker #1 (4965) forked
Jan 31 10:36:58 haproxy haproxy[4964]: Proxy letsencrypt-backend started.
Jan 31 10:36:58 haproxy haproxy[4964]: Proxy letsencrypt-backend started.
Jan 31 10:36:58 haproxy haproxy[4964]: Proxy public-web-in started.
Jan 31 10:36:58 haproxy haproxy[4964]: Proxy public-web-in started.
Jan 31 10:36:58 haproxy haproxy[4964]: Proxy ncserver started.
Jan 31 10:36:58 haproxy haproxy[4964]: Proxy ncserver started.
Jan 31 10:36:58 haproxy haproxy[4964]: Proxy myrouter started.
Jan 31 10:36:58 haproxy haproxy[4964]: Proxy myrouter started.
Jan 31 10:37:42 haproxy haproxy[4965]: 123.45.67.890:45204 [31/Jan/2019:10:37:39.272] public-web-in~ ncserver/nc1 0/0/-1/-1/3007 503 213 - - SC-- 1/1/0/0/3 0/0 "GET / HTTP/1.1"
Jan 31 10:37:42 haproxy haproxy[4965]: 123.45.67.890:45204 [31/Jan/2019:10:37:39.272] public-web-in~ ncserver/nc1 0/0/-1/-1/3007 503 213 - - SC-- 1/1/0/0/3 0/0 "GET / HTTP/1.1"
Jan 31 10:37:45 haproxy haproxy[4965]: 123.45.67.890:45204 [31/Jan/2019:10:37:42.764] public-web-in~ ncserver/nc1 0/0/-1/-1/3005 503 213 - - SC-- 1/1/0/0/3 0/0 "GET /favicon.ico HTTP/1.1"
Jan 31 10:37:45 haproxy haproxy[4965]: 123.45.67.890:45204 [31/Jan/2019:10:37:42.764] public-web-in~ ncserver/nc1 0/0/-1/-1/3005 503 213 - - SC-- 1/1/0/0/3 0/0 "GET /favicon.ico HTTP/1.1"

Jan 31 10:38:42 haproxy haproxy[4964]: [WARNING] 030/103842 (4964) : Exiting Master process...
Jan 31 10:38:42 haproxy haproxy[4964]: [ALERT] 030/103842 (4964) : Current worker #1 (4965) exited with code 143 (Terminated)
Jan 31 10:38:42 haproxy haproxy[4964]: [WARNING] 030/103842 (4964) : All workers exited. Exiting... (143)
Jan 31 10:38:42 haproxy haproxy[4977]: [WARNING] 030/103842 (4977) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.
Jan 31 10:38:42 haproxy haproxy[4977]: Proxy for-letsencrypt started.
Jan 31 10:38:42 haproxy haproxy[4977]: Proxy for-letsencrypt started.
Jan 31 10:38:42 haproxy haproxy[4977]: [NOTICE] 030/103842 (4977) : New worker #1 (4978) forked
Jan 31 10:38:42 haproxy haproxy[4977]: Proxy letsencrypt-backend started.
Jan 31 10:38:42 haproxy haproxy[4977]: Proxy letsencrypt-backend started.
Jan 31 10:38:42 haproxy haproxy[4977]: Proxy public-web-in started.
Jan 31 10:38:42 haproxy haproxy[4977]: Proxy public-web-in started.
Jan 31 10:38:42 haproxy haproxy[4977]: Proxy ncserver started.
Jan 31 10:38:42 haproxy haproxy[4977]: Proxy ncserver started.
Jan 31 10:38:42 haproxy haproxy[4977]: Proxy myrouter started.
Jan 31 10:38:42 haproxy haproxy[4977]: Proxy myrouter started.
Jan 31 10:38:50 haproxy haproxy[4978]: 123.45.67.890:46022 [31/Jan/2019:10:38:47.493] public-web-in~ ncserver/nc1 0/0/-1/-1/3006 503 213 - - SC-- 2/2/0/0/3 0/0 "GET / HTTP/1.1"
Jan 31 10:38:50 haproxy haproxy[4978]: 123.45.67.890:46022 [31/Jan/2019:10:38:47.493] public-web-in~ ncserver/nc1 0/0/-1/-1/3006 503 213 - - SC-- 2/2/0/0/3 0/0 "GET / HTTP/1.1"
Jan 31 10:38:53 haproxy haproxy[4978]: 123.45.67.890:46022 [31/Jan/2019:10:38:50.563] public-web-in~ ncserver/nc1 0/0/-1/-1/3005 503 213 - - SC-- 2/2/0/0/3 0/0 "GET /favicon.ico HTTP/1.1"
Jan 31 10:38:53 haproxy haproxy[4978]: 123.45.67.890:46022 [31/Jan/2019:10:38:50.563] public-web-in~ ncserver/nc1 0/0/-1/-1/3005 503 213 - - SC-- 2/2/0/0/3 0/0 "GET /favicon.ico HTTP/1.1"
Jan 31 10:39:33 haproxy haproxy[4978]: 123.45.67.890:46022 [31/Jan/2019:10:39:30.343] public-web-in~ ncserver/nc1 0/0/-1/-1/3005 503 213 - - SC-- 2/2/0/0/3 0/0 "GET / HTTP/1.1"
Jan 31 10:39:33 haproxy haproxy[4978]: 123.45.67.890:46022 [31/Jan/2019:10:39:30.343] public-web-in~ ncserver/nc1 0/0/-1/-1/3005 503 213 - - SC-- 2/2/0/0/3 0/0 "GET / HTTP/1.1"
Jan 31 10:39:36 haproxy haproxy[4978]: 123.45.67.890:46022 [31/Jan/2019:10:39:33.489] public-web-in~ ncserver/nc1 0/0/-1/-1/3004 503 213 - - SC-- 2/2/0/0/3 0/0 "GET /favicon.ico HTTP/1.1"
Jan 31 10:39:36 haproxy haproxy[4978]: 123.45.67.890:46022 [31/Jan/2019:10:39:33.489] public-web-in~ ncserver/nc1 0/0/-1/-1/3004 503 213 - - SC-- 2/2/0/0/3 0/0 "GET /favicon.ico HTTP/1.1"
Jan 31 10:40:04 haproxy haproxy[4978]: 123.45.67.890:49065 [31/Jan/2019:10:40:01.584] public-web-in~ ncserver/nc1 0/0/-1/-1/3006 503 213 - - SC-- 3/3/1/0/3 0/0 "GET /ocs/v2.php/apps/notifications/api/v2/notifications?format=json HTTP/1.1"
Jan 31 10:40:04 haproxy haproxy[4978]: 123.45.67.890:49065 [31/Jan/2019:10:40:01.584] public-web-in~ ncserver/nc1 0/0/-1/-1/3006 503 213 - - SC-- 3/3/1/0/3 0/0 "GET /ocs/v2.php/apps/notifications/api/v2/notifications?format=json HTTP/1.1"
Jan 31 10:40:06 haproxy haproxy[4978]: 123.45.67.890:55079 [31/Jan/2019:10:40:03.805] public-web-in~ ncserver/nc1 0/0/-1/-1/3006 503 213 - - SC-- 2/2/0/0/3 0/0 "PROPFIND /remote.php/dav/files/sarah/ HTTP/1.1"
Jan 31 10:40:06 haproxy haproxy[4978]: 123.45.67.890:55079 [31/Jan/2019:10:40:03.805] public-web-in~ ncserver/nc1 0/0/-1/-1/3006 503 213 - - SC-- 2/2/0/0/3 0/0 "PROPFIND /remote.php/dav/files/sarah/ HTTP/1.1"
Jan 31 10:40:11 haproxy haproxy[4978]: 123.45.67.890:44558 [31/Jan/2019:10:40:11.772] public-web-in/2: SSL handshake failure
Jan 31 10:40:11 haproxy haproxy[4978]: 123.45.67.890:44558 [31/Jan/2019:10:40:11.772] public-web-in/2: SSL handshake failure
Jan 31 10:40:14 haproxy haproxy[4978]: 123.45.67.890:54795 [31/Jan/2019:10:40:11.880] public-web-in~ ncserver/nc1 0/0/-1/-1/3005 503 213 - - SC-- 4/4/2/0/3 0/0 "GET /status.php HTTP/1.1"
Jan 31 10:40:14 haproxy haproxy[4978]: 123.45.67.890:54795 [31/Jan/2019:10:40:11.880] public-web-in~ ncserver/nc1 0/0/-1/-1/3005 503 213 - - SC-- 4/4/2/0/3 0/0 "GET /status.php HTTP/1.1"
Jan 31 10:40:16 haproxy haproxy[4978]: 123.45.67.890:39289 [31/Jan/2019:10:40:13.649] public-web-in~ ncserver/nc1 0/0/-1/-1/3007 503 213 - - SC-- 3/3/1/0/3 0/0 "GET /ocs/v2.php/core/navigation/apps?absolute=true&format=json HTTP/1.1"
Jan 31 10:40:16 haproxy haproxy[4978]: 123.45.67.890:39289 [31/Jan/2019:10:40:13.649] public-web-in~ ncserver/nc1 0/0/-1/-1/3007 503 213 - - SC-- 3/3/1/0/3 0/0 "GET /ocs/v2.php/core/navigation/apps?absolute=true&format=json HTTP/1.1"
Jan 31 10:40:20 haproxy haproxy[4978]: 123.45.67.890:52104 [31/Jan/2019:10:40:17.071] public-web-in~ ncserver/nc1 0/0/-1/-1/3006 503 213 - - SC-- 2/2/0/0/3 0/0 "GET /ocs/v2.php/core/navigation/apps?absolute=true&format=json HTTP/1.1"
Jan 31 10:40:20 haproxy haproxy[4978]: 123.45.67.890:52104 [31/Jan/2019:10:40:17.071] public-web-in~ ncserver/nc1 0/0/-1/-1/3006 503 213 - - SC-- 2/2/0/0/3 0/0 "GET /ocs/v2.php/core/navigation/apps?absolute=true&format=json HTTP/1.1"
john
  • 23
  • 1
  • 4
  • Start by finding the matching HAProxy log entry for the request that generates the 503. – Michael - sqlbot Jan 30 '19 at 04:33
  • No errors in `/var/log/haproxy.log` if that's what you meant, @Michael-sqlbot – john Jan 30 '19 at 21:14
  • HAProxy should create a log entry for every request, success or failure, unless configured not to. You'll need to address the lack of logging, first. – Michael - sqlbot Jan 30 '19 at 21:18
  • Thank you so much for your help, @Michael-sqlbot. I set the logging level to debug and added the logs to my post above. I think the `public-web-in/2: SSL handshake failure` part is the issue. I know that my nextcloud server is set up to use http/2 so perhaps that's where the hiccup is? I'm using haproxy version 1.9 after attempting to use 1.7. – john Jan 31 '19 at 15:50

1 Answers1

1

SSL handshake failure looks to be on the front-side of the proxy and is probably unrelated.

The most valuable information here is SC-- -- this field is called session state at disconnection and the value of the information provided here is difficult to overstate. On a successful request is set to ----.

The value SC-- means your back-end server is refusing connections on the target port, and the problem occurred very early, before the HTTP request could be sent to the backend over the connection (sensible enough, since no connection was established).

Note that for this backend, you appear to have ssl configured, so that server's address probably should be set to port 443, not 80.

You haven't enabled any health-check on the backend, so HAProxy doesn't know it's down/unreachable/stopped/not accessible/misconfigured/firewalled, and tries to send requests to it.

These fail, of course, and HAProxy writes the contents of /etc/haproxy/errors/503.http to the wire to signal the failure to the client.

Unrelated, be sure you understand what send-proxy means. This is a great feature if you need it, but there isn't strictly a need for that in your configuration, since you're running the proxy in Layer 7 mode. You could instead just add option forwardfor to set the X-Forwarded-For header.

Michael - sqlbot
  • 21,988
  • 1
  • 57
  • 81
  • Yes sorry, regarding your point about the `send-proxy`, I had stripped down the config file to the base essentials to try to debug this issue and forgot to remove that option. After I changed the port from 80 to 443 for the nextcloud backend it resolved the 503 issue. THANK YOU! Doof - I feel dumb. Thank you so much for your thorough explanation. :-) – john Jan 31 '19 at 16:49