haproxy http check with a backup server shows 503 at main server down

Question

I have been trying to setup a forward with haproxy.

listen POC-2019-02-03
    bind 0.0.0.0:8083
    timeout connect  14000
    timeout client   180000
    timeout server   180000
    mode http
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpclose
    option httpchk GET / HTTP/1.1
    http-check expect rstring "Welcome"
    server S-ONE 127.0.0.1:6001 check
    server S-TWO 127.0.0.1:6002 check backup

which normally will serve as follows when request to http://127.0.0.1:8083:

• Shows the content of S-ONE
• But when requesting as like normal production scenario, multiple requests, as soon as I turn down the S-ONE, it shows

503 Service Unavailable

No server is available to handle this request.

Then the second request served via S-TWO

• It continuously then served by S-TWO. After then I do start the S-ONE and obviously request still serves from S-TWO, which is ok for me.

• I turn down S-TWO then without showing any error, it serves by S-ONE, Which is cool. But I do not get the reverse one in this scenario.

Is there anything I have to tweak here?

Answer 1

You should look at fall and inter:

fall <count>

The "fall" parameter states that a server will be considered as dead after consecutive unsuccessful health checks. This value defaults to 3 if unspecified. See also the "check", "inter" and "rise" parameters.

inter <delay>

The "inter" parameter sets the interval between two consecutive health checks to milliseconds. If left unspecified, the delay defaults to 2000 ms.

I've never tried that exact scenario, but I suspect before it actually marks the host as down (and thus activating the backup(s)), it has nowhere else to route traffic to, so continues sending it to {{s-ONE}} until it's officially "down".