Weird interaction with systemctl with Haproxy on CentOS 7

Question

Running haproxy through systemctl is different than running it manually.

on manual start, everything works.
starting through systemctl, haproxy is not able to find the service it proxies, and answers with a 503.

Here's the output when starting through systemctl:

> sudo systemctl status  -l haproxy.service
haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled)
                Active: active (running) since Wed 2014-12-24 08:08:49 EST; 4min 59s ago
                Main PID: 20307 (haproxy)
                CGroup: /system.slice/haproxy.service
                └─20307 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

                Dec 24 08:08:49 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
                Dec 24 08:08:49 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.

Like this, haproxy just says 503 for my backend.

If I now stop the haproxy service, and instead run it manually on the command line, like so:

sudo /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

it proxies fine to my server.

I've installed haproxy through yum, and I'm using an exceedingly simple configuration:

# defaults above

frontend  main *:80
    default_backend app

backend app
    option httpclose
    option forwardfor
    server web1 127.0.0.1:8001

Any idea what the difference could be? I've tried running it both daemonized and not.

This is the log when running with systemctl:

Proxy app started.
127.0.0.1:44391 [24/Dec/2014:09:01:42.403] main app/web1 0/0/-1/-1/0 503 212 - - SC-- 0/0/0/0/3 0/0 "GET / HTTP/1.1"

This is the log when running manually:

Proxy app started.
127.0.0.1:44393 [24/Dec/2014:09:02:11.758] main app/web1 0/0/0/2/2 200 5699 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

To be clear, I can successfully curl 127.0.0.1:8001 as both my deploy user, with sudo and the haproxy user.

Edit with more info

I tried using nginx as a proxy instead, to no avail. So it is not haproxy that is at fault here. nginx says 502 Bad Gateway, and the logs says:

2014/12/24 16:57:04 [crit] 23214#0: *1 connect() to 127.0.0.1:8001 failed (13: Permission denied) while connecting to upstream, client: 192.168.34.1, server: www.zombieclj.local, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8001/", host: "www.zombieclj.local"

So yeah, there is some security thing going on. I am still able to curl to 127.0.0.1:8001. Stopping the firewall doesn't change anything.

When you start haproxy manually, what user account do you use? — JasonAzze, Dec 24 '14 at 13:46
@MichaelHampton I've only seen it log something along the lines of "Proxy app started", which isn't very useful. — Magnar, Dec 24 '14 at 13:57
@MichaelHampton I've got some more logs now. The first is started from systemctl (503), the second is manually (200): https://gist.github.com/magnars/0dbf5d37f9c25b630f07 — Magnar, Dec 24 '14 at 14:03

score 7 · Answer 1 · edited Apr 13 '17 at 12:14

7

The problem was SELinux only allowing the web server to make outbound connections to a limited set of ports.

Fixed by doing:

semanage port --add --type http_port_t --proto tcp 8001

after installing semanage with

yum install policycoreutils-python

Nodejs Nginx error: (13: Permission denied) while connecting to upstream

edited Apr 13 '17 at 12:14

Community

1

answered Dec 24 '14 at 22:33

Magnar

1,017
2
11
15

1

This solution solved my problem. it should be marked as solution. – Sombriks Nov 09 '15 at 23:40
I ran into this issue when installing haproxy on a stock RHEL 7 image, for me the the port I wanted to use "8000" was already taken up by soundd_port_t so my solution involved some research using the Audit2Allow answer below. – Vijay Katam Nov 29 '16 at 23:37

score 2 · Answer 2 · edited May 10 '17 at 07:28

2

I used the Audit2Allow function found on the centos wiki

Review the rule:

grep haproxy /var/log/audit/audit.log | audit2allow -m haproxy

Create a policy file (*.pp).

grep haproxy /var/log/audit/audit.log | audit2allow -M haproxy

now when installing the package I also include the haproxy.pp file and load it to selinux with

semodule -i haproxy.pp

edited May 10 '17 at 07:28

kubanczyk

13,502
5
40
55

answered Dec 21 '15 at 14:13

Jacob Evans

7,636
3
25
55

1

The port I was running the backend service on "8000" was already taken by something called soundd_port_t, I used the solution documented at https://www.mankier.com/8/haproxy_selinux#Booleans which is to allow haproxy to connect to any tcp port. – Vijay Katam Nov 29 '16 at 23:53

Weird interaction with systemctl with Haproxy on CentOS 7

Edit with more info

2 Answers2

Linked