Not able to start apache in Wildfly 10 domain mode cluster with Apache mod_cluster, due to SELinux issue

Question

I am trying to setup a Wildfly 10 cluster in domain mode with Apache mod_cluster.

In my Centos 7 webserver node I have installed Apache(2.4.6) using:

# yum install httpd

Then copied the following .so files to the /etc/httpd/modules directory

mod_cluster_slotmem.so
mod_manager.so
mod_proxy_cluster.so
mod_advertise.so

and appended the following in the httpd.conf file

# LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
.
.
.

LoadModule cluster_slotmem_module modules/mod_cluster_slotmem.so
LoadModule manager_module modules/mod_manager.so
LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
LoadModule advertise_module modules/mod_advertise.so

<IfModule manager_module>
    Listen 192.168.56.105:10001
    ManagerBalancerName other-server-group
<VirtualHost 192.168.56.105:10001>
    <Location />
       Require all granted
    </Location>
    <Location /mod_cluster-manager>
       SetHandler mod_cluster-manager
       Require all granted
    </Location>
</VirtualHost>
</IfModule>

Now when I am trying to start the httpd, it is throwing some error:

# systemctl start httpd
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

Mar 25 17:39:03 webserver01.internal setroubleshoot[2772]: SELinux is preventing /usr/sbin/httpd from write access on the file /var/log/httpd/manager.node.nodes. For co
Mar 25 17:39:03 webserver01.internal python[2772]: SELinux is preventing /usr/sbin/httpd from write access on the file /var/log/httpd/manager.node.nodes.

                                               *****  Plugin catchall (100. confidence) suggests   **************************

                                               If you believe that httpd should be allowed write access on the manager.node.nodes file by default.
                                               Then you should report this as a bug.
                                               You can generate a local policy module to allow this access.
                                               Do
                                               allow this access for now by executing:
                                               # ausearch -c 'httpd' --raw | audit2allow -M my-httpd
                                               # semodule -i my-httpd.pp

Detailed log:

type=AVC msg=audit(1521962682.292:313): avc:  denied  { write } for  pid=3891 comm="httpd" path="/var/log/httpd/manager.node.nodes.lock" dev="dm-0" ino=656345 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=file
type=SYSCALL msg=audit(1521962682.292:313): arch=c000003e syscall=2 success=no exit=-13 a0=5583cf525ce0 a1=80041 a2=1b6 a3=ffffff00 items=0 ppid=1 pid=3891 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521962682.292:313): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521964160.534:399): avc:  denied  { write } for  pid=4580 comm="httpd" path="/var/log/httpd/manager.node.nodes" dev="dm-0" ino=656323 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=file
type=SYSCALL msg=audit(1521964160.534:399): arch=c000003e syscall=2 success=no exit=-13 a0=560012130cb8 a1=800c1 a2=1b6 a3=ffffff00 items=0 ppid=1 pid=4580 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521964160.534:399): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521964202.459:432): avc:  denied  { remove_name } for  pid=4642 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656323 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1521964202.459:432): arch=c000003e syscall=87 success=no exit=-13 a0=560acdcc7cb8 a1=560acdd6a748 a2=180 a3=7ffc42786620 items=0 ppid=1 pid=4642 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521964202.459:432): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521964203.462:433): avc:  denied  { remove_name } for  pid=4642 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656323 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1521964203.462:433): arch=c000003e syscall=87 success=no exit=-13 a0=560acdcc7cb8 a1=560acdd6a748 a2=180 a3=ffffff00 items=0 ppid=1 pid=4642 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521964203.462:433): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521964204.463:434): avc:  denied  { remove_name } for  pid=4642 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656323 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1521964204.463:434): arch=c000003e syscall=87 success=no exit=-13 a0=560acdcc7cb8 a1=560acdd6a748 a2=180 a3=ffffff00 items=0 ppid=1 pid=4642 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521964204.463:434): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521964206.467:436): avc:  denied  { remove_name } for  pid=4642 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656323 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1521964206.467:436): arch=c000003e syscall=87 success=no exit=-13 a0=560acdcc7cb8 a1=560acdd6a748 a2=180 a3=ffffff00 items=0 ppid=1 pid=4642 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521964206.467:436): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521964205.465:435): avc:  denied  { remove_name } for  pid=4642 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656323 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1521964205.465:435): arch=c000003e syscall=87 success=no exit=-13 a0=560acdcc7cb8 a1=560acdd6a748 a2=180 a3=ffffff00 items=0 ppid=1 pid=4642 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521964205.465:435): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521974913.642:174): avc:  denied  { remove_name } for  pid=2738 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656323 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=dir
type=SYSCALL msg=audit(1521974913.642:174): arch=c000003e syscall=87 success=yes exit=0 a0=55fac7b30cb8 a1=55fac7bd3598 a2=180 a3=7ffedd5736e0 items=0 ppid=1 pid=2738 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521974913.642:174): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521974913.642:175): avc:  denied  { write } for  pid=2738 comm="httpd" path="/var/log/httpd/manager.node.nodes" dev="dm-0" ino=656322 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=file
type=SYSCALL msg=audit(1521974913.642:175): arch=c000003e syscall=2 success=yes exit=18 a0=55fac7b30cb8 a1=800c1 a2=1b6 a3=ffffff00 items=0 ppid=1 pid=2738 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521974913.642:175): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521974913.643:176): avc:  denied  { name_bind } for  pid=2738 comm="httpd" src=23364 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=udp_socket
type=SYSCALL msg=audit(1521974913.643:176): arch=c000003e syscall=49 success=yes exit=0 a0=16 a1=55fac7b31140 a2=10 a3=0 items=0 ppid=1 pid=2738 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521974913.643:176): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521975145.939:226): avc:  denied  { write } for  pid=2738 comm="httpd" path="/var/log/httpd/manager.node.nodes.slotmem" dev="dm-0" ino=656174 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=file
type=SYSCALL msg=audit(1521975145.939:226): arch=c000003e syscall=2 success=yes exit=3 a0=55fac7bd44d8 a1=80042 a2=1b6 a3=0 items=0 ppid=1 pid=2738 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521975145.939:226): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521975145.940:227): avc:  denied  { remove_name } for  pid=2738 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656322 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=dir
type=AVC msg=audit(1521975145.940:227): avc:  denied  { unlink } for  pid=2738 comm="httpd" name="manager.node.nodes" dev="dm-0" ino=656322 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=file
type=SYSCALL msg=audit(1521975145.940:227): arch=c000003e syscall=87 success=yes exit=0 a0=55fac7bd3960 a1=55fac7bd3598 a2=0 a3=7ffedd5738a0 items=0 ppid=1 pid=2738 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521975145.940:227): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=AVC msg=audit(1521979740.681:199): avc:  denied  { write } for  pid=2761 comm="httpd" path="/var/log/httpd/manager.node.nodes" dev="dm-0" ino=655447 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_log_t:s0 tclass=file
type=SYSCALL msg=audit(1521979740.681:199): arch=c000003e syscall=2 success=no exit=-13 a0=5598961f3cb8 a1=800c1 a2=1b6 a3=ffffff00 items=0 ppid=1 pid=2761 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1521979740.681:199): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44

When I ran SELinux in permissive mode then httpd started properly and I was able to access the cluster.

As running SELinux in permissive mode is not recommended please help me to find out the root cause of this behavior and how to fix it?

-------UPDATE------------

As suggested by Tom H, the output of inspection using audit2allow:

# audit2allow -i /var/log/audit/audit.log -m my-httpd

module my-httpd 1.0;

require {
    type gssproxy_t;
    type httpd_log_t;
    type httpd_t;
    type fs_t;
    type unreserved_port_t;
    class udp_socket name_bind;
    class file { unlink write };
    class dir remove_name;
    class filesystem getattr;
}

#============= gssproxy_t ==============

#!!!! This avc is allowed in the current policy
allow gssproxy_t fs_t:filesystem getattr;

#============= httpd_t ==============
allow httpd_t httpd_log_t:dir remove_name;

#!!!! The file '/var/log/httpd/manager.node.nodes.lock' is mislabeled on your system.
#!!!! Fix with $ restorecon -R -v /var/log/httpd/manager.node.nodes.lock
allow httpd_t httpd_log_t:file { unlink write };

#!!!! This avc can be allowed using the boolean 'nis_enabled'
allow httpd_t unreserved_port_t:udp_socket name_bind;

Other o/p:

# sesearch -s httpd_t -t httpd_log_t --allow
Found 6 semantic av rules:
   allow daemon logfile : file { ioctl getattr lock append } ;
   allow httpd_t httpd_log_t : lnk_file { read getattr } ;
   allow httpd_t httpd_log_t : file { ioctl read write create getattr setattr lock append unlink open } ;
   allow httpd_t file_type : filesystem getattr ;
   allow httpd_t file_type : dir { getattr search open } ;
   allow httpd_t httpd_log_t : dir { ioctl write create getattr setattr lock add_name remove_name search open } ;

# rpm -qa | egrep 'httpd|selinux'
libselinux-2.5-11.el7.x86_64
httpd-2.4.6-67.el7.centos.6.x86_64
pcp-selinux-3.11.8-7.el7.x86_64
selinux-policy-3.13.1-166.el7_4.9.noarch
httpd-manual-2.4.6-67.el7.centos.6.noarch
libselinux-python-2.5-11.el7.x86_64
libselinux-utils-2.5-11.el7.x86_64
selinux-policy-targeted-3.13.1-166.el7_4.9.noarch
httpd-tools-2.4.6-67.el7.centos.6.x86_64
libselinux-2.5-11.el7.i686

to give a better answer to this, can you paste the output of `sesearch -s httpd_t -t httpd_log_t --allow` and `rpm -qa | egrep 'httpd|selinux'` (you might need to install `yum install setools-console` ) — Tom, Mar 25 '18 at 16:57
Take a look http://modcluster.io/faq/#using-mod_cluster-and-selinux — Federico Sierra, Mar 26 '18 at 14:41

score 2 · Answer 1 · answered Mar 25 '18 at 20:20

Your audit2allow output contains an interesting comment, which you should have read:

#!!!! The file '/var/log/httpd/manager.node.nodes.lock' is mislabeled on your system.
#!!!! Fix with $ restorecon -R -v /var/log/httpd/manager.node.nodes.lock

I would also guess that you have more mislabeled files on your system. I'd fix the labels recursively for the whole directory myself.

restorecon -R -v /var/log/httpd

You should also ensure that the system is up to date and in particular has received SELinux policy updates.

Also `#!!!! This avc can be allowed using the boolean 'nis_enabled' allow httpd_t unreserved_port_t:udp_socket name_bind;` — dawud, Mar 25 '18 at 20:22

Tom · Answer 2 · 2018-03-25T15:29:52.523

This is the steps for dealing with selinux denials, using the auto-suggestions from audit2allow.

1) Install packages for selinux policy stuff

# yum install -y checkpolicy \
               policycoreutils \
               policycoreutils-python

2) Use the audit2allow tool from policycoreutils-python to generate a policy file. You can inspect it like so

# audit2allow -i /var/log/audit/audit.log -m my-httpd

It will look something like this (with values appropriate to your app);

module my-httpd 1.0;

require {
        type var_log_t;
        type zabbix_var_run_t;
        type zabbix_t;
        type mysqld_t;
        class sock_file { create unlink };
        class unix_stream_socket connectto;
        class process setrlimit;
        class file open;
}

#============= mysqld_t ==============

#!!!! The file '/var/log/mysql/slow.log' is mislabeled on your system.
#!!!! Fix with $ restorecon -R -v /var/log/mysql/slow.log
allow mysqld_t var_log_t:file open;

#============= zabbix_t ==============
allow zabbix_t self:process setrlimit;

#!!!! The file '/run/zabbix/zabbix_server_preprocessing.sock' is mislabeled on your system.
#!!!! Fix with $ restorecon -R -v /run/zabbix/zabbix_server_preprocessing.sock
#!!!! This avc can be allowed using the boolean 'daemons_enable_cluster_mode'
allow zabbix_t self:unix_stream_socket connectto;
allow zabbix_t zabbix_var_run_t:sock_file { create unlink };

3) Use the tools to generate a custom selinux policy for your specific application;

# ausearch -c 'httpd' --raw | audit2allow -M my-httpd

4) install the policy;

# semodule -i my-httpd.pp

More infos on creating policyfiles are here;
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security-enhanced_linux/sect-security-enhanced_linux-fixing_problems-allowing_access_audit2allow
and
https://danwalsh.livejournal.com/24750.html

thanks for the answer. I did step no. 3 & 4 and it worked! Now my question is whether it is recommended to use audit2allow utility in production environment. The redhat link that you had shared says "Do not use the example in this section in production." I have updated my question with the output of the command you mentined in step 2. — abhishek, Mar 25 '18 at 17:00
Yes, it's too general a relaxation of the rules. Can you post the output of the policy file? i.e. step 2. I'd like to see what rules it added... — Tom, Mar 25 '18 at 17:09
I think this might be more specific `chcon -v --type=httpd_sys_rw_content_t /var/log/httpd/manager.node.nodes.slotmem` — Tom, Mar 25 '18 at 17:16
Hello Tom, posted the output that you had asked in updated section of my question. — abhishek, Mar 25 '18 at 17:27

Not able to start apache in Wildfly 10 domain mode cluster with Apache mod_cluster, due to SELinux issue

2 Answers2