Apache mod_cluster 1.2.6 and jboss 6: All workers are in error state for only one jboss web-application

Question

One of my Jboss web-applications under the URL "/api/hornet-q" always returns 503.

I'm hosting multiple web-applications in Jboss and I'm clustering it through Apache and mod_cluster. For now the Apache node and Jboss cluster node are on the same machine.

I observe:

Jboss web-applications successfully registered in Apache mod_cluster_manager:

Node: [1],Name: NL-AMS-SPAC-D52:server1,Balancer: mycluster,LBGroup: ,Host: 192.168.52.81,Port: 8080,Type: http,Flushpackets: Off,Flushwait: 10,Ping: 10,Smax: 1,Ttl: 60,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: 50
Vhost: [1:1:1], Alias: default-host
Vhost: [1:1:2], Alias: localhost
Vhost: [1:1:3], Alias: example.com
Context: [1:1:1], Context: /idp, Status: ENABLED
Context: [1:1:2], Context: /mainui, Status: ENABLED
Context: [1:1:3], Context: /deviceManagerWeb, Status: ENABLED
Context: [1:1:4], Context: /api/info, Status: ENABLED
Context: [1:1:5], Context: /api/hornet-q, Status: ENABLED
Context: [1:1:6], Context: /api/space/application-management, Status: ENABLED

Jboss logs: /var/log/jboss/servers/server1/default-host/access_log no entries

httpd logs: /var/log/httpd/access_log

192.168.224.165 - - [08/Mar/2017:14:54:35 +0000] "POST /api/hornet-q/queues HTTP/1.1" 503 323 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"

/var/log/httpd/error_log

[error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state

/var/log/httpd/modsec_audit.log

--5e41fb56-A--
[09/Mar/2017:10:01:56 +0000] WMEoFMCoNFAAADURA-AAAAAE 192.168.224.127 63255 192.168.52.81 80
--5e41fb56-B--
POST /api/hornet-q/queues HTTP/1.1
Host: 192.168.52.81
Connection: keep-alive
Content-Length: 58
Authorization: Basic xxxxxxxxxxxxxxxxxxxxxxxx
Postman-Token: 542b541e-f85e-97e9-97cc-a21f6c5f196b
Cache-Control: no-cache
Origin: chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
Content-Type: application/hornetq.jms.queue+xml
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8

--5e41fb56-C--
<queue name="qname"><durable>false</durable></queue>
--5e41fb56-F--
HTTP/1.1 503 Service Temporarily Unavailable
X-Frame-Options: SAMEORIGIN
Content-Length: 323
Connection: close
Content-Type: text/html; charset=iso-8859-1

--5e41fb56-H--
Apache-Handler: proxy-server
Stopwatch: 1489053716151582 10977 (- - -)
Stopwatch2: 1489053716151582 10977; combined=110, p1=16, p2=88, p3=1, p4=0, p5=5, sr=0, sw=0, l=0, gc=0
Producer: ModSecurity for Apache/2.6.8 (http://www.modsecurity.org/).
Server: Apache

--5e41fb56-Z--

I've run a "tcpdump" and noticed httpd is not forwarding requests to Jboss for "/api/hornet-q/queues" but does for "/api/space".

tcpdump -s 0 -i any -w /home/vagrant/tcpdump.pcap

tcpdump

Configuration

/etc/hosts

127.0.0.1               localhost.localdomain localhost
192.168.52.80   NL-AMS-SPAC-D52 server1 server2
192.168.52.80   NL-AMS-SPAC-D52 space-005056a25f15 server1 server2
192.168.52.81   jmp-CLUSTER     jmp-CLUSTER
228.168.52.81   jmp-MCAST       jmp-MCAST

Here's the configuration for mod_cluster:

Jboss config domain.xml

<subsystem xmlns="urn:jboss:domain:web:2.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="true">
    <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
    <connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp"/>
    <virtual-server name="default-host" enable-welcome-root="false">
        <alias name="localhost"/>
        <alias name="example.com"/>
        <sso cache-container="web" cache-name="sso" reauthenticate="false"/>
    </virtual-server>
</subsystem>

<subsystem xmlns="urn:jboss:domain:modcluster:1.2">       
    <mod-cluster-config advertise-socket="modcluster" proxy-list="jmp-CLUSTER:8888" advertise="true" excluded-contexts="ROOT" connector="http">
        <dynamic-load-provider>
            <custom-load-metric class="net.juniper.jmp.cmp.modcluster.EarsInitializedCustomMetric"/>
        </dynamic-load-provider>
    </mod-cluster-config>
</subsystem>

 <socket-binding-groups>
    <socket-binding-group name="full-ha-sockets" default-interface="jmpnodename">
        <socket-binding name="ajp" port="8009"/>
        <socket-binding name="http" interface="public" port="8080"/>
        <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address}" multicast-port="45700"/>
        <socket-binding name="jgroups-tcp" port="7600"/>
        <socket-binding name="jgroups-tcp-fd" port="57600"/>
        <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address}" multicast-port="45688"/>
        <socket-binding name="jgroups-udp-fd" port="54200"/>
        <socket-binding name="messaging" port="5445"/>
        <socket-binding name="messaging-group" port="0" multicast-address="${jboss.default.multicast.address}" multicast-port="${jboss.messaging.group.port:9876}"/>
        <socket-binding name="messaging-throughput" port="5455"/>
        <socket-binding name="modcluster" port="0" multicast-address="${jboss.default.multicast.address}" multicast-port="23364"/>
        <socket-binding name="remoting" port="4447"/>
        <socket-binding name="txn-recovery-environment" port="4712"/>
        <socket-binding name="txn-status-manager" port="4713"/>
        <outbound-socket-binding name="mail-smtp">
            <remote-destination host="127.0.0.1" port="25"/>
        </outbound-socket-binding>
        <outbound-socket-binding name="remote-ejb-1">
            <remote-destination host="${jgroups.bind_addr:127.0.0.1}" port="4447"/>
        </outbound-socket-binding>
    </socket-binding-group>
</socket-binding-groups>

httpd.config

Listen jmp-CLUSTER:8888
<VirtualHost jmp-CLUSTER:8888>
    <Location />
        Order allow,deny
        Allow from localhost jmp-CLUSTER NL-AMS-SPAC-D52
        Options none
    </Location>
    RewriteEngine On
    RewriteOptions Inherit
    ServerAdvertise On
    AdvertiseFrequency 5
    AdvertiseGroup 228.168.52.81:23364
    AdvertiseBindAddress 228.168.52.81:23364
    EnableMCPMReceive
    ManagerBalancerName mycluster
</VirtualHost>

<VirtualHost jmp-CLUSTER:80>
    …

    # Primary entry point
    ProxyPass /mainui balancer://mycluster/mainui stickysession=JSESSIONID
    ProxyPassReverse /mainui/ balancer://mycluster/mainui/
    ProxyPassReverse /idp/ balancer://mycluster/idp/

    # local services & blacklist
    ProxyPass /jmx-console !
    ProxyPass /web-console !
    ProxyPass /jbossws !
    ProxyPass /juddi !
    ProxyPass /invoker !

    # whitelist: from NmaWebproxy
    ProxyPass /api balancer://mycluster/api stickysession=JSESSIONID
    …

    <Location /mainui>
       <LimitExcept GET POST>
         Deny from all
       </LimitExcept>
       Options None
       Order Allow,Deny
       Allow from all
       ErrorDocument 503 /maintenance/503.html
    </Location>
    …

</VirtualHost>

I've tried:

A curl request from the server itself to Jboss

curl -H 'Authorization: Basic xxxxxxxxxxxxxxx' -H 'Content-Type: application/hornetq.jms.queue+xml' http://192.168.52.81:8080/api/hornet-q/queues -d 'false'
suggestion from the following post: mod_cluster reports "MEM: Can't read node" and "All workers are in error state" for JBoss AS 7.1.1 cluster

Try remove all `ProxyPass` and`ProxyPassReverse` directives. — Federico Sierra, Mar 09 '17 at 15:51
I removed all the `ProxyPass` and `ProxyPassReverse` but now I get 403 errors. But then I've allowed only one `ProxyPass /api balancer://mycluster/api stickysession=JSESSIONID` and I get again 503 errors for /api/hornet-q and others /api/~ are working normally. — Ravindra Solanki, Mar 09 '17 at 16:22
I'm not sure but 403 errors can be for modsec, try disable this and try without ProxyPass directive. — Federico Sierra, Mar 09 '17 at 17:22

Apache mod_cluster 1.2.6 and jboss 6: All workers are in error state for only one jboss web-application

0 Answers0