Apache in front of JBoss 6 -> Some proxy errors

Question

Here under my Apache configuration:

<IfDefine TOMCAT>
    <IfModule mod_ssl.c>
       <IfModule mod_proxy_http.c>
          ProxyTimeout      300
          SetEnv force-proxy-request-1.0 1
          SetEnv proxy-nokeepalive 1
          ProxyPass        /unproxied/ !
          ProxyPass        /error/ !
          ProxyPass        /   http://127.0.0.1:8080/ retry=0 ttl=600
          ProxyPassReverse /   http://127.0.0.1:8080/
       </IfModule>
    </IfModule> </IfDefine>

Here my JBoss connector configuration:

    <Connector
        address              = "${jboss.bind.address}"
        port                 = "8080"
        maxThreads           = "200"
        acceptCount          = "100"
        scheme               = "https"
        secure               = "true"
        proxyName            = "mydomain.net"
        proxyPort            = "443"
        xpoweredBy           = "true"
        connectionTimeout    = "600000"
        compression          = "on"
        enableLookups        = "false"
        maxKeepAliveRequests = "-1"
        maxHttpHeaderSize    = "50000"
        />

In my Apache error.log:

[Tue Jul 16 09:54:54 2013] [error] ap_proxy_connect_backend disabling worker for (127.0.0.1)
[Tue Jul 16 09:57:35 2013] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8080 (127.0.0.1) failed

[EDIT] These errors appear around 1000 times per day but my web application seems to work as expected.

What could be the problem ?

Thanks,

Bob !

score 0 · Answer 1 · answered Jul 16 '13 at 17:55

0

Is SELinux enabled and enforcing?

If so, you'll need to setsebool httpd_can_network_connect 1 to allow Apache to make connections to JBoss.

answered Jul 16 '13 at 17:55

Shane Madden

112,982
12
174
248

I've forgotten an important thing. My web application works as expected, but some times, every 10 minutes perhaps, I've these errors in my log file. – Bob Sauvage Jul 16 '13 at 17:57
@BobSauvage Ah, ok. Do you know if the errors coincide with failures of user requests? (they'll get a 503 response, look for those in the logs) What kind of load is the application under? – Shane Madden Jul 16 '13 at 18:00
Indeed, several proxy errors coincide with a 503 response... The load is difficult to measure but it is quite high. Just to have an idea: netstat -aln | grep 8080 | wc -l => 810 – Bob Sauvage Jul 16 '13 at 18:07
@BobSauvage Might be bumping into the JBoss thread limit, then, causing rejected connection attempts -- check what the `maxThreads` setting is for your port 8080 connector in the `server.xml` file? – Shane Madden Jul 16 '13 at 18:17
maxThreads = "200" How can I check the number of threads ? – Bob Sauvage Jul 16 '13 at 18:28
No problem, I've checked with JMX: --mbean="jboss.web:name=http-127.0.0.1-8080,type=ThreadPool --attribute="currentThreadCount" => 20. You already helped me a lot, I have to discuss about 503 errors with the development team tomorrow. These errors concerns a new view ;) – Bob Sauvage Jul 16 '13 at 18:35
@BobSauvage Gotcha. Let me know what you find out! – Shane Madden Jul 16 '13 at 20:08
The problem is still present but appears less frequent, about 10 times a day. :( – Bob Sauvage Feb 13 '14 at 14:34
In the kern.log, I can see that for each 503 error (proxy error), I have an invalid SYN tcp packet: kernel: invalid:IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14985 DF PROTO=TCP SPT=50708 DPT=8080 WINDOW=32792 RES=0x00 SYN URGP=0 – Bob Sauvage Feb 14 '14 at 15:55
@BobSauvage Interesting. Can you try capturing one of those apparently invalid packets via `tcpdump`? – Shane Madden Feb 14 '14 at 16:29
Hahaha, I am glad to see that we are thinking along the same lines ! I scheduled this test on monday :) ! Will keep you updated, maybe I will try tomorrow. – Bob Sauvage Feb 14 '14 at 18:53
tcpdump didn't help me a lot... I found the TCP packet: `15:27:11.863103 IP 127.0.0.1.53716 > 127.0.0.1.8080: Flags [S], seq 1102362290, win 32792, options [mss 16396,sackOK,TS val 969253337 ecr 0,nop,wscale 7], length 0 E..<.k@.@.%O..........$.A............'....@.... 9...........` But the followed packet was another request. So the JBoss server didn't seem to ack the SYN packet... What could be the reason ? – Bob Sauvage Feb 17 '14 at 14:55
@BobSauvage You'll want to get it into a pcap file with `-s 0 -w filename.pcap` - feeding it to wireshark will do some more thorough analysis, and should tell us if the packet's invalid or if the kernel's wrong about saying it's invalid. – Shane Madden Feb 17 '14 at 17:56
The pcap file is opened in wireshark, how can I check if the packet is really invalid ? Many thanks for your help in this topic... – Bob Sauvage Feb 18 '14 at 10:23
since I've added the rule in order to accept invalid packets, I've another error in the Apache log: "(110)Connection timed out: proxy:". With this rule, in wireshark, I can see that JBoss is accepting the connection with a SYN ACK but the connection is immediately closed by Apache with a RST packet. Why is Apache sending an invalid or RST packet ? How can I check that ? – Bob Sauvage Feb 18 '14 at 15:26
@BobSauvage Wait, you needed an `iptables` rule for accepting the invalid packet, on the loopback interface? That's.. strange. Since it's loopback and all, would you be able to add a rule accepting all traffic? I kinda suspect that `RST` is coming from `iptables`, not Apache. – Shane Madden Feb 18 '14 at 20:12
Yes, firewall rules have been build by another administrator. In the INPUT chain, I've a rule that logs and REJECTS all INVALID packets: `iptables -A INPUT -m state --state INVALID -j LOG --log-prefix "IPTBL invalid:"` and then logs and REJECTS not SYN but new: `iptables -A INPUT -p tcp ! --syn -m state --state NEW -j LOG --log-prefix "IPTBL new-not-syn:"`, `iptables -A INPUT -p tcp ! --syn -m state --state NEW -j REJECT --reject-with tcp-reset` (cause of the RST). You absolutely right, RST is coming from iptables. I noticed this there are few hours. – Bob Sauvage Feb 18 '14 at 21:26
Thanks a lot for your tips. The problem was related to these iptables rules. I accept now all packets from 127.0.0.1 to and from 8080 and I don't see this kind of errors anymore. I've sent an email to the iptables mailing-list. Do you think I should add a rule in order to accept ALL packets from and to 127.0.0.1 in the begining ? – Bob Sauvage Feb 19 '14 at 09:38

Apache in front of JBoss 6 -> Some proxy errors

1 Answers1