Error 502 - Random SSL Library Error: 336151568 error:14094410

Question

I'm facing a random 502 error on my webservers.

Rate : 0.5%

Error in my logs: SSL Library Error: 336151568 error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

OpenSSL Version : OpenSSL 1.0.0-fips 29 Mar 2010

Network I can't post images, so here is a link to my network's scheme : http://i.stack.imgur.com/oOOnL.jpg and the same image with another hoster : http://s28.postimg.org/q9012t8l7/error502.jpg

Some details :

I host JEE applications on a tomcat 7.0.37

When someone tries to connect to one of those application, he connects to the external IP of my router. The router redirects him to my DMZ.

Apache configuration on the DMZ

<VirtualHost *:443>
  ServerName www.myapp.fr
  ServerAlias myapp.fr
  RedirectMatch ^/$ /CRA
  ProxyRequests Off

  SSLEngine on
  SSLProxyEngine on
  SSLCertificateFile /etc/ssl/certs/www.myapp.fr.pem


  ProxyPass / https://myapp.fr/
  ProxyPassReverse / https://myapp.fr/

      LogLevel info
     CustomLog "|/usr/sbin/rotatelogs /var/log/httpd/myapp.fr/SSL.myapp.fr_access.%Y-%m-%d.log 5M" combined
     ErrorLog  "|/usr/sbin/rotatelogs /var/log/httpd/myapp.fr/SSL.myapp.fr_error.%Y-%m-%d.log 5M"
</VirtualHost>

Myapp.fr is defined in /etc/hosts as 192.168.1.51, a webserver's IP

The DMZ use ProxyPass to this Web server. This Web server is used to ProxyPass to the real Application Server.

Apache configuration on the Webserver

    <VirtualHost *:443>
    ServerName myapp.fr
    RedirectMatch ^/$ https://myapp.fr/Collaborateurs/
         ProxyRequests Off
    SSLCertificateFile /etc/ssl/certs/www.myapp.fr.pem
     SSLProxyEngine On
    SSLEngine on

    ProxyPass /Collaborateurs/ https://ApplicationServer:8443/Collaborateurs/
    ProxyPassReverse /Collaborateurs/ https://ApplicationServer:8443/Collaborateurs/

     LogLevel info
        CustomLog "|/usr/sbin/rotatelogs ${APACHE_LOG_DIR}/myapp/SSL.myapp.fr_access.%Y-%m-%d.log 5M" combined
        ErrorLog  "|/usr/sbin/rotatelogs ${APACHE_LOG_DIR}/myapp/SSL.myapp.fr_error.%Y-%m-%d.log 5M"
</VirtualHost>

Here is the problem. Sometimes a random error appears in my logs, corresponding to a bad gateway error (502) for the user.

logs from /var/log/httpd/myapp.fr/SSL.myapp.fr_error.%Y-%m-%d.log :

[Mon Dec 23 08:58:31 2013] [info] Seeding PRNG with 136 bytes of entropy
[Mon Dec 23 08:58:31 2013] [info] Initial (No.1) HTTPS request received for child 11 (server myapp.fr:443)
[Mon Dec 23 08:58:31 2013] [info] [client 192.168.1.52] Connection to child 0 established (server myapp.fr:443)
[Mon Dec 23 08:58:31 2013] [info] Seeding PRNG with 136 bytes of entropy
[Mon Dec 23 08:58:31 2013] [info] [client 192.168.1.52] SSL Proxy connect failed
[Mon Dec 23 08:58:31 2013] [info] SSL Library Error: 336151568 error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
[Mon Dec 23 08:58:31 2013] [info] [client 192.168.1.52] Connection closed to child 0 with abortive shutdown (server myapp.fr:443)
[Mon Dec 23 08:58:31 2013] [error] (502)Unknown error 502: proxy: pass request body failed to 192.168.1.52:8443 (ApplicationServer.domain.lan)
[Mon Dec 23 08:58:31 2013] [error] proxy: pass request body failed to 192.168.1.52:8443 (ApplicationServer.red.lan) from 10.0.0.2 ()

Here are the same logs, when there isn't any error occuring :

[Mon Dec 23 09:08:30 2013] [info] Initial (No.1) HTTPS request received for child 8 (server myapp.fr:443)
[Mon Dec 23 09:08:30 2013] [info] Initial (No.1) HTTPS request received for child 0 (server myapp.fr:443)
[Mon Dec 23 09:08:30 2013] [info] [client 192.168.1.52] Connection to child 0 established (server myapp.fr:443)
[Mon Dec 23 09:08:30 2013] [info] Seeding PRNG with 136 bytes of entropy
[Mon Dec 23 09:08:31 2013] [info] Initial (No.1) HTTPS request received for child 3 (server myapp.fr:443)
[Mon Dec 23 09:08:31 2013] [info] [client 192.168.1.52] Connection to child 0 established (server myapp.fr:443)
[Mon Dec 23 09:08:31 2013] [info] Seeding PRNG with 136 bytes of entropy

I can't understand why this error happens. does anyone has an idea? Thanks for your eventual lead - and sorry for my bad english :D

score 0 · Answer 1 · answered Dec 23 '13 at 11:45

0

SSL proxy connect fails means there's been some kind of network error between your webserver, and your application server, before the SSL handshake completed. Without getting packet traces (and a bunch more) from your network, figuring out exactly why the TCP connection failed is pretty much impossible. I'd look for network congestion, or very high loads on the app server around the time of the errors though.

answered Dec 23 '13 at 11:45

Cian

5,777
1
27
40

thanks for your answer ! The app server isn't overloaded, I'll try to figure out why it isn't working with a wireshark.... but this 502 is kind of hard to reproduce ! – foru Dec 24 '13 at 10:13

score 0 · Accepted Answer · answered Jan 09 '14 at 11:00

0

Hyper-V integration was the problem :

Updating Linux integration components from 3.4 to 3.5 on the proxy server solved it.

answered Jan 09 '14 at 11:00

foru

11
3

Error 502 - Random SSL Library Error: 336151568 error:14094410

2 Answers2