4

We are using apache proxy to enable our application servers to reach specific web sites over the internet. The setup is as follows:

application servers --> apache proxy --> Internet website

Some of the requests fail with the below error in the application server log:

<head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a href="/link">POST
nbsp;/link</a></em>.<p>
Reason: <strong>Error reading from remote server</strong></p></p>
</body>

and the below debug log in the apache error.log file:

[Mon May 20 09:57:54 2013] [debug] mod_proxy_http.c(56): proxy: HTTP: canonicalising URL //myURL.com
[Mon May 20 09:57:54 2013] [debug] proxy_util.c(1506): [client 172.20.101.71] proxy: https: found worker `https://myurl.com/` for `https://myurl.com/link`
[Mon May 20 09:57:54 2013] [debug] mod_proxy.c(1015): Running scheme https handler (attempt 0)
[Mon May 20 09:57:54 2013] [debug] mod_proxy_http.c(1973): proxy: HTTP: serving URL https://myurl.com/link
[Mon May 20 09:57:54 2013] [debug] proxy_util.c(2011): proxy: HTTPS: has acquired connection for (mu=y url)
[Mon May 20 09:57:54 2013] [debug] ssl_engine_io.c(1908): OpenSSL: I/O error, 5 bytes expected to read on BIO#136e90 [mem: 190593]
[Mon May 20 09:57:54 2013] [debug] proxy_util.c(2067): proxy: connecting https://myurl.com/link to myurl.com:443
[Mon May 20 09:57:54 2013] [debug] proxy_util.c(2193): proxy: connected /link to myurl.com:443
[Mon May 20 09:57:54 2013] [debug] ssl_engine_io.c(1908): OpenSSL: I/O error, 5 bytes expected to read on BIO#136e90 [mem: 190593]
[Mon May 20 09:57:54 2013] [info] [client ip] (131)Connection reset by peer: SSL input filter read failed.
[Mon May 20 09:57:54 2013] [error] [client ip2] (131)Connection reset by peer: proxy: error reading status line from remote server myurl.com:443
[Mon May 20 09:57:54 2013] [debug] mod_proxy_http.c(1466): [client ip2] proxy: NOT Closing connection to client although reading from backend server "myurl.com:443 failed.
[Mon May 20 09:57:54 2013] [error] [client ip2] proxy: Error reading from remote server returned by /link
[Mon May 20 09:57:54 2013] [debug] proxy_util.c(2029): proxy: HTTPS: has released connection for (myurl.com)

Any ideas how I can solve this issue, knowing that the majority of the requests are being successfully sent and the response is being received normally. All the requests are the same length and generated automatically so the problem couldn't be in the request itself.

Læti
  • 2,075
  • 21
  • 33
Hytham
  • 41
  • 1
  • 1
  • 3
  • What's up with that question title? – MDMarra May 20 '13 at 11:56
  • 6
    @MDMarra: An astonishing number of people put their job title in the "title" field.... – Sven May 20 '13 at 11:57
  • 1
    ಠ_ಠ Wow. I can't believe I've never seen that before. – MDMarra May 20 '13 at 13:41
  • Are all of your failures in relation to HTTPS links? – jeffatrackaid May 20 '13 at 13:44
  • @jeffatrackaid: yes all the failures are in relation to https, actually the proxy server is connecting to only one link through https. – Hytham May 20 '13 at 18:11
  • What software is your application server? – Shane Madden May 20 '13 at 23:56
  • @ShaneMadden tibco business events, but do you think this has anything to do with the proxy error? – Hytham May 21 '13 at 10:43
  • @Hytham Absolutely - those error messages imply that the Apache server is getting invalid or broken SSL communication from the application server. Is there any logging you can look at on that system? – Shane Madden May 21 '13 at 19:29
  • which apache version? apache2.2 is different than 2.4 in regards to SSL Proxy handling. post configs and/or logs, or we can't help. – Marcel May 22 '13 at 00:22
  • @ShaneMadden the broken SSL communication is with the website, the application servers should be connecting directly to the internet website but due to security reasons the application servers are in a DMZ and accessing the internet through proxy instances, so the problem should be in the web site side which we don't have access to its logs. – Hytham May 22 '13 at 11:18
  • @Marcel Apache/2.2.21 the log for a failed request is already posted and the configuration and any other logs are available upon request.. I have tried the options available at this link but it didn't help `http://serverfault.com/questions/185894/proxy-error-502-reason-error-reading-from-remote-server-with-apache-2-2-3-de` – Hytham May 22 '13 at 11:23

2 Answers2

5

Looks like there are some issue described in this Apache Bugzilla. The solution was adding these line in httpd.conf inside <Proxy> section

SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1

Update

After deploying above solution, I faced another problem as the client was sending the request using HTTP/1.1 and the proxy was forced to use HTTP/1.0 as per the previous setEnv parameters, this caused HTTP error HTTP/1.1 417 Expectation Failed. This thread on SO mention that error could be solved from the client side or from the proxy side.

In the end, I implemented solution in proxy side based on this page. Now I've three parameters inside <Proxy> Section

SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
RequestHeader unset Expect early

I have been monitoring the solution since yesterday and it is working perfect until now. Also I performed a test yesterday with 500 transactions and they were all successful.

Update 2

It has been added to Apache docs since then.

Community
  • 1
masegaloeh
  • 17,978
  • 9
  • 56
  • 104
  • 1
    just wanted to add a note - this worked for me on apache 2.4 as well. – Brandt Solovij Jan 26 '16 at 22:15
  • I would also like to add that for .Net clients you need to add "RequestHeader unset Expect early" otherwise you will get "HTTP/1.1 417 Expectation Failed" as pointed out by masegaloeh. – Rob Mascaro Jan 09 '19 at 01:43
0

Check if you have both SSLProxyVerify none and SSLProxyCheckPeerCN off on your SSL Proxy Virtual Host.

Marcel
  • 1,575
  • 8
  • 14