92

Apache is receiving requests at port :80 and proxying them to Jetty at port :8080

The proxy server received an invalid response from an upstream server
The proxy server could not handle the request GET /.

My dilemma: Everything works fine normally (fast requests, few seconds or few tens of seconds long requests are processed ok). Problems occur when request processing takes long (few minutes?).

If I issue request instead directly to Jetty at port :8080 the request is processed OK. So problem is likely to sit somewhere between Apache and Jetty where I am using mod_proxy. How to solve this?

I have already tried some "tricks" related to KeepAlive settings, without luck. Here is my current configuration, any suggestions?

#keepalive Off                     ## I have tried this, does not help
#SetEnv force-proxy-request-1.0 1  ## I have tried this, does not help
#SetEnv proxy-nokeepalive 1        ## I have tried this, does not help
#SetEnv proxy-initial-not-pooled 1 ## I have tried this, does not help
KeepAlive 20                       ## I have tried this, does not help
KeepAliveTimeout 600               ## I have tried this, does not help
ProxyTimeout 600                   ## I have tried this, does not help

NameVirtualHost *:80
<VirtualHost _default_:80>
    ServerAdmin webmaster@mydomain.fi

    ServerName www.mydomain.fi

    ServerAlias mydomain.fi mydomain.com mydomain www.mydomain.com

    ProxyRequests On
    ProxyVia On
    <Proxy *>
            Order deny,allow
            Allow from all
    </Proxy>

    ProxyRequests Off
    ProxyPass / http://www.mydomain.fi:8080/ retry=1 acquire=3000 timeout=600
    ProxyPassReverse / http://www.mydomain.fi:8080/

    RewriteEngine On
    RewriteCond %{SERVER_NAME} !^www\.mydomain\.fi
    RewriteRule /(.*) http://www.mydomain.fi/$1 [redirect=301L]

    ErrorLog /var/log/apache2/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    CustomLog /var/log/apache2/access.log combined
    ServerSignature On

</VirtualHost>

Here is also the debug log from a failing request:

74.125.43.99 - - [29/Sep/2010:20:15:40 +0300] "GET /?wicket:bookmarkablePage=newWindow:com.mydomain.view.application.reports.SaveReportPage HTTP/1.1" 502 355 "https://www.mydomain.fi/?wicket:interface=:0:2:::" "Mozilla/5.0 (Windows; U; Windows NT 6.1; fi; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10"
[Wed Sep 29 20:20:40 2010] [error] [client 74.125.43.99] proxy: error reading status line from remote server www.mydomain.fi, referer: https://www.mydomain.fi/?wicket:interface=:0:2:::
[Wed Sep 29 20:20:40 2010] [error] [client 74.125.43.99] proxy: Error reading from remote server returned by /, referer: https://www.mydomain.fi/?wicket:interface=:0:2:::
  • Hi.. I am still stuck with this one. All above settings tried and also increasing jetty maxIdleTime did not help. Any pointers what to try next? – Martin Feb 12 '11 at 07:49

6 Answers6

108

I have solved the problem. The Keepalive=On should be inserted into ProxyPass config line:

ProxyPass / http://www.dom.fi:8080/ retry=1 acquire=3000 timeout=600 Keepalive=On

See that 

Keepalive=On

there? It is critical ;)

arne.z
  • 357
  • 6
  • 24
Martin
  • 1,081
  • 1
  • 7
  • 3
  • 16
    I believe you can mark your own answers as Accepted. It marks the question as resolved in the system for other people to find. – sysadmin1138 Feb 18 '11 at 22:50
  • Where exactly do you put this? – AlxVallejo Jan 27 '16 at 15:34
  • 1
    @AlxVallejo You should find your config files here /etc/apache2/sites-enabled/[sitename].conf – Steven Feb 08 '16 at 18:04
  • 1
    We have exactly the same proxy errors. They happen very rarely (one in a thousand requests). *Why* exactly is that `Keepalive=On` critical? – dokaspar Apr 15 '16 at 12:22
  • 1
    Could it not be the `timeout=600` or `retry=1` that fixes this instead? (or the combo) – MattBianco Sep 13 '16 at 07:20
  • @dokaspar I know it's been a long time and this is a long shot but have you ever figured out what the problem was? I am also getting 502 very rarely (about 3 in 2000 requests). Did the `KeepAlive=on` option fix this for you? – HomeIsWhereThePcIs Feb 12 '20 at 08:56
9

This error can also occur if you don't end your proxy url with a /. Either both paths should end with a / or neither.

Mark
  • 221
  • 3
  • 6
5

Have you tried setting setenv proxy-initial-not-pooled 1?

Reference here

TCampbell
  • 2,014
  • 14
  • 14
  • 1
    No, this didn't help. Then it's not about race condition, it's the long delay and something happens in between (some kind of misunderstanding between Jetty and mod_proxy). –  Sep 29 '10 at 19:47
  • `setenv: command not found` – Jamie Hutber Oct 18 '21 at 21:20
1

Looking at the log, there's something that times out at 5 minutes (=300 seconds). That's a pretty long time to wait for a response. When you access the Jetty server directly, does this resource really take that long to produce a response?

If the five minutes really is within possible response times, you might try tweaking the ProxyTimeout configuration directive.

Depending on your network set-up, it might well be that there's no reason to even try to use any keepalive system (is there a firewall between the app server and proxy which might be configured to drop sessions that are idle for too long?), but the ProxyTimeout would affect the behaviour of the proxy itself.

If the same proxy also serves other backends, it would be better to keep the current ProxyTimeout, and configure the timeout in the ProxyPass directive (see mod_proxy documentation).

If, however, the responses without the proxy are consistently something much less than the five minutes see here as the cut-off limit, then there might really be some odd interference between the proxy and app server, but you're not providing anything of value for identifying what it might be.

  • "When you access the Jetty server directly, does this resource really take that long to produce a response?" -- YES. I also tried setting this ProxyTimeout to 600. Does not help. There is no firewall between proxy and jetty. Timeout is configured also in ProxyPass. –  Oct 11 '10 at 12:39
  • "you're not providing anything of value for identifying what it might be": I don't know what that could be. All i am getting is an error message from the server: Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /. Reason: Error reading from remote server –  Oct 11 '10 at 12:40
  • As for increasing the proxy timeout, did this also change the time your browser did spin before getting the 502 error? –  Oct 11 '10 at 17:01
  • Then to ways how you could find out what is happening in the application server: you could take a couple of thread dumps, and look from them what is executing when the browser is waiting. Alternatively, add enough debug logging statements to be able to trace your code to pinpoint the cause of slowness. –  Oct 11 '10 at 17:04
  • I know why it is slow. I don't know why apache proxy refuses the response when it is finally ready. –  Oct 12 '10 at 03:47
0

For me removing a header value called Transfer-Encoding" (binary) in my server-app (PHP) solved the problem for:

[proxy_http:error] [pid 17623] (22)Invalid argument: [client 127.0.0.1:44929] AH01102: error reading status line from remote server 0.0.0.0:80

All the other suggestions like SetEnv proxy-initial-not-pooled or Keep-Alive did not.

BE77Y
  • 2,577
  • 3
  • 17
  • 23
stackoverclan
  • 101
  • 2
0

If the above solutions don't work, one thing you can try is to enable all your apache modules to make sure that there isn't some module you need that somehow got accidentally disabled.

For example, how I found the cause of my problem was to replace all instances of #LoadModule with LoadModule in all my Apache config files. Since that solved the problem for me, therefore I knew that my problem was not a missing "KeepAlive" directive argument, but rather, my problem was a missing dependency.

Because, remember, .so files are basically static libraries. Having a module enabled does not mean it will get used, but having one disabled does mean that it can't get used, and therefore, anything that depends on it will necessarily fail.

Note: this answer has received some down-votes due to the fact that my initial answer seemed to suggest leaving all the modules enabled, forever. While you could theoretically do that without necessarily breaking anything, it's obviously not a best practice solution.

So, please understand, I am merely suggesting this as a troubleshooting step, not a final solution.

Also, please note: I use a special git project to track all of my local machine's apache config files. That way I can do these sorts of global search-and-replace operations in my apache config working directory, as a troubleshooting step. If enabling all modules succeeds, then try disabling them again one-by-one and restarting apache in between, until you find which module it is that needs to remain enabled. Once you've figured that out, then reset the repo back to its original state, and only enable that one module that needs to remain enabled.

You will also find that using git to track your apache config files cleans up those directories, since you won't be needing those old-fashioned .bak and .default files anymore.

CommaToast
  • 147
  • 5