1

So we use apache as a reverse proxy to send request to our solr (search application) which is hosted in tomcat.

We have another server hosted in jboss which sends data in binary format to the apache, which in turns sends it to tomcat to be indexed and put into solr database. The data is sent as multi-parted POST request. This process can took up to 8 hours.

So here's the problem, when we send the data through apache, for about 30-40 minutes in the middle of the indexing (the sending data part), we will receive a lot of proxy and bad gateway error in the jboss (the one sending data), and in the tomcat log, it will say that the received data has an invalid EOF. So it's obvious that the data is not sent properly. And if we look at apache logs, there're a lot of this error:

[Wed Jan 08 16:10:45 2014] [error] [client 10.60.6.6] (70007)The timeout specified has expired: proxy: error reading status line from remote server localhost:8080
[Wed Jan 08 16:10:45 2014] [error] [client 10.60.6.6] proxy: Error reading from remote server returned by /application-path/application-url

and this error:

[Mon Jan 13 11:38:49 2014] [error] (103)Software caused connection abort: proxy: pass request body failed to [::1]:8080 (localhost)
[Mon Jan 13 11:38:49 2014] [error] proxy: pass request body failed to [::1]:8080 (localhost) from 10.3.40.76 ()

The strange thing is, if we bypass apache (:80) and send the data directly to tomcat (:8080) we never encountered this problem. At first, this problem might look the same as this.

But tomcat and apache doesn't have keep-alive turned on.

Timeout 120
KeepAlive Off

<VirtualHost *:80>
  ServerName application.com
  ServerAlias x.application.com
  DocumentRoot /var/www/something/
  ServerAdmin linux@application.com

  # Possible values include: debug, info, notice, warn, error, crit,
  # alert, emerg.
  LogLevel warn
  LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" \"%{soapAction}i\"" logging
  CustomLog /log/httpd/access.log logging
  ErrorLog /log/httpd/error.log

  ServerSignature On

  RewriteEngine On
  RewriteLogLevel 2

  # ensure that Tomcat sees the original host and port (and not the proxy-host and -port)
  ProxyPreserveHost On

  # Rewrite Rules 
  RewriteRule ^/application-path/(.*) http://localhost:8080/application-path/$1 [P]
</VirtualHost>

and in tomcat

<Connector connectionTimeout="10000" port="8080" protocol="HTTP/1.1"
redirectPort="8443" maxThreads="1000" processorCache="1000" maxKeepAliveRequests="1"
keepAliveTimeout="10" socket.soReuseAddress="true" socket.soKeepAlive="true"
compression="on" compressionMinSize="10"
compressableMimeType="text/html,text/xml,text/plain,application/javascript,application/json,text/javascript,text/css"
socket.soLingerOn="false" socket.soLingerTime="0" URIEncoding="UTF-8" />

And for those wondering, yes, apache has a greater timeout than tomcat, and both have keep-alive turned off, so it doesn't make sense why apache will return a timeout when tomcat doesn't.

Community
  • 1
Rowanto
  • 121
  • 1
  • 5

1 Answers1

1

We had a similar issue to yours, We bypassed Apache completely by using by using iptables to forward all traffic from port 80 to 8080. Not sure if this will work for you or not.

You could also just have tomcat listen on port 80. (Not sure if you have other sites listening on 80 or not.)

joeg1ff
  • 391
  • 3
  • 6