apache reverse proxy sometimes not working

Question

I am using a digital ocean droplet (Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-52-generic x86_64)) to host both jira and confluence. They are started on the same ip but different ports. I wanted to access them using jira.team.domain.com and confluence.team.domain.com so I went for the reverse proxy solution, using apache 2.4.7.

Things went really good and I had them working pretty fast. The issue is that after a couple of days, for a certain amount of time, the reverse proxy is not working and I get 'hostname not resolved' in the browser. I checked and the jira and confluence apps are accessible at the ip:port addresses (ports 8081 and 8091). After a while, don't know exactly how much, it starts working again.

The setup is the following:

Jira server.xml contains two connectors:

            <Connector port="8080"

               maxThreads="150"
               minSpareThreads="25"
               connectionTimeout="20000"

               enableLookups="false"
               maxHttpHeaderSize="8192"
               protocol="HTTP/1.1"
               useBodyEncodingForURI="true"
               redirectPort="8443"
               acceptCount="100"
               disableUploadTimeout="true"
               proxyName="jira.team.domain.com"
               proxyPort="80"/>

            <Connector port="8081"

               maxThreads="150"
               minSpareThreads="25"
               connectionTimeout="20000"

               enableLookups="false"
               maxHttpHeaderSize="8192"
               protocol="HTTP/1.1"
               useBodyEncodingForURI="true"
               redirectPort="8443"
               acceptCount="100"
               disableUploadTimeout="true"/>

Confluence server.xml also has 2 connectors:

<Connector port="8091" connectionTimeout="20000" redirectPort="8443"
            maxThreads="200" minSpareThreads="10"
            enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"
            protocol="org.apache.coyote.http11.Http11NioProtocol" />

<Connector port="8090" connectionTimeout="20000" redirectPort="8443"
            maxThreads="200" minSpareThreads="10"
            enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"
            protocol="org.apache.coyote.http11.Http11NioProtocol"
            proxyName="confluence.team.domain.com" proxyPort="80" />

and the /etc/apache2/sites-enabled/000-default.conf looks like this:

<VirtualHost *:*>
  ServerName localhost
  # DocumentRoot /var/www/html

  <Proxy *>
    Require all granted
  </Proxy>

  # ProxyRequests Off
  ProxyPreserveHost On
  ProxyPass / http://jira.team.domain.com:8080/
  ProxyPassReverse / http://jira.team.domain.com:8080/
</VirtualHost>

<VirtualHost *:*>
  ServerName confluence.team.domain.com
  DocumentRoot /var/www/html
  <Proxy *>
    Require all granted
  </Proxy>

  ProxyRequests Off
  ProxyPreserveHost On
  ProxyPass / http://confluence.team.domain.com:8090/
  ProxyPassReverse / http://confluence.team.domain.com:8090/
</VirtualHost>

Could anyone help me solve this issue?

score 0 · Answer 1 · answered Nov 03 '15 at 22:58

This sounds somewhat similar to issues I've had reverse proxying Atlassian applications characterized by intermittent issues with Apache while the Tomcat instances seem just fine. The "hostname not resolved" in the browser doesn't jive with that so perhaps not, but you can give these a shot. It won't hurt anything to try.

If the Apache server itself is occasionally having trouble doing a DNS query that would be pretty simple to resolve with the hosts file. Make sure the fqdn is in the /etc/hosts assigned to either the loopback (127.0.0.1) or whatever interface you have JIRA/Confluence listening on:

127.0.0.1 localhost jira.team.domain.com confluence.team.domain.com

If you assign it to the loopback you'll need to make sure JIRA and Confluence are listening on the loopback interface (or all of the interfaces).

A problem I've seen a few times (typically returning a 503 error to the user) is Apache detecting some kind of issue with the backend. When it does it backs off for 60 seconds by default. This can be a helpful clue that there's a resource contention issue with JIRA and/or Confluence so it shouldn't be ignored, but 60 seconds of downtime every time the backend gets flakey is a tad much. You can disable this by configuring the retry parameter on the ProxyPass directives:

<VirtualHost *:*>
  ServerName localhost
  # DocumentRoot /var/www/html

  <Proxy *>
    Require all granted
  </Proxy>

  # ProxyRequests Off
  ProxyPreserveHost On
  ProxyPass / http://jira.team.domain.com:8080/ retry=0
  ProxyPassReverse / http://jira.team.domain.com:8080/
</VirtualHost>

<VirtualHost *:*>
  ServerName confluence.team.domain.com
  DocumentRoot /var/www/html
  <Proxy *>
    Require all granted
  </Proxy>

  ProxyRequests Off
  ProxyPreserveHost On
  ProxyPass / http://confluence.team.domain.com:8090/ retry=0
  ProxyPassReverse / http://confluence.team.domain.com:8090/
</VirtualHost>

From the mod_proxy docs:

Connection pool worker retry timeout in seconds. If the connection pool worker to the backend server is in the error state, Apache httpd will not forward any requests to that server until the timeout expires. This enables to shut down the backend server for maintenance and bring it back online later. A value of 0 means always retry workers in an error state with no timeout.

https://httpd.apache.org/docs/2.4/mod/mod_proxy.html

If that doesn't work try using your browser's Developer Tools to see exactly what is or isn't being returned by Apache and what headers are being set. Taking a look at the Apache logs using tail -f /path/to/logs/goes/here while you mash the refresh button wouldn't hurt either.

score 0 · Answer 2 · answered Dec 03 '18 at 21:02

You don't even need to work with hostname in Apache conf in the part for backends / proxy. As far as you have correctly setup proxy part of "backends":

proxyName="jira.team.domain.com"
proxyPort="80"

... or in case of https ...

proxyName="jira.team.domain.com"
scheme="https"
proxyPort="443"

and similar setting for confluence (the option for http you have already done as you have mentioned in the question). So the ProxyPass can easily use just an IP:

<VirtualHost *:*>
  ServerName jira.team.domain.com
  ...
  ProxyPass / http://127.0.0.1:8080/
  ...
</VirtualHost>

<VirtualHost *:*>
  ServerName confluence.team.domain.com
  ...
  ProxyPass / http://127.0.0.1:8090/
  ...
</VirtualHost>

With this setting you need just dns record pointing to the host to prevent need of /etc/hosts records on client side and correct ServerName in Apache config to properly match with Virtual Host (you have just localhost for jira definition) and proper proxyName in Atlassian products for the correct response which is sent back to the user...

In case of this setting there is even no space for dns resolving on Apache as proxying layer so I would say you are bypassing the issue you have mentioned ;-).

apache reverse proxy sometimes not working

2 Answers2