2

I have Apache httpd (2.2.22) configured as reverse proxy.

There is a situation in which I need to take my web server (the receiver of the proxied requests) offline for a few milliseconds. During this time the web server should not receive any requests but I also don't want to reject any requests.

What I'd like to do is to somehow get httpd to delay all requests for that period of time. In steps:

  1. tell httpd to delay all requests until further notice
  2. stop web server
  3. start web server
  4. continue request forwarding from httpd

A fixed delay would also do the trick but since I cannot foresee the actual amount of time the web server will be offline (other than that it will be less than one second), a dynamical approach would be better suited.

I've looked at how mod_proxy or mod_balancer could help me but I didn't find an obvious solution.

I'll be happy about any pointers you can give me.

Edit:

It looks like a static approach is sufficient. Some helpful resources:

Alternative solution:

We will be deploying systemd to our servers. systemd will solve my problem because it can retain requests on sockets that are closed on one end. This means that when I stop the backend server for a short period of time, all the requests will be queued until I start it again and connect to the socket. That's what I call elegant :)

Community
  • 1
Max Leske
  • 93
  • 1
  • 8
  • 1
    What about requests which were in progress at the moment you decided to stop the web server? Either these requests will be aborted (bad for clients), or the stop procedure needs to wait for their completion (may take significantly longer than “a few milliseconds” — note that Apache, unlike nginx, does not fully buffer proxied requests, therefore you will need to wait for slow clients), or your web server must support a graceful restart procedure similar to Apache (in which case you will not need to do anything extra to avoid losing requests). – Sergey Vlasov Apr 21 '13 at 19:39
  • Good thought. Luckily the web server does restart gracefully, so I really only need to worry about new requests coming in. – Max Leske Apr 22 '13 at 18:42
  • I'd just setup caching and serve stale content during maintenance with expires settings. Or, use mod_proxy_balancer, which would handle several backends without serving error pages. – Marcel Apr 25 '13 at 21:41
  • @Marcel as explained, balancing is not an option. Neither is caching; all content (except for images etc.) is generated dynamically for every request. – Max Leske Apr 27 '13 at 14:23

4 Answers4

2

The ProxyPass directive accepts many parameters to configure how the connection is handled with the backend server.

Among those parameters, you may be interested by:

  • connectiontimeout <n>: The number of seconds Apache waits for the creation of a connection to the backend to complete.
  • timeout <n>: The number of seconds Apache waits for data sent by/to the backend.
  • ttl <n>: Time to live for inactive connections and associated connection pool entries, in seconds.

A better solution would be to have multiple backend servers and balance the load over the members and detect offline servers using the pingparameter. So, when you reboot a backend server, another one could take the relay.

Spack
  • 1,594
  • 13
  • 22
  • In this particular case I don't have the option to use multiple balancer members; there can only be one. Therefore, traditional balancing is not an option (no active balancer members would result in httpd returning a 500 or 503 I assume). The ProxyPass options you mentioned might be helpful, I'll have to try them out and see how that would work. I'll accept / award bounty as soon as I can confirm that this is a working solution. Thanks! – Max Leske Apr 21 '13 at 11:29
  • Using ProxyPass looks very promising. The default timeouts are already large enough (ProxyTimeout default is 300 seconds) so that I don't even need the dynamic approach. – Max Leske Apr 22 '13 at 18:53
1

Instead of Apache, you can use Iptables on your webserver to obtain this in the following way:

  1. drop new connections (tcp-SYN's) to webserver
  2. restart webserver
  3. re-accept all connections

As new connection's are silently dropped by the firewall, the reverse proxy will keep on re-sending tcp-syn's to the web-server. Established connections won't be impacted.

If the cycle is completed in less that one second, new clients will see a delay of (exactly) 1 second, equal to the delay to the first SYN retry

Example iptables config (say the INPUT chain has an empty ACCEPT policy):

/sbin/iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A INPUT -p tcp -m conntrack --ctstate NEW -m tcp --dport 80 -j DROP

after restarting the webserver:

/sbin/iptables -F

You could also apply the policy on OUTPUT chain of the reverse proxy, but quick synchronization of the actions might be more challanging.

Zabuzzman
  • 733
  • 10
  • 25
0

A low-tech approach, similar to the iptables proposal but completely transparent to the clients: Just stop apache with kill -stop and restart it with kill -cont. Depending on the amount of incoming packets the kernel should buffer everything for this short time. Probably this buffering can even be configured.

Hauke Laging
  • 5,157
  • 2
  • 23
  • 40
  • Interesting. But usually there will be several httpd processes running at the same time. One would have to figure out which process to suspend or suspend all of them. Since I don't have any knowledge about the internal workings of httpd I'd rather not mess with the processes. – Max Leske Apr 27 '13 at 14:34
  • @MaxLeske It would probably be necessary (or at least useful) to stop all Apache processes as all (but the listening one first) of them create traffic to the real webserver. AFAIK there is no relevant "internal working". If a new connection is requested then httpd forks. And even if they should communicate (for e.g. logging): Outside a real-time environment a process never knows how long it will take to get a response from the other one. Should be easy to find out whether this causes any problems. – Hauke Laging Apr 27 '13 at 14:46
0

If it's only meant to be in stasis for a few milliseconds, just use a graceful restart of Apache. It already has these properties

  • in-progress requests one existing connections can complete
  • new connections will sit in the listen backlog for a second or so
covener
  • 1,665
  • 9
  • 15
  • I suppose this is similar to using `kill -stop` and `kill -cont` as suggested by @Hauke Laging but on a more abstract level. Good point. – Max Leske Apr 27 '13 at 14:38