Apache buffers tomcat responses, and closes connection to tomcat when request size exceeds 10397232 bytes

Question

I've deployed a web-service application as a Docker container based on tomee:8-jre-7.0.4-plume which is sitting behind a apache httpd:2.4 instance running in another Docker container running on the same machine.

I'm having the following issues a) apache is buffering the response which creates significant latency and b) if the amount of data is too big (10397232 bytes) apache closes the connection with TomEE which reports a org.apache.catalina.connector.ClientAbortException: java.net.SocketTimeoutException. I think this corresponds with a message I see in the apache error_log (LogLevel trace6) output_filter: flushing because of THRESHOLD_MAX_BUFFER. This happens regardless of whether I use https or http.

If I send requests directly to TomEE, then no response buffering occurs and no errors occur with large requests.

I am including some code which is able to reproduce the problem. You'll notice that I'm basically piping the request input to 'cat', then piping it's output to another 'cat'. then copy the output of the final 'cat' output to the response output stream. In the 'real' application, I'm not using cat and am doing something much more interesting, but this example illustrates the crux of the problem. Also, if I modify my code to write the output of the final 'cat' to a local file, then simply copy the contents of that file to the response output stream, then the problems go away, however, this introduces the same sort of latency as buffering the entire response.

I've tried a number of configuration changes in apache httpd.conf

<IfModule proxy_ajp_module>
    ProxyPass "/" "http://localhost:8080/" timeout=18000
    TimeOut 18000
</IfModule>

I tried various combinations of the following:

    ProxyPass "/" "http://localhost:82/" timeout=18000 flushpackets=on receivebuffersize=8192 iobuffersize=8192

I also tried explicitly setting a number of TomEE/Tomcat config settings as mentioned in the documentation. I am not using the ajp protocol at present.

The only server.xml change that is currently in place is:

<Connector port="8080" allowTrace="true" ajpFlush="false" maxPostSize="-1" maxSavePostSize="-1"  proxyName="mycustomhost.com" protocol="HTTP/1.1"

I'm including representative webservice code here:

package com.spike

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.atomic.AtomicBoolean;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.core.Context;

@Path("test")
public class MinTestingResource {
    private final ExecutorService execSvc = Executors.newFixedThreadPool(10);
    @POST
    public void post(
            @Context HttpServletRequest request,
            @Context HttpServletResponse response) throws IOException, InterruptedException, ExecutionException {
        InputStream in = request.getInputStream();
        OutputStream out = response.getOutputStream();
        convertStream(in, out);
    }
    private void convertStream(InputStream in, OutputStream out) throws ExecutionException, IOException, InterruptedException {
        Process p1 = new ProcessBuilder("cat").start();
        Process p2 = new ProcessBuilder("cat").start();
        copyInThread("in-p1", in, p1.getOutputStream());
        copyInThread("p1-p2", p1.getInputStream(), p2.getOutputStream());
        copy("p2-out", p2.getInputStream(), out);
    }
    private Future<Long> copyInThread(String nameToLog, InputStream in, OutputStream out) {
        AtomicBoolean copierStarted = new AtomicBoolean(false);
        final Future<Long> future = execSvc.submit(() -> {copierStarted.set(true); return copy(nameToLog, in, out);});
        waitForStart(nameToLog, copierStarted);
        return future;
    }
    private long copy(String nameToLog, InputStream in, OutputStream out) throws IOException {
        long copied = 0;
        try {
            byte[] buf = new byte[8192];
            for (int bytesRead = 0; (bytesRead = in.read(buf)) > 0;) {
                out.write(buf, 0, bytesRead);
                copied += bytesRead;
            }
            return copied;
        } finally {
            out.close();
        }
    }
    private void waitForStart(String nameToLog, AtomicBoolean copierStarted) {
        pause(5);
        while (!copierStarted.get()) {
            pause(5);
        }
    }
    private void pause(long millis) {
        try {
            Thread.sleep(millis);
        } catch (InterruptedException ignore) {
        }
    }
}

I found an the following issue on the apache bugzilla Bug 61616 - mod_proxy_connect: stall and connection loss on bi-directional traffic but am not sure that this is the same problem, and it's not clear that a solution is available as a Docker container.

Apache buffers tomcat responses, and closes connection to tomcat when request size exceeds 10397232 bytes

0 Answers0