Spring Boot 1.2.7, Tomcat, CSS and Thymeleaf = Very Very slow service

Question

I have a Spring Boot 1.2.7.RELEASE web application (with Tomcat embedded) which needs to handle a very high load spike, ~ 10k connections in 4 to 5 minutes on one server (there is actually a cluster of servers but this is the load one node will see).

Its really, really slow. If I hit it with JMeter with 1,000 threads in 10 seconds, but just a simple GET request for the Thymeleaf login page - nothing else - The average response will soon climb in excess of 13 seconds. The page is just a log in form, using Thymeleaf.

This is running on a VM with RHEL7, 32GB RAM (with the JVM heap set to use 28GB of it) and 4 CPU cores. There is plenty of horsepower here, yet I'm struggling to get it to respond under load.

As a test, in an effort to reduce the number of sockets and threads, I commented out two links in the page to css:

<HEAD>

    <!-- Other meta stuff omitted -->


    <!-- I commented out these next two lines -->
    <link rel="stylesheet" href="css/index.css"/>
    <link href="css/gridset.css" rel="stylesheet"/>
</HEAD>

When I did this the exact same load test it yielded an average response of 277ms! If I put either back, the slowness returns.

So without exaggeration, the response for the same test went from 13 seconds to less than 0.3 seconds.

I attempted to bring the css inline, but Spring throws all kinds of errors trying to parse it. The W3C Validator shows both files contain many errors.

Someone is working to correct the errors in the css, but I'm wondering what the cause of the slowness is. Is it the fact that the css is broken or is it the fact that its being served by Tomcat from the static area? I won't have valid css to retest for several days and I'm under the gun to make this work.

I hit it with firebug, and the single login page loads in 437ms. The first index.css file loads in 158ms and the gridset.css file 53ms. There are then 3 images that take a total of 205ms.The total size of the images is 19kb.

I'll post a thread dump below. This was generated by New Relic.

I found some info about Tomcat being slow to serve static content, but I can't imagine its this slow, and other people say that a recent version of Tomcat does as well as httpd.

I'm primarily a developer, so any assistance in fixing this up is greatly appreciated, thank you!

75% java.lang.Thread.run() :745 70% org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() :61
 70% java.util.concurrent.ThreadPoolExecutor$Worker.run() :617
 70% java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor.Worker) :1127,1142
 69% java.util.concurrent.ThreadPoolExecutor.getTask() :1066
 69% org.apache.tomcat.util.threads.TaskQueue.poll(long, java.util.concurrent.TimeUnit) :31
 69% org.apache.tomcat.util.threads.TaskQueue.poll(long, java.util.concurrent.TimeUnit) :85
 69% java.util.concurrent.LinkedBlockingQueue.poll(long, java.util.concurrent.TimeUnit) :462,474,467
 69% java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) :2069,2083,2078
 69% java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) :215
 69% sun.misc.Unsafe.park(boolean, long) :native

Hello Jim, I may be experiencing a very similar problem, did you ever find a solution/root cuase for this? — rich green, May 23 '16 at 15:25
From the best I recall, it was a combination of things. First, we moved the assets to Cloudfair. But also, I needed to increase the number of processes for my user. It turned out that the server (RHEL) was not able to create threads because I had too few set. I did manage to get killer performance out of it! — Jim Archer, May 23 '16 at 21:31

Spring Boot 1.2.7, Tomcat, CSS and Thymeleaf = Very Very slow service

0 Answers0