Virtualbox HTTP load testing, host CPU overload issues

Question

I'm doing HTTP load testing benchmarks (using Apache Benchmark and Siege) on a small Java EE 1.7.0 / Tomcat 7.0.26 application running on a Debian Squeeze 6.0.4 x64 virtualized with Virtualbox 4.1.8. The computer host is Ubuntu 11.10 x64.

I've modified those parameters in the Tomcat server.xml :

<Connector 
    port="8080" 
    protocol="HTTP/1.1"
    connectionTimeout="200000"
    redirectPort="8443" 
    acceptCount="2000"
    maxThreads="150"
    minSpareThreads="50" />

The application executed on the server takes around 300ms.

This app is running well until a certain amount of concurrent connections like those one :

ab -n 500 -c 150 http://xx.xx.xx.xx:8080/myapp/
ab -n 1000 -c 50 http://xx.xx.xx.xx:8080/myapp/
siege -b -c 100 -r 20 http://xx.xx.xx.xx:8080/myapp/

A lot of socket connection timed out happens and this completly overload the host processor (but the CPU load inside the VM is normal).

Doing an htop on the host, i can see that the Virtualbox processus is running under 300% CPU and never come down even after the load test is finished. (I've allocated 4 processors to the VM, if I allocate only one processor, CPU load goes under 100%).

Restarting Tomcat don't do anything, i'm forced to restart the whole VM.

I've tryed to launch those ab/siege commands locally on the VM and everything goes well.

I first thought it was related to a linux network limit as explained here: Running some benchmarks using ab, and tomcat starts to really slow down So I've modified those TCP parameters :

echo 15 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 30 > /proc/sys/net/ipv4/tcp_keepalive_intvl
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

It seems to be better, but it continues to overload the host CPU and output socket connections time out at a certain amount of concurrent connections.

I'm wondering if this is not related to how Virtualbox handles external concurrent connections.

So how did you set up the network for the VM? NAT, Bridge, ...? — Nils, Mar 04 '12 at 21:42
Yes, wireless. While it can be the cause of `socket connections time out`, i doubt this is the source of the CPU overload. — aschuler, Mar 06 '12 at 16:07

score 2 · Answer 1 · answered Nov 10 '12 at 02:02

I've seen this problem before when you have a loop that causes your network traffic to be re-sent forever. Try checking to see if you're getting a absurdly high amount of traffic going through any of your network interfaces.

If that doesn't work, does the CPU usage stop when the guest OS is stopped, or do you need to shutdown the whole VM container?

score 1 · Answer 2 · answered Nov 10 '12 at 04:47

First of all you don't say anything about the VBox host. How powerful is it? When I do VBox load testing I use a machine with 12 cores and 32G of RAM to run a virtual server with 2 to 8 cores and up to 12G of RAM. In otherwords, the underlying host is more powerful than the VBox guest so I am reasonably sure that the guest has dedicated cores and RAM.

Next, I tune both the host and the guest OSes to handle load. Things like the max open files, max sockets being opened, and various TCP/IP settings like buffers. These need to be set right to do load testing. If you graph your data with a range of loads, you will see it get suddenly much worse when you hit an OS limit. Fix this before you load test your app.

score 0 · Answer 3 · answered Mar 08 '12 at 13:03

0

Are you testing from within the VM? Or from elsewhere? If its the VM try pointing to 127.0.0.1 instead for the testing.

ab -n 500 -c 150 http://127.0.0.1:8080/myapp/

answered Mar 08 '12 at 13:03

SuperBOB

450
3
7

I've tried both. As i said, local tests are successful. This is when I launch thoses tests outside the VM (from host or another network), CPU overload happens. – aschuler Mar 08 '12 at 15:42

Virtualbox HTTP load testing, host CPU overload issues

3 Answers3