VM transfer speed between XCP-ng (XEN) servers

Question

I have two HP DL380 G8 servers with 4x 1TB on HP p420 RAID controllers in RAID 1+0 setup. Eth0s are connected to the router, and Eth3&Eth4s are bonded (LACP) and connected directly between machines.

If I run

#!/bin/bash
clear

echo 'Starting disk speed analysis..'
echo -e '\n  Reading different size files (1M, 100M, 1G):\n \e[93m'

dd if=/dev/sda of=/dev/zero iflag=direct bs=1M count=1000 &> test-results.log
tail -1 test-results.log

dd if=/dev/sda of=/dev/zero iflag=direct bs=100M count=10 &> test-results.log
tail -1 test-results.log

dd if=/dev/sda of=/dev/zero iflag=direct bs=1G count=1 &> test-results.log
tail -1 test-results.log

echo -e '\n  \e[39mWriting different size files (1M, 100M, 1G):\n \e[93m'
dd if=/dev/zero of=/root/testfile oflag=direct bs=1M count=1000 &> test-results.log
tail -1 test-results.log

dd if=/dev/zero of=/root/testfile oflag=direct bs=100M count=10 &> test-results.log
tail -1 test-results.log

dd if=/dev/zero of=/root/testfile oflag=direct bs=1G count=1 &> test-results.log
tail -1 test-results.log

rm test-results.log
echo -e '\e[39m'

I get :

Reading different size files (1M, 100M, 1G):
1048576000 bytes (1.0 GB) copied, 2.81374 s, 373 MB/s
1048576000 bytes (1.0 GB) copied, 1.98058 s, 529 MB/s
1073741824 bytes (1.1 GB) copied, 1.88088 s, 571 MB/s

Writing different size files (1M, 100M, 1G):
1048576000 bytes (1.0 GB) copied, 0.871918 s, 1.2 GB/s
1048576000 bytes (1.0 GB) copied, 3.08039 s, 340 MB/s
1073741824 bytes (1.1 GB) copied, 3.2694 s, 328 MB/s

and

Reading different size files (1M, 100M, 1G):
1048576000 bytes (1.0 GB) copied, 2.80229 s, 374 MB/s
1048576000 bytes (1.0 GB) copied, 2.50451 s, 419 MB/s
1073741824 bytes (1.1 GB) copied, 2.136 s, 503 MB/s

Writing different size files (1M, 100M, 1G):
1048576000 bytes (1.0 GB) copied, 1.64036 s, 639 MB/s
1048576000 bytes (1.0 GB) copied, 3.48586 s, 301 MB/s
1073741824 bytes (1.1 GB) copied, 4.5464 s, 236 MB/s

And this seems to be fair speeds, but if I try to migrate VM of 100Gb size to another machine over the bonded network, I only get ~60MB/s network transfer speed and a short transfer of 120MB/s if that WM is running at the time of transfer.

Network vs storage speed of a single VM transfer

However, storage I/O rates can go quite high.. way above the network speed, o I presume storage speed is not a problem.. right?

I am using XCP-ng Center, connected over VPN. Its a fresh install, XCP-ng is v7.6.

Ideally I would expect around 2x125 MB/s transfer speed between the servers, any ideas why this is not happening?

Maybe anyone with a similar stack could share the experience? Thanks!

Answer 1

Bonding wont help here as your source and destination IP addresses are fixed. LACP calculates a hash based on these IP addresses (and optionally also TCP port numbers) to determine which physical link to use. A single TCP session thus always puts packets from the same flow on the same physical link.

Gigabit ethernet interface use 8b/10b encoding so it can send at a maximum of 800Mbps, including layer 2, 3, and 4 overhead. Given the overhead of:

• 18 bytes of Ethernet header
• 20 bytes for an IP header; and
• 20 bytes for a TCP header

That gives 58 bytes of overhead per 1518-byte layer-2 frame or about 4 percent. Subtracting this from the 800Mbps (=100MB/s) gives us a maximum of 96MB/s. This still excludes the interpacket gap which is 96ns for gigabit ethernet.

The speeds you're getting aren't thus as abnormal as you might expect.

Edit to answer rjt's questions:

LACP is a protocol which creates a virtual interface that bonds together one or more (up to sixteen of which maximum eight can be active) physical interfaces. This protocol can be implemented in servers, routers and switches. However there is no requirement that one end of the link must be a switch.

LACP can not use round robin. Below is a quote from CCIE Routing & Switching v5.0, the official certification guide for the CCIE exam, page 155 (emphasis mine):

This hashing function is deterministic, meaning that all frames in a single flow produce the same hash value, and are therefore forwarded over the same physical link. Hence, the increase in the available bandwidth is never experienced by a single flow; rather, multiple flows have a chance of being distributed over multiple links, achieving higher aggregated throughput. The fact that a single flow is carried by a single link and thus does not benefit from a bandwidth increase can be considered a disadvantage; however, this approach also prevents frames from being reordered. This property is crucial, as EtherChannel---being a transparent technology---must not introduce impairments that would not be seen on plain Ethernet.