I am facing a performance issue with my small virtualization environnment.
I have two Dell PowerEdge 2950 and a Qnap TS-453-BU. The Dells are equipped with XenServer 6.5.0 and they are attached to an iSCSI target with a pair of LUNs on the Qnap.
So far I installed several VM with no issues. Recently I installed a few debian 8 with a LVM partitioning and i faced some performance issues.
To analyze them, i installed two twin vms, with debian 8, one with LVM (testlvm) and one without (testnolvm) and i ran some bonnie++ tests:
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
testnolvm 4G 89 99 47597 25 29559 26 127 99 95660 41 443.2 23
Latency 190ms 520ms 575ms 127ms 267ms 1584ms
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
testlvm 4G 85 99 48245 27 30707 27 120 99 94637 45 491.7 22
Latency 214ms 203ms 589ms 241ms 189ms 411ms
Version 1.97 ------Sequential Create------ --------Random Create--------
testnolvm -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 12529 77 +++++ +++ 20241 89 16491 90 +++++ +++ 20389 90
Latency 2099us 637us 777us 1013us 270us 522us
Version 1.97 ------Sequential Create------ --------Random Create--------
testlvm -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 14261 86 +++++ +++ 19722 90 16002 91 +++++ +++ 12159 56
Latency 29901us 580us 718us 232us 1236us 602us
I see very similar results on the throughputs but different in the latency. I am not able to figure out what is wrong, because i also made other tests (mixing debian 7, debian 8, lvm, no lvm scenarios) getting confusing results.
It is a fact that i have a debian 7 with no lvm (installed at least one year ago) and a recently installed debian 8 with lvm that have very different performance;
for example, timing a "cat -file-" (i tried /etc/issue but also /proc/diskstats, so both real fs and memory) gives me a real of 0.005s on the first machine and 10 times more on the second machine.
Another example is that activating (on the second machine) a zabbix agent that monitors disk performance (making a lot of cat and grep on the /proc/diskstats) brings the cpu util to 50% without any other load (while on the old deb7 vm is almost seamless).
I even tried to move this "slow" vm on a new (thick provisioned) LUN without any improvement.
Any hints? Thanks in advance.
