37

I'm having some serious disk performance problems while setting up a KVM guest. Using a simple dd test, the partition on the host that the qcow2 images reside on (a mirrored RAID array) writes at over 120MB/s, while my guest gets writes ranging from 0.5 to 3MB/s.

  • The guest is configured with a couple of CPUs and 4G of RAM and isn't currently running anything else; it's a completely minimal install at the moment.
  • Performance is tested using time dd if=/dev/zero of=/tmp/test oflag=direct bs=64k count=16000.
  • The guest is configured to use virtio, but this doesn't appear to make a difference to the performance.
  • The host partitions are 4kb aligned (and performance is fine on the host, anyway).
  • Using writeback caching on the disks increases the reported performance massively, but I'd prefer not to use it; even without it performance should be far better than this.
  • Host and guest are both running Ubuntu 12.04 LTS, which comes with qemu-kvm 1.0+noroms-0ubuntu13 and libvirt 0.9.8-2ubuntu17.1.
  • Host has the deadline IO scheduler enabled and the guest has noop.

There seem to be plenty of guides out there tweaking kvm performance, and I'll get there eventually, but it seems like I should be getting vastly better performance than this at this point in time so it seems like something is already very wrong.

Update 1

And suddenly when I go back and test now, it's 26.6 MB/s; this is more like what I expected w/qcrow2. I'll leave the question up in case anyone has any ideas as to what might have been the problem (and in case it mysteriously returns again).

Update 2

I stopped worrying about qcow2 performance and just cut over to LVM on top of RAID1 with raw images, still using virtio but setting cache='none' and io='native' on the disk drive. Write performance is now appx. 135MB/s using the same basic test as above, so there doesn't seem to be much point in figuring out what the problem was when it can be so easily worked around entirely.

poige
  • 9,171
  • 2
  • 24
  • 50
El Yobo
  • 926
  • 1
  • 8
  • 11
  • You didn't mention the distribution and software versions in use. – dyasny Jul 15 '12 at 07:42
  • Added some info on versions. – El Yobo Jul 15 '12 at 07:55
  • ah, as expected, ubuntu... any chance you can reproduce this on fedora? – dyasny Jul 15 '12 at 08:13
  • The server is in Germany and I'm currently in Mexico, so that could be a little tricky. And if it did suddenly work... I still wouldn't want to have to deal with a Fedora server ;) I have seen a few comments suggesting that Debian/Ubuntu systems did have more issues than Fedora/CentOS for KVM as much of the development work was done there. – El Yobo Jul 15 '12 at 08:18
  • my point exactly. and in any case, if you are after a server grade OS you need RHEL, not Ubuntu – dyasny Jul 15 '12 at 08:51
  • It's probably a whole new question and an invitation to a flame war, I'd be interest to know why you think that. My experience with RHEL and CentOS in the past has been negative, but mainly due to their limited repositories and out of date packages. Having used Debian for more than 10 years, I'd generally been happy with it as a server OS, but I thought I'd play around with Ubuntu this time. – El Yobo Jul 15 '12 at 15:44
  • And Debian doesn't have out of date packages?! Yep, you've invited a flame war all right. :) – Michael Hampton Jul 15 '12 at 16:13
  • Oh, Debian has definitely had their problems (three years between stable releases at one time, or thereabouts), so no flame war on that front :) But at least you could use the official backports to get reasonably recent versions on a largely stable system, and they've picked up their game since then. But that is one reason to use Ubuntu over Debian, more packages, more up to date. – El Yobo Jul 15 '12 at 16:20
  • Anyhow, it's a massive digression, I shouldn't have asked. If anyone cares to comment on http://serverfault.com/questions/91917/is-there-a-big-difference-in-working-with-fedora-and-debian-servers then perhaps that would be a more appropriate place to discuss. – El Yobo Jul 15 '12 at 16:24
  • No flame war meant really, I'm just seeing lots of these very hard to nail issues with the way virtualization works on ubuntu, compared to RHEL and Fedora. Usually, it's simple enough to reinstall another distro and see if the issue reproduces – dyasny Jul 15 '12 at 16:52
  • Yeah, I know :) Added new notes anyway, switched to raw on lvm as suggested and performance is where it should be. – El Yobo Jul 15 '12 at 17:01
  • Your Update 2 should be added as answer, I just did that because accepted answer is not the best for these who want to keep image files. – gertas Jun 15 '13 at 21:35
  • I'm not sure what you mean, sorry; my Update 2 is to do what was suggested in the accepted answer (use raw partitions with LVM). – El Yobo Jul 18 '13 at 08:04
  • Can you also provide data around write latency for small writes? ioping -W / – Federico Nov 19 '18 at 12:06

6 Answers6

19

Well, yeah, qcow2 files aren't designed for blazingly fast performance. You'll get much better luck out of raw partitions (or, preferably, LVs).

womble
  • 95,029
  • 29
  • 173
  • 228
  • 10
    Obviously, but they're also not meant to be quite as crap as the numbers I'm getting either. – El Yobo Jul 15 '12 at 07:52
  • [citation needed], as the wikipedians say. Who says they're not meant to be that crap? – womble Jul 15 '12 at 07:55
  • 1
    Most examples out there show similar performance with qcow2, which seems to be a significant improvement over the older version. The KVM site itself has come numbers up at http://www.linux-kvm.org/page/Qcow2 which show comparable times for a range of cases. – El Yobo Jul 15 '12 at 08:15
  • 2
    18:35 (qcow2) vs 8:48 (raw) is "comparable times"? – womble Jul 15 '12 at 08:19
  • That's the worst rate on the page, but even if you take that as the example and say that raw is appx. 2.5 times faster than qcow2... multiplying my 3MB/s by 2.5 gets you 7.5MB/s with raw and again, that's not exactly something you'd be happy with. – El Yobo Jul 15 '12 at 08:21
  • 1
    I've switched them over to LVM backed raw images on top of RAID1, set the io scheduler to noop on the guest and deadline on the host and it now writes at 138 MB/s. I still don't know what it was that caused the qcow2 to have the 3MB/s speeds, but clearly it can be sidestepped by using raw, so thanks for pushing me in that direction. – El Yobo Jul 15 '12 at 16:57
  • 1
    That is not quite true - latest patches in qemu speeds qcow2 a lot! We are almost on par. – lzap Nov 25 '13 at 20:43
  • Nice work. I'm not running the same infrastructure now, so I'm not in a good state to test it, but thanks for the feedback. – El Yobo Nov 25 '13 at 22:14
  • This answer is not correct. The qcow2 format does not leads to this level of inefficiency! There are other issues. – Federico Nov 19 '18 at 12:04
  • @Federico The question is from July 2012. While Ubuntu 12.04 LTS was released 6 days after the patches for support of qcow2 version 3 had been committed to the qemu repository, it is highly unlikely that these changes made it into the release. Version 2 was known to have poor performance: https://wiki.qemu.org/Features/Qcow3 – Joachim Wagner Jan 27 '22 at 12:00
12

How to achieve top performance with QCOW2:

qemu-img create -f qcow2 -o preallocation=metadata,compat=1.1,lazy_refcounts=on imageXYZ

The most important one is preallocation which gives nice boost, according to qcow2 developers. It is almost on par with LVM now! Note that this is usually enabled in modern (Fedora 25+) Linux distros.

Also you can provide unsafe cache if this is not production instance (this is dangerous and not recommended, only good for testing):

<driver name='qemu' cache='unsafe' />

Some users reports that this configuration beats LVM/unsafe configuration in some tests.

For all these parameters latest QEMU 1.5+ is required! Again, most of modern distros have these.

lzap
  • 2,704
  • 2
  • 22
  • 22
  • 3
    It is **not** a good idea to use cache=unsafe: an unexpected host shutdown can wreak havoc on the _entire_ guest filesystem. It is _much_ better to use cache=writeback: similar performance, but much better reliability. – shodanshok Apr 02 '15 at 17:18
  • 2
    As I've said: *if this is not production instance (good for testing)* – lzap Apr 03 '15 at 13:11
  • Fair enough. I missed it ;) – shodanshok Apr 03 '15 at 15:34
9

I achieved great results for qcow2 image with this setting:

<driver name='qemu' type='raw' cache='none' io='native'/>

which disables guest caches and enables AIO (Asynchronous IO). Running your dd command gave me 177MB/s on host and 155MB/s on guest. The image is placed on same LVM volume where host's test was done.

My qemu-kvm version is 1.0+noroms-0ubuntu14.8 and kernel 3.2.0-41-generic from stock Ubuntu 12.04.2 LTS.

gertas
  • 1,007
  • 10
  • 11
  • 9
    You set a qcow2 image type to "raw"? – Alex Aug 18 '13 at 11:02
  • I guess I have copied older entry, I suppose speed benefits should be same for `type='qcow2'`, could you check that before I edit? I have no more access to such configuration - I migrated to LXC with `mount bind` directories to achieve real native speeds in guests. – gertas Aug 18 '13 at 11:15
5

If you're running your vms with a single command, for arguments you can use

kvm -drive file=/path_to.qcow2,if=virtio,cache=off <...>

It got me from 3MB/s to 70MB/s

Kourindou Hime
  • 51
  • 1
  • 1
5

I experienced exactly the same issue. Within RHEL7 virtual machine I have LIO iSCSI target software to which other machines connect. As underlying storage (backstore) for my iSCSI LUNs I initially used LVM, but then switched to file based images.

Long story short: when backing storage is attached to virtio_blk (vda, vdb, etc.) storage controller - performance from iSCSI client connecting to the iSCSI target was in my environment ~ 20 IOPS, with throughput (depending on IO size) ~ 2-3 MiB/s. I changed virtual disk controller within virtual machine to SCSI and I'm able to get 1000+ IOPS and throughput 100+ MiB/s from my iSCSI clients.

<disk type='file' device='disk'>
   <driver name='qemu' type='qcow2' cache='none' io='native'/>
   <source file='/var/lib/libvirt/images/station1/station1-iscsi1-lun.img'/>
   <target dev='sda' bus='scsi'/>
   <address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
Greg W
  • 51
  • 1
  • 1
4

On old Qemu/KVM versions, Qcow2 backend was very slow when not preallocated, more so if used without writeback cache enabled. See here for more information.

On more recent Qemu versions, Qcow2 files are much faster, even when using no preallocation (or metadata-only preallocation). Still, LVM volumes remain faster.

A note on the cache modes: writeback cache is the preferred mode, unless using a guest with no or disabled support for disk cache flush/barriers. In practice, Win2000+ guests and any Linux EXT4, XFS or EXT3+barrier mount options are fines. On the other hand, cache=unsafe should never be used of production machines, as cache flushes are not propagated to the host system. An unexpected host shutdown can literally destroy guest's filesystem.

shodanshok
  • 44,038
  • 6
  • 98
  • 162