31

I am using a baseimage and based on that creating many VMs. And now I want to know which is better, qcow2 or raw to use for a baseimage. Moreover, can you please tell me if there is any advantage of using this baseimage thing, instead of cloning the whole disk. Speed can be one factor but in term of efficiency is there any problem in using a baseimage and then creating VMs using that baseimage ?

Edit 1:

I performed some experiments and got enter image description hereenter image description hereenter image description here

First one is when both baseimage and overlay are qcow2. Second When baseimage is raw but the overlay is qcow2 and in third case I am giving individual raw disk image to each VM. Surprisingly, last case is much more efficient as compared to the other two.

Experimental Setup: 
OS in baseimage : Ubuntu Server 14.04 64 bit.
Host OS: Ubuntu 12.04 64bit
RAM : 8GB
Processor : Intel® Core™ i5-4440 CPU @ 3.10GHz × 4 
Disk : 500 GB

On x-axis : Number of VM booted simultaneously. Starting from 1 and incremented upto 15.

On y-axis : Total Time to boot "x" number of machines.

From the graphs, it seems that giving full disk image to VM is much more efficient then other 2 methods.

Edit 2: enter image description here

This is for the case when we are giving individual raw image to each VM. After doing cache flushing, this is the graph. It is almost similar to the raw baseimage + qcow overlay.

Thanks.

A-B
  • 563
  • 2
  • 6
  • 17
  • 1
    I'd use a raw base-image and overlay QED instead of qcow2. QED is faster than qcow2 and less likely to be corrupted by design. – hookenz Mar 23 '15 at 20:12
  • @Matt please comment on the edited question. – A-B Mar 23 '15 at 22:55
  • The performance differences are interesting. Can you try it with qed? might be a contention/locking issue in the qcow2 implementation. I believe it has some problems which is why they invented qed. – hookenz Mar 23 '15 at 23:04
  • 1
    @Matt I checked with raw baseimage-qed overlay and raw baseimage-qcow2 overlay. But qed is much slower than qcow2 in this setting. – A-B May 05 '15 at 04:50

2 Answers2

20

For your specific use case (base image + qcow2 overlay), the RAW format should be preferred:

  1. It's faster: as it has no metadata associated, it is as fast as possible. On the other hand, Qcow2 has two layer of indirection that must be crossed before to hit the actual data
  2. As the overlay layer must be a Qcow2 file, you don't lose the ever-useful snapshot capability (RAW images don't support snapshots by themselves)

The choice between base image + qcow2 overlay vs multiple full copies depends on your priority:

  1. For absolute performance, use fallocated RAW images. This has the downside of not supporting snapshot, with in most environments is a too high price to pay
  2. For flexibility and space-efficiency use RAW base images + Qcow2 overlays.

Anyway, I found Qcow2 files somewhat fragile.

For my production KVM hypervisors I basically use two different setups:

  1. where performance is #1 I use LVM volumes directly attached to the virtual machines, and I use LVM snapshot capability to take consistent backups
  2. where I can sacrifice some performance for enhanced flexibility, I use a single, big LVM Thin Provisioned Volume + XFS + RAW images

Another possibility is to use a normal LVM volume + XFS + RAW images. The only downside is that normal (non-thin) LVM snapshots are very slow and snapshotting a busy normal LVM volume will kill performance (for the lifetime of the snapshot). Anyway, if you plan to use only a sporadic use of snapshots, this can be the simpler and safer bet.

Some references:
KVM I/O slowness on RHEL 6
KVM storage performance and Qcow2 prellocation on RHEL 6.1 and Fedora 16
KVM storage performance and cache settings on Red Hat Enterprise Linux 6.2
LVM thin volume explained

shodanshok
  • 44,038
  • 6
  • 98
  • 162
  • As stated above: individual RAW images are going to be faster due to no indirection layers. Remember that you are giving up snapshots (at image level) and space efficiency, however. Moreover, I'm under the impression that your third result was skewed by host cache. To be sure, redo all your tests issuing "sync; echo 3 > /proc/sys/vm/drop_caches" between each run. – shodanshok Mar 24 '15 at 07:35
  • Actually I was thinking the same. But then I thought that, in case of actual use also it will follow the same flow. – A-B Mar 25 '15 at 07:25
  • Not really: In this test environment, you are simply booting your virtual machines, without using them. On a real case, after booting the virtual machines will be used, polluting the cache. In general, is best to benchmark with a reasonable use case. – shodanshok Mar 25 '15 at 08:14
  • Have you got any idea why caching is not playing that much role in graph 1 and graph 2 i.e Qcow2 base-image + Qcow2 overlay and Raw base-image + Qcow2 overlay. – A-B May 04 '15 at 05:09
8

Please be advised....if you are using linux you can use raw and get the same benifits of qcow2 as far as size goes.

...If your file system supports holes (for example in ext2 or ext3 on Linux or NTFS on Windows), then only the written sectors will reserve space.

https://docs.fedoraproject.org/en-US/Fedora/18/html/Virtualization_Administration_Guide/sect-Virtualization-Tips_and_tricks-Using_qemu_img.html

raw Raw disk image format (default). This can be the fastest file-based format. If your file system supports holes (for example in ext2 or ext3 on Linux or NTFS on Windows), then only the written sectors will reserve space. Use qemu-img info to obtain the real size used by the image or ls -ls on Unix/Linux.

d hee
  • 220
  • 3
  • 13
  • Good point! But is seems in backup/copying/compression the whole size of the raw image-file is used. A 20GB raw file only allocating 4MB on disk was zipped to 20MB – MrCalvin Apr 03 '18 at 15:58
  • This was completely new for me. I have a VM in Proxmox VE with a virtual disk of 30 GB ... and I can see that the disk has the size in the host file system. But the host file system has a over all usage of << 10GB. Now that´s the reason what I was looking for. – cljk Aug 29 '19 at 05:39