KVM/qemu - use LVM volumes directly without image file?

Question

I'm not quite sure how to phrase this question (hence the poor title), so let me provide an example of what I'm trying to do.

On my (old) Xen host, I'm able to present LVM filesystems directly to each guest. These filesystems are actually created and formatted on the host, and passed directly through. Eg., for one of my hosts using a separate tmp and swap partitions, I define the storage like this:

disk = [
'phy:/dev/vg1/guest1-swap,sda1,w',
'phy:/dev/vg1/guest1-disk,sda2,w',
'phy:/dev/vg1/guest1-tmp,sda3,w',
]

So, guest1-swap is formatted as a swap partition, guest1-disk and guest1-tmp are formatted with ext4, and from the guest's perspective it simply sees them as three formatted partitions under /dev/sda.

(This may sound like a lot of work, but there are provisioning scripts, such as the awesome xen-tools, that automated pretty much everything).

This provides some really useful capabilities, two of which I'm especially interested in figuring out for KVM:

• Mount the guest filesystems from the host OS. I can do a read-only mount of any guest filesystem at any time, even while the guest is running. This has the side benefit of allowing my to create LVM snapshots of any existing volume while the guest is running. This way, I'm able to centrally backup all my guests, while running, from the host.

• Online volume resizing. Because the volumes contain standard Linux filesystems, I can use a combination of lvextend and resize2fs to grow my guest filesystems, again while they're online.

I'm currently setting up a KVM host that will replace the Xen host. Similar to the Xen setup I'm leveraging LVM to provide direct filesystem access, but KVM/qemu behaves differently in that it always creates a image file for the guests, even on the LVM volume. From the guest's perspective, it sees this as an unpartitioned disk, and it's up to the guest to apply a partition label, then create the partitions and filesystems.

From a guest perspective that's fine, but from a server/management perspective it seems to be far less flexible than the Xen setup I described. I'm still new to KVM, so I may be (hopefully) missing something.

I ran into this problem when trying to re-implement my former backup solution on the KVM host and the mount command chocked when I tried to mount one of the guest's filesystems. So, addressing that is my current concern, but it also made me concerned about the resizing thing, because I'm sure that issue will come up at some point as well.

So, here are my questions:

1. Is there any way to have kvm/qemu use LVM volume filesystems directly as I described for my Xen setup? I use libvirt for management if that makes a difference.

2. If not, what can I do to get similar mounting/backup functionality under KVM? I've seen discussions about using libguestfs w/ FUSE to do this, but is that really the best option? I'd prefer to stick with a native filesystem mount if at all possible.

3. Also if not, is it possible to do an online filesystem resize under KVM? I've found several discussions/howtos about this, but the answers seem to be all over the place with no clear, and definitely no straightforward, solutions.

Sorry for the long post, just wanted to make sure it was clear. Please let me know if I can provide any other info that would be helpful. Looking forward to the discussion. :-)

Answer 1

1. qemu-kvm can use LVs as virtual disks instead of files. this is quite a common use case actually.
2. libguestfs (and just look for a set of virt-* tools) can provide access to guest filesystems in a cleaner way than anything you remount to the host directly, though both are possible.
3. Online FS resizing is not a feature of kvm, but something the guest OS should be capable of. resize2fs will work in a VM as well as it does on physical hardware, the only problem being the guest redetecting the size changes. Try virt-resize as the standard tool, but lvresize and qemu-img can also easily be used (though in offline mode, requiring a guest restart usually).

I think lvresize with resize2fs will actually work without a guest restart, but I haven't tried it yet

Answer 2

I use qemu-kvm+libvirt with exactly the configuration you're asking about, for the reasons you listed, but additionally because I get much better performance without the KVM host's filesystem layer in scope. If you add the VG as a 'storage pool' in virt-manager, you can create such VMs using its user-friendly wizard. (But I just write the XML by hand these days using an existing VM as a template).

Here's sanitised output of 'virsh dumpxml' for one of my guests:

<domain type='kvm'>
  <name>somevm</name>
  <uuid>f173d3b5-704c-909e-b597-c5a823ad48c9</uuid>
  <description>Windows Server 2008 R2</description>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-1.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Nehalem</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='rdtscp'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='acpi'/>
  </cpu>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/vg1/somevm'/>
      <target dev='hda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='bridge'>
      <mac address='00:00:00:00:00:00'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <video>
      <model type='vga' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none' model='none'/>
</domain>

Another thought (not relevant to your question but it might help): if you can, make sure you're using the 'paravirtualised' network, block, random, clock etc drivers - they're significantly faster than the fully virtualised ones. This is the "model=virtio" stuff above. You have to load driver modules into the host's kernel such as virtio_net.

Here is output of 'virsh pool-dumpxml vg1':

<pool type='logical'>
  <name>vg1</name>
  <uuid>9e26648e-64bc-9221-835f-140f6def0556</uuid>
  <capacity unit='bytes'>3000613470208</capacity>
  <allocation unit='bytes'>1824287358976</allocation>
  <available unit='bytes'>1176326111232</available>
  <source>
    <device path='/dev/md1'/>
    <name>vg1</name>
    <format type='lvm2'/>
  </source>
  <target>
    <path>/dev/vg1</path>
    <permissions>
      <mode>0700</mode>
    </permissions>
  </target>
</pool>

Answer 3

I don’t know of a way of exactly replicating the Xen behaviour you describe. However, you can use kpartx to expose the partitions within an LV that contains a whole-disk image as block devices on the host, which you can then mount, etc.

Answer 4

See my answer to my own question on this issue at KVM booting off-image kernel and existing partition. In short, getting virt-install to create a config for this is pretty straight-forward, given a slight modification of guest /etc/fstab.