Convert directory to QEMU/KVM virtual disk image

Question

I have a directory filled with data at /var/backups/disk1 that I want to convert into a virtual disk image which I'll then be able to boot using QEMU or KVM (the directory contains the file system for a virtual machine, copied out via rsync).

While there are plenty of instructions out there for converting a complete physical disk to a virtual disk, packaging up only the contents of a single directory into a virtual disk image is turning out to be much more difficult than I expected. Any ideas?

By the way, I know I can use qemu-img to convert a block device into a virtual disk (e.g., qemu-img convert -f /dev/sdc -O qcow2 disk.qcow2), so if only I could get the directory /var/backups/disk1 to appear to be a block device, then theoretically I should be able to accomplish my goal using qemu-img. I've thought about creative ways to expose the directory as a block device using NBD or a loopback device, however, and have not had success.

Answer 1

First, create a raw image of the required size. I'll assume 10G is enough. Using seek creates a sparse file, which saves space.

dd if=/dev/null of=example.img bs=1M seek=10240

Next, create a filesystem on it.

mkfs.ext4 -F example.img

(Note that you need the -F option for mkfs.ext4 to operate on a file as opposed to a disk partition)

Then, mount it.

mkdir /mnt/example
mount -t ext4 -o loop example.img /mnt/example

Now you can copy your files to /mnt/example. Once this is done, unmount it and you can use example.img as a drive in a virtual machine. If you want you can convert it from a raw image to another format like qcow2e using qemu-img, but this isn't required.

Answer 2

Seems like libguestfs can do it, see virt-make-fs(1) and maybe virt-copy-in(1).

Answer 3

virt-make-fs from libguestfs qcow2 example

https://serverfault.com/a/332114/163884 mentioned it, but here is a full example:

sudo apt-get install libguestfs-tools

# Workarounds for Ubuntu 18.04 bugs. See section below.
sudo rm -rf /var/cache/.guestfs-*
echo dash | sudo tee /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/zz-dash-packages
sudo chmod +r /boot/vmlinuz-*

mkdir sysroot
# Just a test file.
dd if=/dev/urandom of=sysroot/myfile bs=1024 count=1024
virt-make-fs --format=qcow2 --type=ext2 sysroot sysroot.ext2.qcow2 

Note how sudo is not required, except for installation and Ubuntu bug workarounds.

I then verified that QEMU can actually read it with:

qemu-system-x86_64 -drive file=sysroot.ext2.qcow2,format=qcow2,if=virtio,snapshot ...

I am then able to mount the image inside QEMU Linux and read the file.

virt-make-fs ext example

This awesome tool can also make raw ext filesystems, e.g.:

virt-make-fs --format=raw --type=ext2 sysroot sysroot.ext2
virt-make-fs --format=raw --type=ext4 sysroot sysroot.ext4

which we can verify directly on the host with:

mkdir -p mnt
dev="$(sudo losetup --show -f -P sysroot.ext4)"
sudo mount -o loop "$dev" mnt
cmp sysroot/myfile mnt/myfile

Image size minimization

A really good feature of virt-make-fs is that it automatically tries to minimize the image size if that is what we want:

Virt-make-fs defaults to minimizing the extra space, but you can use the --size flag to leave space in the filesystem if you want it.

so:

df -h

tells me that the image is 82% filled:

/dev/loop17    1.5M  1.1M  244K  82% /home/ciro/test/guestfs/mnt

We can easily add some extra space on top of the minimum with --size-=+:

virt-make-fs --format=raw --size=+8M --type=ext2 sysroot sysroot.ext2

ext4 journal overhead

The manual also mentions that:

Note that ext3 filesystems contain a journal, typically 1-32 MB in size. If you are not going to use the filesystem in a way that requires the journal, then this is just wasted overhead.

and it is interesting to verify that with:

du -bs *

which produces:

1052672 sysroot
1446297 sysroot.ext2
2599731 sysroot.ext4

so we see that the ext4 came out significantly bigger.

libguestfs Ubuntu bugs

Now for the downside: there appears to be no Ubuntu maintainer currently and the library is generally buggy on Ubuntu.

sudo is not in theory required, but it is required due to an Ubuntu packaging bug unless we do a workaround: https://askubuntu.com/questions/1046828/how-to-run-libguestfs-tools-tools-such-as-virt-make-fs-without-sudo/1046829#1046829

libguestfs: error: /usr/bin/supermin exited with error status 1.
To see full error messages you may need to enable debugging.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.
    libguestfs: error: /usr/bin/supermin exited with error status 1.

After that, without our workarounds, 18.04 (but not 16.04) fails with: https://bugzilla.redhat.com/show_bug.cgi?id=1591617

libguestfs: error: tar_in: write error on directory: /: 

due to a bug that was already fixed upstream.

Tested in Ubuntu 18.04, libguestfs-tools 1:1.36.13-1ubuntu3, QEMU 1:2.11+dfsg-1ubuntu7.3.

Answer 4

Using Michaels strategy in Fedora 23:

# dnf install -y libguestfs-tools-c
# virt-make-fs --format=raw --label mylabel the_directory myfs.raw

This creates ext2 file system, that you can mount to some directory:

# mount file.fs /mnt 

Now /mnt contains all your files

Answer 5

What about accessing the directory as a network share from a VM that is also mounted to your target virtual disk image? You than than just perform a simple copy from the share to the mounted disk image.

Answer 6

I prefer (and use) the answer by Ciro Santilli, but here's something that works with a guestfish shell:

# assumes extant directory "rootfs"
tar -cf rootfs.tar -C rootfs .
guestfish <<EOF
disk-create rootfs.qcow2 qcow2 "$((2 * $(stat -c%s rootfs.tar)))"
add-drive rootfs.qcow2
run
part-disk /dev/sda gpt
mkfs ext4 /dev/sda
mount /dev/sda /
tar-in rootfs.tar /
umount-all
exit
EOF