How do I convert a Linux disk image into a sparse file?

Question

I have a bunch of disk images, made with ddrescue, on an EXT partition, and I want to reduce their size without losing data, while still being mountable.

How can I fill the empty space in the image's filesystem with zeros, and then convert the file into a sparse file so this empty space is not actually stored on disk?

For example:

> du -s --si --apparent-size Jimage.image 
120G Jimage.image
> du -s --si Jimage.image 
121G Jimage.image

This actually only has 50G of real data on it, though, so the second measurement should be much smaller.

This supposedly will fill empty space with zeros:

cat /dev/zero > zero.file
rm zero.file

But if sparse files are handled transparently, it might actually create a sparse file without writing anything to the virtual disk, ironically preventing me from turning the virtual disk image into a sparse file itself. :) Does it?

Note: For some reason, sudo dd if=/dev/zero of=./zero.file works when cat does not on a mounted disk image.

Answer 1

First of all, sparse files are only handled transparently if you seek, not if you write zeroes.

To make it more clear, the example from Wikipedia

dd if=/dev/zero of=sparse-file bs=1k count=0 seek=5120

does not write any zeroes, it will open the output file, seek (jump over) 5MB and then write zero zeroes (i. e. nothing at all). This command (not from Wikipedia)

dd if=/dev/zero of=sparse-file bs=1k count=5120

will write 5MB of zeroes and will not create a sparse file!

As a consequence, a file that is already non-sparse will not magically become sparse later.

Second, to make a file with lots of zeroes sparse, you have to cp it

cp --sparse=always original sparsefile

or you can use tar's or rsync's --sparse option as well.

Answer 2

Perhaps the easiest way to sparsify a file in place would be to use fallocate utility as follows:

fallocate -v --dig-holes {file_name}

fallocate(1) is provided by util-linux package on Debian.

Answer 3

Editing my answer for completeness:

1. Balloon empty FS space with zeroes (WARNING: this changes your disk image):

losetup --partscan --find --show disk.img

Assume it gives /dev/loop1 as the disk and there is only one partition, otherwise we need to repeat this for every partition with mountable FS in it (ignore swap partition etc.).

mkdir -p /mnt/tmp mount /dev/loop1p1 /mnt/tmp dd if=/dev/zero of=/mnt/tmp/tempfile

Let it finish to failure with ENOSPC.

/bin/rm -f /mnt/tmp/tempfile umount /mnt/tmp losetup -d /dev/loop1

2. Copy into a sparse image:

'dd' has an option to convert a file with zeroes to a sparse file:

dd if=disk.img of=disk-sparse.img conv=sparse

Answer 4

Do you mean that your ddrescue created image is, say, 50 GB and in reality something much less would suffice?

If that's the case, couldn't you just first create a new image with dd:

dd if=/dev/zero of=some_image.img bs=1M count=20000

and then create a filesystem in it:

mkfsofyourchoice some_image.img

then just mount the image, and copy everything from the old image to new one? Would that work for you?

Answer 5

PartImage can create disk images that only store the used blocks of a filesystem, thus drastically reducing the required space by ignoring unused block. I don't think you can directly mount the resulting images, but going:

image -> partimage -> image -> cp --sparse=alway

Should produce what you want (might even be possible to stick the last step, haven't tried).

Answer 6

There's now a tool called virt-sparsify which will do this. It fills up the empty space with zeros and then copies the image to a sparse file. It requires installing a lot of dependencies, though.

Answer 7

I suspect you'll require a custom program written to that spec if that's REALLY what you want to do. But is it...?

If you've actually got lots of all-zero areas then any good compression tool will get it down significantly. And trying to write sparse files won't work in all cases. If I recall correctly, even sparse files take up a minimum of 1 block of output storage where the input block contains ANY bits that are non-zero. For instance - say you had a file that had an average of even 1 non-zero bit per 512 byte block - it can't be written "sparsely". By the way, you're not going to lose data if you compress the file with zip, bzip, bzip2 or p7zip. They aren't like mpeg or jpeg compression that is lossy.

On the other hand, if you need to do random seek reads into the file then compression might be more trouble than it's worth and you're back to the sparse write. A competent C or C++ programmer should be able to write something like that in an hour or less.