Debian guest on Windows host using discard/TRIM.
This isn't a direct answer per se, as I'm addressing the problem, not the question. Instead of periodically compacting the image, this solution uses discard to automatically remove unused blocks in the host's VM disk image.
This solution requires a guest filesystem that supports continuous TRIM.
The Arch Linux wiki has a list of filesystems supporting TRIM operations.
FDE and cryptoroot are specifically not covered, as there are security concerns and none of the other solutions to this question would allow compacting either. The Arch Linux wiki has information about TRIM and dm-crypt devices.
In theory, this will work for all Linux guests on VBox hosts using VDI storage.
Host configuration
With VBox exited and no VMs running, add discard support to your disks by setting both discard
and nonrotational
for each disk in the config file for the VM. At this time discard
is not in the GUI, but nonrotational
is exposed as the "Solid-state Drive" checkbox. (ref: vbox forums, discard support)
<AttachedDevice discard="true" nonrotational="true" type="HardDisk" [..other options..] >
Boot the VM up, and verify that TRIM support is enabled:
sudo hdparm -I /dev/sda | grep TRIM
Guest Configuration
If LVM is in use, change the discard setting in /etc/lvm/lvm.conf
. (ref: debian wiki, lvm.conf example)
devices {
...
issue_discards = 1
}
In fstab, add the discard
option to the filesystems you wish to auto-discard (ref: debian wiki, fstab example)
UUID=8db6787f-1e82-42d8-b39f-8b7491a0523c / ext4 discard,errors=remount-ro 0 1
UUID=70bfca92-8454-4777-9d87-a7face32b7e7 /build ext4 discard,errors=remount-ro,noatime 0 1
Remount the filesystems to have them pick up their new options.
sudo mount -o remount /
sudo mount -o remount /build
Manually trim free blocks now with fstrim
. fstrim
uses the mounted filesystem, not the block device backing it. Instead of setting continuous discard in fstab
, this could be done on a weekly cron. (The weekly cron is recommended for physical SSDs which may have questionable support for TRIM, but this is not relevant here since underlying SSDs are handled by the host OS. see: ssd trim warning).
fstrim /
fstrim /build
At this point, the size of the filesystems inside the VM and the size of the VM images should be pretty close in value.
Tested with:
- Guest1: Debian 8.7, kernel: linux 4.8 grsec from backports, filesystem: ext4
- Guest2: Debian 9 RC2, kernel: linux 4.9, filesystem: ext4
- Host1: VBox 5.1.14, Win7, image fmt: VDI
- Host2: VBox 5.1.14, Win8.1, image fmt: VDI
this dd trick doesn't work for ext3 guest filesystem, but works well for ext4 – Reinaldo Gil – 2014-07-07T01:14:06.053
2Is it normal to have this error?
dd: error writing ‘/bigemptyfile’: No space left on device – Vincent – 2014-11-12T13:16:24.810
4@Vincent That's the point of this procedure. You're writing zeros to a new file until it fills entire free space, then you delete that file. – gronostaj – 2015-05-12T21:29:35.897
On Linux, you can defrag this way: using gparted, shrinking then moving the partition (see the link for more information)
– Anthony O. – 2015-05-29T14:39:44.917OSX Host - Win8.1 VM - When I try to optimize my C drive, it tells me that optimization is unavailable. - how can I defrag my virtual C drive? – Robert Achmann – 2015-09-24T02:34:26.873
1@RobertAchmann post a picture of the defrag UI of Win8.1 – magicandre1981 – 2015-09-24T04:02:44.773
I will in the morning, but both the analyze and optimize buttons are greyed out - and beside the drive in the list it says optimization is not available. but it looks as though you can schedule optimization to occur... – Robert Achmann – 2015-09-24T04:07:55.967
1@RobertAchmann ignore it and run sdelete now. – magicandre1981 – 2015-09-24T04:09:04.153
1If you experience
/ is busy
onmount
command, then reboot your guest in recovery mode. – Vasilly.Prokopyev – 2015-11-30T14:22:01.920Note that the zerofree man page says the "filesystem has to be unmounted or mounted read-only", so changing the run level with telinit then remounting the device read-only is wasted effort. Just unmount, zerofree, shut down, compact. – Urhixidur – 2016-03-23T19:33:04.653
1
The
– Katu – 2016-05-18T09:00:37.173dd
operation will take a while, you can see its progress installingpv
and usingsudo dd if=/dev/zero | pv | sudo dd of=/bigemptyfile bs=4096k
. Source: http://askubuntu.com/questions/215505/how-do-you-monitor-the-progress-of-dd1Awesome! Down to 15Gb from 27Gb – JohnTortugo – 2016-06-08T23:18:10.060
1
sdelete
requires a target as argument. Examplesdelete.exe c: -z
– Francesco Frassinelli – 2016-07-08T07:30:37.0601ok there was an update, before it used the current drive. I'll add it. – magicandre1981 – 2016-07-08T15:30:15.387
I suggested an edit to this post but it seems it has been somehow rejected. Anyway, the command is
VBoxManage
also on Linux, notvboxmanage
. – Andrea Lazzarotto – 2016-08-15T21:08:44.1571@Andrea Lazzarotto actually both VBoxManage and vboxmanage exist on my Linux box with VirtualBox 5.0.24 installed. – PolyTekPatrick – 2016-08-17T15:37:33.287
Ah! That's weird, I am pretty confident it was not working for me when I first tried to edit the answer. Maybe I was using the Ubuntu version. I am now using the version from their website so I cannot really check. – Andrea Lazzarotto – 2016-08-17T15:39:37.610
telinit 1
didn't work for me in Ubuntu, so I rebooted holding down the left shift to get the grub menu, I booted in recovery mode and in the recovery menu I selected the root shell. After that themount
and thezerofree
commands worked like a charm. VM reduced from 25 GB to the expected 11 GB. – f.ardelian – 2016-08-19T19:39:26.9303As a reference I had to delete my snapshots in order for this to make any difference. – Marcus – 2016-09-04T16:28:26.380
@VlastimilBurian thanks for the edit about the new options – magicandre1981 – 2016-11-27T20:32:06.093
@magicandre1981 No problem, as I use VirtualBox daily, I have just contributed to your good answer. – LinuxSecurityFreak – 2016-11-28T01:38:29.530
For a WindowsServer/7 guest on linux host, I used ccleaner to nullify empty space and then used your answer to compact vdi file. Worked great! – Alfabravo – 2017-01-17T17:16:36.687
1Great answer! This works well. I just wanted to point out that you don't need
sudo
when writing zeros to disk on a Linux guest. Just write to /tmp/bigemptyfile instead of /bigemptyfile:dd if=/dev/zero | pv | dd of=/tmp/bigemptyfile bs=4096k; rm /tmp/bigemptyfile
– sbleon – 2017-09-27T17:00:23.287@sbleon ok, edit the answer and add this change. I don't use linux that much – magicandre1981 – 2017-09-28T14:56:51.980
@sbleon thanks, I've approved your edit. thanks for pointing this out – magicandre1981 – 2017-09-28T15:16:20.730
@AndreaLazzarotto as PolyTekPatrick already mentioned there are both versions (
vboxmanage
andVBoxManage
) of the tool. Even more, they are just symlinks to a real executable: commandls -o $( which vboxmanage VBoxManage )
produceslrwxrwxrwx 1 root 4 Sep 13 16:37 /usr/bin/VBoxManage -> VBox
andlrwxrwxrwx 1 root 4 Sep 13 16:37 /usr/bin/vboxmanage -> VBox
. – Victor Yarema – 2017-10-10T10:19:55.5131@Victor, when I left my previous comment (and we are talking about more than a year ago) there was no lowercase version on my Ubuntu system. Only the uppercase one was present. – Andrea Lazzarotto – 2017-10-10T13:41:53.667
@AndreaLazzarotto, copy that. I didn't want to say that you are wrong. I know that things change. I just wanted to add some up to date info (including command example) for anyone that may come here for answers. Your comment was useful at that moment. Thanks. – Victor Yarema – 2017-10-10T14:10:29.417
If you create a file full of zeroes till ALL of the space is taken, won't the file grow to its absolute maximum size? My disk size is 100Gb, whereas the actual size is 15Gb. I know it should be around 9, If I run the command, will I have to wait for
dd
to write 85Gb?!? – Merc – 2017-11-25T05:47:24.3471Excellent! From 187GB to 46GB! I am confirming that this procedure works on Ubuntu 17.10 as host OS and Windows 10 as guest OS. – Bosko Mijin – 2018-01-10T16:03:09.467
A similar
– atomh33ls – 2018-01-12T12:23:27.977dd
command suggested here; https://askubuntu.com/a/903178/516133Had to remove my shared folder first. Worked great after that. – mgershen – 2018-04-29T10:09:32.107
Can somebody explain why defrag is needed? – vehsakul – 2018-10-18T16:52:08.347
zerofree: Worked with Win10 host, Ubuntu 18.04 guest and Ubuntu 18.04 live CD/iso. Please note you need plenty of free disk space on the host, otherwise VM stops working! – Tobias Weibel – 2018-11-02T07:31:27.177
It's worth noting that Windows 10 guests will not defrag a disk tagged as an SSD. The rest of this worked for me. Thanks. – Mooseman – 2019-03-27T11:25:10.683
1Be honest, I do not know why this is accepted answer. If I have 500GB VDI, only used about 100GB in win guest. The VDI is about 150GB. The sdelete will actually enlarge the VDI to 500GB. And after compact it only shrink down about 480GB. So it actually make the VDI larger. Just clone the 150GB VDI and run compact on it will shrink down to 120GB. I really want to middle finger the guy who did not test this on recent NTFS at all. Wasting lots of time and energy. This works on linux guest only! – Wang – 2019-04-08T01:06:32.017
VBoxManage: error: Cannot register the hard disk '/path/to/thedisk.vdi' {eee104d7-cd65-402f-b816-a3be4ac30eb3} because a hard disk '/path/to/thedisk.vdi' with UUID {eee104d7-cd65-402f-b816-a3be4ac30eb3} already exists
VBoxManage: error: Details: code NS_ERROR_INVALID_ARG (0x80070057), component VirtualBoxWrap, interface IVirtualBox, callee nsISupports
VBoxManage: error: Context: "OpenMedium(Bstr(pszFilenameOrUuid).raw(), enmDevType, enmAccessMode, fForceNewUuidOnOpen, pMedium.asOutParam())" at line 179 of file VBoxManageDisk.cpp
– Ed Randall – 2019-06-10T11:59:57.557@EdRandall submit the issue in their bug tracker. I stopped using Virtualbox for some times – magicandre1981 – 2019-06-10T13:43:39.307
@Vasilly.ProkopyevI hadd to
swapoff -a
in addition to booting in recovery mode and disabling systemd services. – Dan M. – 2019-08-29T16:40:23.420That bigemptyfile hung my VM, eating all the space leaving nothing for OS. Horrible advice. – Oleg Mihailik – 2019-10-28T10:49:33.793
As what @Marcus inferred, a good addition to this answer would be Step 0: For best results, delete all snapshots of the machine before performing the following steps. – Tfb9 – 2019-12-20T03:26:29.383
If running the command in the accepted answer produces an unhelpful error message like this
VBoxManage.exe: error: Cannot register the hard disk 'thedisk.vdi' {aaaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeeee} because a hard disk 'thedisk.vdi' with UUID {aaaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeeee} already exists Simply run the command by the UUID instead of the filename:
VBoxManage.exe modifyhd {aaaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeeee} --compact – DustWolf – 2020-02-27T10:09:58.887
@user248749 FYI,
– Gilles 'SO- stop being evil' – 2013-08-27T00:15:57.853dd
with no argument copies one block at a time, not one byte. Using a larger block size can speed things up, but the effect is pretty minor.19For the next person, my command ended up looking like this: "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyhd "C:\Users\daniel\VirtualBox VMs....\thedisk.vdi" --compact – Daniel – 2013-12-13T17:57:54.607
38
According to the zerofree Linux utility manpage (http://manpages.ubuntu.com/manpages/natty/man8/zerofree.8.html), zerofree should be better than dd for this job. dd would not been recommended because "it is slow", "it makes the disk image (temporarily) grow to its maximal extent", "it (temporarily) uses all free space on the disk, so other concurrent write actions may fail".
Zerofree is available on Ubuntu Linux via apt, or you can compile it yourself.
@Dakatine I never got it working. Add a working command to the post. – magicandre1981 – 2014-02-20T19:35:57.817
27It's fun that the the manpage of zerofree states that with dd other concurrent writes will fail, but zerofree needs the filesystem to be mounted read-only! *duh* – Madarco – 2014-02-25T12:04:31.540
1What is
/bigemptyfile
? A special path recognized bydd
? – Frozen Flame – 2014-04-30T06:24:06.047https://en.wikipedia.org/wiki//dev/zero this is simple a large file with zeros – magicandre1981 – 2014-04-30T18:13:51.887
3@FrozenFlame /bigemptyfile is just an arbitrary name that magicandre1981 used for the temporary file, you can name it anything you want.
sudo dd if=/dev/zero of=/zero.fill bs=4096k
is functionally exactly the same, it will just write to a file namedzero.fill
instead (you'd need to change therm
command to remove this file instead of/bigemptyfile
though) – Jason Larke – 2014-05-05T05:01:23.1037Tip: Put the two commands on one line like so:
dd ...; rm /bigfile
, this will minimize the time with a full disk in case you're not waiting for thedd
to complete. – jlh – 2014-05-14T17:45:57.19721@Dakatine Using VirtualBox 4.3.10, the disk image file did not grow to its maximal extent. VirtualBox is clever enough to not bother writing all zero blocks to the physical disk. – jlh – 2014-05-14T17:50:06.740