54

How to zero fill a virtual disk's free space on windows for better compression?

I would like a simple open source tool (or at least free) for that. It should probably write an as big as possible file full of 0and erase it afterwards. Only one pass (this is not for security reasons but for compression, we are backing up virtual machines).

Should run from inside windows and not from a disk.

On Linux I do it like this (as a user):

cd
mkdir wipe
sudo sfill -f -l -l -z ./wipe/

Edit 1: I decided to use sdelete from the accepted answer. I had a look at the sdelete's help:

C:\WINDOWS\system32>sdelete /?

SDelete - Secure Delete v1.51
Copyright (C) 1999-2005 Mark Russinovich
Sysinternals - www.sysinternals.com

usage: sdelete [-p passes] [-s] [-q] <file or directory>
       sdelete [-p passes] [-z|-c] [drive letter]
   -c         Zero free space (good for virtual disk optimization)
   -p passes  Specifies number of overwrite passes (default is 1)
   -q         Don't print errors (Quiet)
   -s         Recurse subdirectories
   -z         Clean free space

This is an old version. I used the -c switch from the 2nd invocation and this was quite fast (syntax only valid for older versions before V1.6):

c:\>sdelete -c c: (OUTDATED!)

I have the impression this does what I want. The sdelete tool is easy to use and easy to get.

Edit 2: As scottbb pointed out in his answer below, there was a September 2011 change to the tool (version 1.6) The -c and -z options have changed meanings. The correct usage from 1.6 onwards is

c:\>sdelete -z c:

Edit 3: There is a 2.0 version of sdelete and sdelete64. They appear to be buggy when zeroing. It will appear to be stuck at 100% for extremely long times. Some people have reported 10 - 40 times longer. The older version 1.61 does not have this issue. See https://social.technet.microsoft.com/Forums/en-US/2ffb2539-34ba-4378-aa8a-941d243f117e/sdelete-hangs-at-100?forum=miscutils

Edit 4: Now there's the issue of dynamically allocated virtual disc space. If you have a 100GB disk that is not full and uses only 30GB on the host, zero filling should not increase dramatically the size of the disc because that contradicts the purpose of dynamic allocation. There is an answer for Oracle VM VirtualBox https://superuser.com/q/907196/44402 - but on other stacks like VMWare, Xen, XenServer, etc., this needs to be answered separately.

mit
  • 1,844
  • 6
  • 29
  • 42
  • Also make sure you use COW (Copy on Write) style images if you have lots of machines based off the same basic install. – hookenz Jan 05 '12 at 03:39
  • 3
    `sdelete` is the right tool for the job, but a September 2011 change to the tool (version 1.6) makes the above answers incorrect. The `-c` and `-z` options have changed meanings. The correct usage now is `sdelete -z c:\ ` – scottbb Jan 05 '12 at 03:23

9 Answers9

53

On windows use the sysinternals tool sdelete to zero out all the empty space. The command you want would look like this sdelete -z c:.

Usage: sdelete [-p passes] [-s] [-q]  ...  
sdelete [-p passes] [-z|-c] [drive letter] ...  
-a         Remove Read-Only attribute.  
-c         Clean free space.  
-p passes  Specifies number of overwrite passes (default is 1).  
-q         Don't print errors (Quiet).  
-s or -r   Recurse subdirectories.  
-z         Zero free space (good for virtual disk optimization).

For Linux I suggest you use zerofree.

Zoredache
  • 128,755
  • 40
  • 271
  • 413
  • Came here to compact my VHDs. Sharing some related links from TechNet.. https://technet.microsoft.com/en-us/library/cc708394%28v=ws.10%29.aspx https://technet.microsoft.com/en-us/library/cc708317(v=ws.10).aspx – Faiz Aug 23 '15 at 13:47
  • I wonder, if zero-filling would have any positive impact on a Linux standard encrypted drive [LUKS dm-crypt](https://en.wikipedia.org/wiki/Dm-crypt). __before__ I subject the entire machine to booting from a clonezilla bootstick and physical backup of the entire drive. — Will zeroed section within the encryption lead to zeroed section on outside disk? Or will it still write (unallocated) noise for security reasons? Or should I zero the raw-mounted (not looking at encrypted contents) crypt drive? (And is that a good idea ie is the allocation table trustworthy in the context of a crypted drive?) – Frank Nocke Nov 21 '16 at 08:47
  • 1
    I'd be shocked if it did anything other than encrypt the nulls and write the encrypted "noise" to the disk. – Dan Pritts Jan 11 '17 at 16:33
21

Windows already has a built-in command line tool that does this: CIPHER. See its /W option.

So the complete command would be: cipher /w:c: (replace c: with actual drive letter)

(it writes zeroes to the unused space, then 0xff, so you have to watch it and stop it with ctrl-c when it begins to write 0xff - not ideal, but it is free and already available in Windows since XP at least)

David Balažic
  • 419
  • 6
  • 19
  • 5
    interesting approach. Break execution just before it starts undoing all of the zeros. +1 for a creative solution using tools that are already there! – PsychoData Nov 12 '14 at 16:10
  • 1
    I wonder whether it's possible to detect when it's finished writing zeros and then make it abort... – Frederick Nord Nov 11 '15 at 13:51
  • If you use SSD hard drive, check you have TRIM enabled. Then all unused block will be quickly wiped out with 0xFF pattern by hard drive firmware. This is very usable on virtual disk images made by virtual servers. That images will be filled as gaps, and will free file system blocks on hosting file system. – Znik May 24 '17 at 13:43
  • This worked very well and a lot faster than SDelete -z (which was stuck at 100% without giving any other feedback). – psychowood Dec 22 '17 at 14:03
8

fsutil isn't open source, but is does ship with Windows and therefore doesn't cost anything extra. I used it to zero the free space on an empty WD 250 GB 7200 RPM drive (F:). Here's what I ran from the command line:

fsutil volume diskfree f:

Which showed this report:

Total # of free bytes        : 249899469856
Total # of bytes             : 249997291520
Total # of avail free bytes  : 249899469856

I used Total # of avail free bytes in the following commands:

fsutil file createnew F:\clear 249899469856
fsutil file setvaliddata F:\clear 249899469856
fsutil file setzerodata offset=0 length=249899469856 F:\clear
del f:\clear

It took about 4 hours to write 250GB of zeros.

James L.
  • 231
  • 3
  • 7
6

SDelete will not zero out the space for compaction of a virtual disk. SDelete starts by zeroing free space, but then fills it with 0xff and then random bytes See How to prepare a Virtual Server 2005 virtual hard disk file to send to Microsoft Product Support Services For how to set up a VHD for compaction

Precompact.exe can be difficult to find for some reason. I found a seperate downloadable copy here. However if you have windows virtual PC you can also get it from %programfiles(x86)%\Windows Virtual PC\Integration Components\Precompact.iso

Jim B
  • 23,938
  • 4
  • 35
  • 58
  • 2
    sdelete with the -c option is a zero fill. Perhaps you are thinking of the Windows built-in cipher utility which does not have a zero-only option? – Jeremy M Jul 29 '10 at 05:41
  • "SDelete implements the Department of Defense clearing and sanitizing standard DOD 5220.22-M" (from the docs) which specifically does not simply 0 out a cluster. sdelete -c does zero out free space (after filling it), however the method that sdelete uses isn't the same as precompact and in many cases will result in a larger drive than precompact, which was designed to zero out virtual drives – Jim B Jul 29 '10 at 12:27
  • This is interesting to know of another tool. Thank you – mit Jul 29 '10 at 19:14
  • 2
    SDelete can leave the disk with 0s, but precompact works much faster and was meant for this sort of situation. – Chris S Jan 05 '12 at 03:30
  • @ChrisS Do you know what precompactor.exe does differently that makes it faster? It's pretty difficult to find any info on exactly what it does. – RomanSt Sep 14 '14 at 23:44
  • 2
    @romkyns It doesn't matter anymore as all modern versions of Windows and Hyper-V support `trim` (as of Win7 and Hyper-V 2012), there's no need to zero before compacting. The PreCompact utility only zeroed parts of the disk with deleted files, the part that hadn't ever been used was left untouched. Using something like `sdelete` would fill the dynamic disk to static size, which could then be compacted back down to only the size actually used. – Chris S Sep 15 '14 at 13:45
  • 1
    Precompact defrags the drive, then fills the allocated space with 0, then copies the filled sectors to a new VHD. Sdelete -c actually writes data then zeros. -z writes the zeros to all free space. (even if it was not allocated) – Jim B Sep 16 '14 at 17:25
5

I too had been looking for a way of using Cipher to only write 0x00 then exit when starting to write 0xFF to the free space. This will allow the maximum free space to be compressed. I have come up with the following basic PowerShell job

replace d:\ with the required drive

Function CipherFreeSpace
{
    $cipherjob = @()
    $Job = start-job -ScriptBlock {cipher /w:d:\ }
    while ($cipherjob -notcontains "Writing 0xFF")
    {
        Write-host "." -nonewline 
        Start-Sleep 2
        $cipherjob += $job | Receive-Job 
    }
    $Job | Stop-Job | Remove-Job -Force
}
CipherFreeSpace

Regards Simon

Simon Stevenson
  • 79
  • 1
  • 2
  • I had to run this with `powershell -ExecutionPolicy Bypass -File scriptfile.ps1` because powershell is nuts with `cannot be loaded because running scripts is disabled on this system`. – user Jan 09 '21 at 20:34
  • -1: Running this caused my machine to full its disk: `VMware Player has paused this virtual machine because the disk on which the virtual machine is stored is almost full. To continue, free an additional 1.0 GB of disk space.` – user Jan 09 '21 at 20:39
1

If you are using VMware, a much better way is to use the tools built into VMware to reclaim any unused space on the virtual drive.

I used this to reduce the size of the virtual disk from 28 gigabytes down to 13 gigabytes. Your mileage may vary.

When I tried using sdelete -z c:, it actually increased the size of the virtual drives to 30 gigabytes, which was its maximum size.

See the VMware website and Compact a Virtual Hard Disk, which is under Using VMware Workstation > Configuring and Managing Devices > Configuring and Maintaining Virtual Hard Disks.

Update

Sometimes, the GUI tools are not quite as reliable as the command line tools. I have run into issues where the GUI tools fail, but the command line tools work just fine. Here are the options for VMware Desktop 10:

C:\Program Files (x86)\VMware\VMware Workstation>vmware-vdiskmanager -h
Diskname or some other argument is missing.
VMware Virtual Disk Manager - build 1895310.
Usage: vmware-vdiskmanager.exe OPTIONS <disk-name> | <mount-point>
Offline disk manipulation utility
  Operations, only one may be specified at a time:
     -c                   : create disk.  Additional creation options must
                            be specified.  Only local virtual disks can be
                            created.
     -d                   : defragment the specified virtual disk. Only
                            local virtual disks may be defragmented.
     -k                   : shrink the specified virtual disk. Only local
                            virtual disks may be shrunk.
     -n <source-disk>     : rename the specified virtual disk; need to
                            specify destination disk-name. Only local virtual
                            disks may be renamed.
     -p                   : prepare the mounted virtual disk specified by
                            the mount point for shrinking.
     -r <source-disk>     : convert the specified disk; need to specify
                            destination disk-type.  For local destination disks
                            the disk type must be specified.
     -x <new-capacity>    : expand the disk to the specified capacity. Only
                            local virtual disks may be expanded.
     -R                   : check a sparse virtual disk for consistency and attempt
                            to repair any errors.
     -e                   : check for disk chain consistency.
     -D                   : make disk deletable.  This should only be used on disks
                            that have been copied from another product.

Here is an example of shrinking a disk. This command line worked, in spite of the same command from the GUI failing:

C:\Program Files (x86)\VMware\VMware Workstation>vmware-vdiskmanager.exe -k
 "H:\VMware\WindowsXP.vmdk"
  VixDiskLib: Invalid configuration file parameter.  Failed to read configuration file.
  Shrink: 100% done.

The error message starting with VizDiskLib is new to VMware Desktop 10 and can safely be ignored.

Community
  • 1
Contango
  • 1,130
  • 5
  • 15
  • 31
  • 1
    sdelete itself is not meant to shrink the vmdk size. It's used to turn all available free space into zeroes in preperation for the shrink operation. If you dont do this, deleted files and other orphaned data will still be present on the disk. VMWare tools won't always be able to reclaim the disk space still occupied by those files. Turning these "deleted files" into zeroes makes the disk space fully reclaimable when you run the shrinking tool. The reason you saw a increase in file size on the virtual disk is because sdelete fills up all available space it has with zeroes. – Leon Lucardie Jun 06 '17 at 11:05
  • Thanks for clarifying this. Methinks that the tools could benefit from better naming and/or a one-line description of what the tool does when you run it. – Contango Jun 09 '17 at 08:52
0

Google-up Eraser 5.8.8 32-bit Works nicely even on Windows XP SP3!

Filip OvertoneSinger Rydlo
  • 101
  • 2
-1

Mit,

I found that Simon's powershell script did exactly what you need, without the versioning problems of sdelete. (switches doing different things in different versions)

It had one problem, that cipher kept running, but I fixed that.

I further modified the script to cleanup the temp folder:

You will need to replace F:\ with your drive letter in both places.

Function CipherFreeSpace
{
    $cipherjob = @()
    $Job = start-job -ScriptBlock {cipher /w:F:\ }
    while ($cipherjob -notcontains "Writing 0xFF")
    {
        Write-host "." -nonewline 
        Start-Sleep 2
        $cipherjob += $job | Receive-Job 
    }
    Stop-Process -processname cipher
    Start-Sleep -s 3
    Remove-Item -recurse F:\EFSTMPWP
    $Job | Stop-Job | Remove-Job -Force
}
CipherFreeSpace
Preston Shirmeyer
  • 1
  • 2
-4

Why third party tools, when Windows already includes all you'll ever need, for "free" and as fast a it can ever be:

1) format X: /fs:XXXX /p:#, where X is the drive letter; XXXX is the file system,Yeither and # is the number of passes;

2) diskpart clean command: http://technet.microsoft.com/library/cc766465.aspx, http://support.microsoft.com/kb/300415;

Btw, you don't need more than a single pass, either - a claim otherwise is either outdated info for obsolete disk red/write strategies, people pushing products, paranoia driven, or bs passed on as "expert advice".

Edit: nvm, missed the free space part...

user180961
  • 9
  • 1
    In many cases there are third party tools available that do good or better jobs, are easier to install or more reliable. But this is totally OT, the question was not about who writes it. Apart from this Microsoft aquired Sysinternals (who wrote SSDelete) because they were lightyears better than the builtin stuff. Just compare ProcessExplorer to the MS product "TaskManager" -1 for starting a flamewar – Alojz Janez Jul 11 '13 at 10:49
  • You missed the "open source" requirement in the question, did you? – Alojz Janez Jul 11 '13 at 10:50