37

Is there a possibility to "defragmentize" a sparse bundle image and reclaim (most) of the free space?

Here is the background: I am using sparse bundles and every now and then I want to reclaim space from them so I run:

hdiutil compact image.sparsebundle

However, as explained in the man page, it only reclaims completely unused band files, so in my case it says:

Reclaimed 0 bytes out of 90.4 GB possible.

Of course there is the possibility to copy the contents of this image to a new sparse bundle that is then used in lieu, but that is both cumbersome and requires enough free space for this operation.

Meanwhile, I found out that the output of the compact command is somewhat misleading (I am currently running OS X 10.5.7) as it sometimes lists a size as possible that is larger than the size currently taken up by the image bundle on the hard drive. I did not look closer but the output seems to be either the maximum size or "maximum size" - "used size".

user4854
  • 373
  • 1
  • 3
  • 5
  • 1
    I had an issue with this, but then realized it was because my sparsebundle fs was EXFAT. After reading the [manpage](https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/hdiutil.1.html), I realized that the compact verb only works on "disk images containing an HFS filesystem". – jsejcksn Dec 14 '16 at 02:27
  • @jsejcksn That is a good point to keep in mind. The man page now says that APFS and HFS+ are supported. – GDP2 Mar 26 '19 at 04:49
  • Relevant question on AskDifferent: https://apple.stackexchange.com/questions/54607/completely-recover-space-from-sparse-bundle-when-hdiutil-compact-is-not-enough – GDP2 Mar 26 '19 at 04:57

6 Answers6

23

Interesting!

From what I've heard, the sparse bundle divides the data into 8Mb bands. Changing the band size might just help, if you're lucky. I mean, you'll never get 100% reclaimed space, but maybe better than what you get now. (Depending on the data on the image etc.)

I did a dirty simple test with two 500Mb sparse bundles, one with 8Mb (default) band size, and one with 1Mb (smallest allowed size from what I can tell). I copied over 400mb of mp3 files and then removed every other file and then run hdiutil compact on their asses.

Size after compact
8Mb bands: 271Mb
1Mb bands: 215Mb

The command to convert your sparse bundle is

hdiutil convert src.sparsebundle -format UDSB -tgtimagekey sparse-band-size=2048 -o dst.sparsebundle

Band size is in the unit 512byte. So the above example sets the band size to 512 * 2048 = 1Mb. Just be careful if you're dealing with TimeMachine images or user home folder images etc. You're deviating from the Apple path :) Keep a fail safe backup!

As for defragmentation: I have a funny feeling it's just as fast (or faster!) to just use hdiutil to convert the sparsefile to a new sparse file with the same format. I think it tries to be smart about it. But I don't know.

(Note that defragmenting a sparse bundle just defragments the disk data, not the sparse bundle bands, unless it's a sparse bundle aware defragmentor. hdiutil convert does a band 'defragmentation' I believe.)

Skyhawk
  • 14,149
  • 3
  • 52
  • 95
0scar
  • 451
  • 3
  • 5
  • Your command line arguments are in the wrong order: `-o` should be directly before `dst.sparsebundle`, not `src.sparsebundle` (`-o` denotes 'output'). – Nick Forge Nov 01 '10 at 12:05
  • Also, note that this method doesn't preserve encryption. There's probably a flag to enable it, but at this point I found it easier just to make myself a new sparsebundle in Disk Utility and copy my files into it. – Justin Searls Dec 24 '10 at 19:28
  • 3
    For encryption, just add the `-encryption` switch to the command. – fideli Sep 20 '11 at 03:37
  • 1
    Thanks a lot. Decreasing sparse-band-size is very useful for Dropbox hosted sparse-bundles – bot47 Sep 17 '12 at 08:04
  • Ok this doesn't make sense... when I convert to defragment the image, the resulting sparse bundle is 16% *bigger* than the original. – Michael Jun 06 '16 at 04:15
9

I think the 90.4 GB possible is plain wrong. When I ran:

hdiutil compact *.sparsebundle

...it freed up 16 GB of space, which is roughly what I expected. Funnily enough it said roughly "800 GB possible" which is way more than the capacity of my HD. So I assume the second figure is just some (flawed) theoretical number.

Community
  • 1
till
  • 91
  • 1
  • 1
3

@Oscar pointed us in the right direction. But you don't need to convert to 1 MB and compact, hoping to regain space.

Instead, just copy your sparsebundle to another new sparsebundle. This, in turn, defrags the image for you. And at the same time you can change filesystem, size and even the bands.

Here's an example I did recently, converting my 20GB "code" image that i sync across devices.

$ hdiutil create -size 20g -type SPARSEBUNDLE \
-imagekey sparse-band-size=2048 -fs HFSX \
-volname code -attach ~/sync/images/code.sparsebundle

See all options with hdiutil create -help.

This command outputs:

/dev/disk3              GUID_partition_scheme
/dev/disk3s1            EFI
/dev/disk3s2            Apple_HFS                       /Volumes/code
created: /Users/eric/sync/images/code.sparsebundle

You can see the -attach option mounted it under /Volumes/code.

Note: I use HSFX for case-sensitive HFS+ because I sync my images to Linux as well and mount them. You'll most likely want to use HFS+ instead, since macOS is normally case-insensitive.

Now all I have to do is copy everything to my new sparsebundle:

$ cp -r /Volumes/IMG_CODE/ /Volumes/code/

And now to compare:

# old sparsebundle
$ du -h ~/sync/images/IMG_CODE.sparsebundle/
15.0G   /Users/eric/sync/images/IMG_CODE.sparsebundle/bands
15.0G   /Users/eric/sync/images/IMG_CODE.sparsebundle/

# new sparsebundle
$ du -h ~/sync/images/code.sparsebundle/
3.0G    /Users/eric/sync/images/code.sparsebundle/bands
3.0G    /Users/eric/sync/images/code.sparsebundle/

Yay. My 15GB sparsebundle is now 3GB, defragged, and bands changed!

Eric Duncan
  • 215
  • 3
  • 14
  • This method was acknowledged in the original question... I won't downvote the answer because it's well written and "informative", but.. c'mon.. – adfaklsdjf Dec 01 '18 at 20:32
0

I don't know of a free way to do it, but I believe Prosoft Drive Genius will do what you're needing here:

http://www.prosofteng.com/products/drive_genius.php

You mount the sparsebundle, defrag it using Drive Genius, and then the 

hdiutil compact

command should work.

palmer
  • 1,279
  • 9
  • 10
0

Reclaimed 0 bytes out of 90.4 GB possible.

I bet the drive that holds the sparse bundle has about 90 GB free space? As sparse bundles often can grow until the disk is full, this space is simply reported as being available free space... You will see the same figures in Disk Utility. Very misleading indeed.

The following might give you some insight:

hdiutil imageinfo image.sparsebundle
Arjan
  • 403
  • 4
  • 8
0

I had a problematic sparse bundle that claimed to have 950 GiB occupied out of 1024 GiB (my SSD is just 320 GiB in size, so that couldn't happen). hdi compact didn't help.

I have ran "First aid" from Disk Utility on both bundle AND partition, and that helped — free space meter started showing real 20 GiB as it should have, and then hdi compact ran without problem.

toriningen
  • 201
  • 2
  • 5