Apple has "sparse bundles" but not sparse files?

Question

Backstory:

I use a Mac (10.6.4), and to this day when I download a torrent using Vuze, no matter what configuration I try Vuze allocates space for the entire file before it starts downloading said file. For a long time, a quick google led me to believe it was a Vuze bug, but after years have passed and nothing has changed a more detailed search tells me that it's because HFS+ doesn't support sparse files.

Question:

The sparse file wiki says that Apple's HFS+ doesn't support sparse files, and Apples' Time Machine and FileVault uses "sparse images" and "sparse bundles".

I understand what a sparse bundle is (image split into 8Meg chunks) but aside from that, is there a difference between these "sparse images" and just regular sparse files? How can HFS+ not support sparse files and yet still support sparse images and sparse bundles?

Answer 1

It's because the sparseness is handled at a different level of abstraction for a sparse image vs. a sparse file. If you're storing a file "directly" on disk, the filesystem (e.g. HFS+) handles mapping the sequence of bytes that make up the file into a sequence of bytes on disk. A sparse file is one that has gaps in its sequence of bytes, and a filesystem that supports this will skip those missing bytes when it maps the file to disk.

When you're using a disk image, the mapping is quite a bit more complicated: you have the bytes in the file being mapped (by the inner filesystem) to bytes in the (virtual) disk. These are then mapped (by the image mounter) to bytes in the disk image (file), which are then mapped (by the outer filesystem) to bytes on the actual disk. The image mounter handles sparse mappings (and AIUI only between/beyond files, not within them), but that doesn't imply that the other layers (the filesystems) can. As far as the outer filesystem is concerned, the image file is just a (non-sparse) sequence of bytes.

BTW, the image mounter can also add other features that the filesystem alone doesn't deal with. For instance, Mac disk images have supported compression roughly forever, but native support for compression wasn't added to HFS+ until OS X v10.6.

Answer 2

Gordon Davisson explanation was enlightening: this is how I am using sparsebundle images as a torrents download directory.

First of all create the image (I set it to 50GB, but any size would work)

hdiutil create -size 50g -type SPARSEBUNDLE -nospotlight -volname "Torrents repository" -fs "Case-sensitive Journaled HFS+" Torrents.sparsebundle

Then double click Torrents.sparsebundle image just created to mount it.

Setup your favorite torrent client to use the volume mounted called "Torrents repository" as temporary repository, I suggest to set it up so that completed files would be moved out of the image (to Downloads folder for example).

It works very well, especially when you select only a few files in a big torrent.

I also added a startup script to my system:

hdiutil compact Torrents.sparsebundle 
hdiutil attach Torrents.sparsebundle

First command will reclaim the unused diskspace, while second will automatically mount the image. Unfortunately the compact operation cannot be done while the image is mounted, so I thought the best chance was on system startup before mounting it.

Answer 3

APFS which is new default FS now does support sparse files.