Why did defragmenting C drive increase my free disk space by 10 GB?

Question

I used Defraggler to defragment free space on my 100 GB C:\ drive which was 85 GB full. After defragmentation, the drive showed only 75 GB full. How did 10 GB  of free space magically appear? Did I lose any data?

I did a disk cleanup before defragmenting and my trash was only about 11 MB, so it can't be because of cleaning the temporary files. Note that I defragmented "free space" which means that it should have rearranged the empty blocks to be contiguous.

Answer 1

What probably happened is that the defrag operation forced Windows to throw out some system restore snapshots. It'd be a pathological case of fragmentation to cause metadata overhead to be a full 10% of your drive space on top of what Windows normally uses. even then, I'm not sure it's possible.

I don't see anything in Defraggler's version history or documentation that indicates that it's able to correctly defragment files to prevent the purging of shadow copies. In fact, this thread from Defraggler's support forum indicates that they know that it's happening (there's a post from a board admin labeled "Official Piriform Bug Fixer" in thread) but don't indicate whether or not they're going to fix it.

Shadow copies may be lost when you defragment a volume: The reason this happens is that by default, VSS operates with 16 KB clusters by default, while most NTFS volumes are formatted with 4 KB clusters. So if a defrag operation is moving data that isn't a multiple of a 16 KB cluster (or the "distance" it's moved isn't a multiple of 16 KB), then VSS will track it as a change and might purge all your snapshots.

MSDN: Defragmenting Files:

When possible, move data in blocks aligned relative to each other in 16-kilobyte (KB) increments. This reduces copy-on-write overhead when shadow copies are enabled, because shadow copy space is increased and performance is reduced when the following conditions occur:

• The move request block size is less than or equal to 16 KB.
• The move delta is not in increments of 16 KB.

Vista's built in defrag doesn't do this:

One change that’s not obvious to users is our shadow copy optimization during defragmentation. Defrag has special heuristics to move file blocks in a way that will minimize the copy-on-write activity and shadow copy storage area consumption. Without this optimization, the defragmentation process would accelerate the deletion of older shadow copies.

Answer 2

Each fragment has to be tracked somewhere. That takes storage space (within the File system's plumbing, not stuff you're meant to directly access).

An example: Suppose you have a single file with 1000 fragments. Thus your file is stored through a collection of random blocks.. Rather than in a single continuous block. This means the fragmented file requires 1000x more storage space within the file system's plumbing, if only for storing addresses for each fragment. The file system plumbing keeps little dictionaries/databases/maps/tables/list to the location of each fragment of the file. So, for the file system plumbing, storing a list of a single fragment pointer doesn't need much space, compared to a list of 1000 fragment pointers..

But hey, maybe I'm wrong...

Edit: Supporting information from here:

When a non-resident data stream is too much fragmented, so that its effective allocation map cannot fit entirely within the MFT record, the allocation map may be also stored as an non-resident stream, with just a small resident stream containing the indirect allocation map to the effective non-resident allocation map of the non-resident data stream.

Translation: If you have heavy fragmentation, the general case assumptions of the file system plumbing won't apply. As such, the FS must take steps to accommodate the fragmentation and end up costing extra storage space, just to manage the fragments. Exactly my guess from the first place.

---

Edit: Given the above, it still seems like 10GB being lost just to file fragmentation is crazy. I'm betting that while defragging, you had some common file system corruption that was corrected automatically. I'm thinking not only did you have massive fragmentation, but also partially deleted files taking up storage space. It would have been nice to see a scandisk log from that defrag (or a run of scandisk prior to defrag)