How to defragment a large TrueCrypt volume that has barely any free space

4

1

I have a TrueCrypt 7.1a volume that is about 50 GB in size, formatted with NTFS and has close to 100 MB of free space. I have close to 300,000 files in it. When I was setting this up, I did this intentionally to save space.

Since the files I have in the volume is very small in size (The largest file is probably 20 MB in size) I enabled the ``Compress this volume to save disk space" option. So, all the files in the volume are compressed with standard NTFS compression.

I then copied all of the files from my hard drive to the volume with the xcopy command. Everything went fine up to this point.

Since there are close to 300,000 files, opening the volume in Windows Explorer crashes it within a minute. So, when I need a file, I mount the volume and use the xcopy command to copy the file to my computer in the place where it needs to be.

The only problem is it's painfully slow. It's pretty much impossible to copy more than one file like this. I then analyzed the volume with the Windows Defragmenter tool and it said it was 38% fragmented. After seeing that I defragmented it. It didn't get far at all. Something like 10% through pass 1 about 10 hours in.

I tried to use UltraDefrag as well, after it stayed at 0.0% done after an hour, I thought it was useless to continue.

So, my question is, how can I defragment this volume? Is it even possible to with so little free space on the volume?

0xAether

Posted 2013-05-13T19:58:59.413

Reputation: 478

2You can't. You won't be able to move the larger files. Of course defragging won't help with the crashing. – Ramhound – 2013-05-13T20:01:30.050

@Ramhound I'm not concerned about Explorer crashing, I just use xcopy anyways. If I had to guess the largest file on the whole volume is about 20 MB. – 0xAether – 2013-05-13T20:04:58.200

If that was the case you can easily defrag the drive. The files must be larger then 100MB if you make no progress. You might be able to get around this be dragging the volume while its not attached although that sounds sort of risky. – Ramhound – 2013-05-13T20:06:41.463

2Create a new encrypted volume, and give yourself some growing space. Move files from old volume to new volume. When the move happens files will be transferred one at a time and not be fragmented. Keep in mind that almost every filesystem starts to behave badly after you have gotten beyond ~95% usage. Give filesystem some extra space. – Zoredache – 2013-05-13T20:10:18.153

@Ramhound I just checked, the largest file is 31 MB. And what do you mean by defragging while the volume isn't attached? – 0xAether – 2013-05-13T20:20:36.643

@Zoredache Looks like that is my only option. Copying the files off of it will take a very long time, though. – 0xAether – 2013-05-13T20:20:58.147

If the largest file is 31MB and it it too slow to do more than a single file with xcopy (which BTW, I don't get from your description), your issue is not fragmentation. with your new volume, don't compress the files and see if that makes a difference. – Frank Thomas – 2013-05-13T20:30:19.673

It must be the compression. Although it's weird how Windows said it was 38% fragmented and UltraDefrag said 98% fragmented. – 0xAether – 2013-05-13T20:38:40.260

"Since there are close to 300,000 files, opening the volume in Windows Explorer crashes it within a minute" Do you have all those files in one directory?? The theoretical limit is higher (http://superuser.com/questions/446282/max-files-per-directory-on-ntfs-vol-vs-fat32), but its a recipe for fragmented indices, defrag issues, explorer choking, and other protential problems (http://stackoverflow.com/questions/197162/ntfs-performance-and-large-volumes-of-files-and-directories). Reorganize your files into multiple folders.

– Jan Doggen – 2013-05-14T06:31:33.043

@0xAether - I figured this volume was a mounted Truecrypt container which would mean it was freespace on a larger partition when it wasn't mounted. – Ramhound – 2013-05-14T11:01:43.317

Answers

3

@Zoredache touched on this in the comments on your question, but let me expand on it a little bit.

At a minimum, any drive should have about 10% minimum free space(20% if you have an OS on the drive), this is because the filesystem needs that extra space to store temporary files and journal information to perform its operations efficiently.

Any operation will be exponentially slowed the less free space you have on the drive past about 10% and defragging is the most free space intensive operation you can perform on a drive.

In this situation you need to transfer the files to another drive that is large enough to hold them, then you can expand the 50 gb partition to ~100 gb if you intend to continue to add files to it or ~65-70 gb if you do not. Afterward you can move all the files back.

jjno91

Posted 2013-05-13T19:58:59.413

Reputation: 313

This is what I'm in the process of doing at the moment. It's going much quick than what I thought it was going to. In a matter of about 30 minutes, xcopy copied about 9 GB of data. That's strange because normally xcopy copies a single file in about 30 seconds to a minute. – 0xAether – 2013-05-13T22:19:50.730

@0xAether - 30 seconds for a 31MB file. There is something wrong going on here. You sure the physical HDD is checking out. Check the S.M.A.R.T data on the device. – Ramhound – 2013-05-14T11:03:34.970

@Ramhound Yes, the drive is fine. It's about 3-4 months old. I actually found out what the problem was. Well, I should say what I think the problem was. I created a new container with about 5% free space. I also did not configure NTFS compression. Read times on the container is normal now. The problem as been resolved by those two things. – 0xAether – 2013-05-15T18:39:34.190

1@0xAether - It sounds like a combination of filesystem compression and the amount of free space aviable was indeed the problem. You might have been able to defrag the hdd if you let it work for a couple days but the access time was the real problem. – Ramhound – 2013-05-16T11:07:40.337