Mass deleting files in Windows

142

81

I have a directory, that contains ~ 3 million files in certain subdirectories on a Windows 2008 server. Manually deleting the files via SHIFT+DEL on the root dir takes ages. Is there any other way to do the deletion in a faster manner?

aaginor

Posted 2009-08-07T07:00:33.420

Reputation:

7

@Jaime With a steady hand, like a real programmer... ;-) https://xkcd.com/378/

– Legionair – 2014-06-19T21:39:01.370

What if it's a SSD Drive? – Benjamin Weiss – 2014-08-22T06:51:34.970

Answer for powershell: https://stackoverflow.com/questions/1752677/how-to-recursively-delete-an-entire-directory-with-powershell-2-0

– Warlike Chimpanzee – 2019-02-10T19:34:49.967

6How could I target the magnet to some subdirectories only? – Jaime Hablutzel – 2014-05-16T18:39:59.020

Answers

180

WARNING: if you have symlinks to directories then del will delete actual directories and not symlinks. Be very careful with this and do not run these commands unless you know there are no symlinks inside target directory.


I regularly need to delete lots of files and directories from a WinXP encrypted drive, typically around 22 GB of 500,000 files in 45,000 folders.

Deleting with Windows Explorer is rubbish because it wastes lots of time enumerating the files. I usually move the stuff I need to delete to C:\stufftodelete and have a deletestuff.bat batch file to rmdir /s/q C:\stufftodelete. This is scheduled to run at night, but sometimes I need to run it during the day so the quicker the better.

Here's the results of a quick time test of a small 5.85 MB sample of 960 files in 303 folders. I ran method 1 followed by method 2, then reset the test directories.

Method 1 removes the files and directory structure in one pass:

rmdir /s/q foldername

Method 2 has a first pass to delete files and outputs to nul to avoid the overhead of writing to screen for every singe file. A second pass then cleans up the remaining directory structure:

del /f/s/q foldername > nul
rmdir /s/q foldername
  • Method 1: 17.5s, 14.9s, 13.9s, 14.8s, 13.8s: average 14.98 seconds
  • Method 2: 14.3s, 12.1s, 11.7s, 14.2s, 11.8s: average 12.82 seconds

Here's results of another test using 404 MB of 19,521 files in 3,243 folders:

  • Method 1: 2 minutes 20 seconds
  • Method 2: 2 minutes 33 seconds

So there's not much in it, probably too close to judge on a single test.


Edit: I've retested with much more data, this is a typical case for me: 28.3 GB of 1,159,211 files in 146,918 folders:

  • Method 1: 2h 15m, 2h 34m: average: 2 hours 25 minutes
  • Method 2: 49m, 57m: average: 53 minutes

Wow, method 2 is nearly three times faster than method 1! I'll be updating my deletestuff.bat!

Hugo

Posted 2009-08-07T07:00:33.420

Reputation: 2 640

1Both rmdir and del fail when there are pathnames longer than 256 characters involved. – Nicolai Ehemann – 2016-11-15T14:24:42.213

1WARNING: if you have symlinks to directories then del will delete actual directories and not symlinks. Be very careful with this and do not run these commands unless you know there are no symlinks inside target directory. – user0103 – 2019-09-19T12:17:10.560

RD /S /Q works great if you specify an absolute path but it does not when working with relative paths. Is there a solution? – Michael S. – 2012-01-31T07:25:39.473

1@Hugo : Hugo, a question: In the above timed tests, for either or both methods, did you count the files immediately before you ran the method? I'm asking because the folder contents might already be in the OS file cache. Thanks! – William C – 2012-04-10T15:19:44.183

@WilliamC: I don't remember exactly, but for the repeated deletes with the same contents I will have counted immediately before only one delete, and won't have repeated it for the other deletes (as I already knew the numbers). – Hugo – 2012-04-10T17:21:05.830

Does your timing include moving the 1,159,211 files in 146,918 folders to the C:\stufftodelete folder? – martineau – 2014-01-19T19:55:09.130

@martineau: No, the test timings were (if I remember correctly) deleting the folders in place. It's usually quick to move a folder to C:\stufftodelete\. Copying is slooow, but you don't want that. If you have another drive (eg E:), also create E:\stufftodelete\ and always move files to the same drive's folder and include it in deletestuff.bat (which also recreates the folders at the end for next time). – Hugo – 2014-01-20T07:22:06.067

18

If you have to delete large directory trees regularly, consider storing the root of that directory tree on a separate partition, then simply quick-format it whenever you need to delete everything. If you need to automate this, you can use this DOS command:

echo Y | format Z: /FS:NTFS /X /Q

where Z: is your 'volatile' partition. Note: the partition must have no label. I blogged about this here.

Mr. Bungle

Posted 2009-08-07T07:00:33.420

Reputation: 297

And use a hardlink to create connection between the disk & directory where files stored if exact directory path needed. – Nime Cloud – 2011-11-22T12:07:43.917

2@NimeCloud: Hardlinks cannot cross filesystems or link directories. – user1686 – 2011-12-12T09:34:19.613

5However, symbolic links can cross filesystems and link directories, so they might be appropriate here. See mklink command (Vista and later; otherwise use junction from Mark Russinovich/SysInternals.) – Robert Calhoun – 2012-07-19T15:37:02.670

9

In command prompt (Start -> Run -> cmd):

del /f /s /q foldername 

lmsasu

Posted 2009-08-07T07:00:33.420

Reputation: 241

1This will fail if the pathname is longer than 256 charachters. – Nicolai Ehemann – 2016-11-15T14:23:47.193

This is great because it lists what its deleting.. but its the same as rd in terms of speed. – bobobobo – 2009-08-08T16:10:03.077

1Oh woah woah woah. Not as good as I thought. It deletes the files quickly but leaves the directory tree intact. – bobobobo – 2009-08-08T16:13:59.947

1Which I might add takes quite a long time to clear out using rd. – bobobobo – 2009-08-08T16:15:01.553

3

@bobobobo I did some tests with ~30GB/1,000,000 files/15,000 folders and del+rd is nearly three times faster than just rd: http://superuser.com/questions/19762/mass-deleting-files-in-windows/289399#289399

– Hugo – 2011-06-01T21:25:23.670

I think rmdir is faster is some situations. del . /f /s /q is nice because you can also delete the contents of all files within a directory. After you are finished, it is just a matter of deleting the folders which is fast enough. Also, remember attrib -s -h -r . /s if you can not delete folders because of hidden files. – Sun – 2013-12-23T17:40:33.337

7

In addition to copying/moving files very fast (using its own API), TeraCopy can delete files and it's very fast too. Ever since finding out TeraCopy I don't use a computer without it installed (if I'm gonna copy/move/delete).

The same installer installs x64 edition but to use it you have to manually force it.

The beta which I recommend over the stable versions: http://blog.codesector.com/2010/09/22/teracopy-2-2-beta-3/

rautamiekka

Posted 2009-08-07T07:00:33.420

Reputation: 345

1Thank you! I've been using TeraCopy for years, but never knew about the mass-delete function. And you're right, it's much, MUCH faster! – AzDayton – 2017-05-23T20:11:17.767

Where do you config teracopy to do file deletion? – Amalgovinus – 2019-06-06T17:56:46.997

1@Amalgovinus You don't. You either open the program and add files/folders through drag-drop or right-click, or right-click your files/folders in Explorer/on desktop. Once it's done walking them, just click 'Delete' and 'Move to Recycle Bin' if you want, but I favor 'Delete from disk'. However, you'll need to boot TeraCopy as Admin to conf it to show up in desktop/Explorer right-click if it's not done by default. – rautamiekka – 2019-06-06T22:48:53.260

Have you ever done or seen some like-with-like time comparisons of TeraCopy vs rmdir or del or del+rmdir? – Hugo – 2011-06-01T21:29:25.150

1@Hugo: Stupid question. I wouldn't recommend TeraCopy if it wasn't seenable by plain eye it's faster ! – rautamiekka – 2011-06-20T20:50:31.817

5So do you estimate it's 10% faster? Twice as fast? Ten times as fast? A million times faster? It would great if you could time it against rmdir with two large and identical test sets. – Hugo – 2011-06-20T21:05:45.943

@Hugo: Depending on file size/count & fragmentation & CPU, 2x to 5x but generally too fast to consider using Win-integrated means for big and/or crapload of files if you'd save a moment or 20. [re-edited after accidental hit of Enter] – rautamiekka – 2011-06-20T22:18:26.233

6

I did a bat file that do the same.

@echo off
echo --------------------WARNING--------------------
echo folder "%~1" will be deleted
echo --------------------WARNING--------------------
pause
echo Deleting folder: "%~1".
time /T
del /f/s/q "%~1" >nul
rmdir /s/q "%~1" >nul
echo Done.
time /T
echo --- Taking ownership.
takeown /f "%~1" /r /d y >nul
icacls "%~1" /reset /t >nul
icacls "%~1" /setowner "%username%" /t >nul
echo Done all.
time /T

it do the work in two step, one, it try to delete the files. And the second is to try to take ownership of the files. y should be changed according your local (stand for yes in english). If fail the task (for example if fail because permission) then you must run it again. However, the second round will not try to delete the files deleted in the first round, so it could be pretty quick.


How to use it. Save as delfolder.bat in a path route (for example c:\windows), then run it as

 delfolder "foldername"   

where foldername is the name of the folder

In my test, deleting 123'000 files took 3 minutes (sata 7200rpm). YAY!

magallanes

Posted 2009-08-07T07:00:33.420

Reputation: 461

1Usually you should not use %1 in a script, but "%~1". This way, if the argument has been passed with quotes, it is dequoted and requoted. If it has not been passed with quotes, it is quoted. – Benoit – 2018-01-23T07:28:35.203

4

Using the code below usually works well for me.

mkdir empty_dir
robocopy empty_dir dir_to_wipe /mir /r:0 /w:0 /e
rmdir empty_dir dir_to_wipe

Benoit

Posted 2009-08-07T07:00:33.420

Reputation: 6 563

It is similar to the use of rsync in Linux. It's even here faster then rmdir? – Hastur – 2016-03-24T15:06:55.033

FYI, this robocopy is very different from del /f/q/s + rmdir /s/q. If your dir_to_wipe has junktions to c:\some_important_dir you will end up losing your data. – Pavel P – 2018-01-16T18:54:04.200

@Pavel then use /XJ to exclude junction points and symbolic links? – Bernhard – 2019-05-22T09:48:29.140

This did the trick for a ~36GB directory that had hundreds of directories and files that were > 256 characters causing the del/rmdir "method" to fail miserably. This took seconds, Windows Explorer was estimating HOURS! – None – 2019-07-19T00:48:58.910

Add a > nul to the end of that robocopy call so you don't have to see it onscreen. – JoshDM – 2013-11-18T18:13:05.403

3

Use the rd /s command from the command prompt.

JP Alioto

Posted 2009-08-07T07:00:33.420

Reputation: 6 278

2No. This is as slow as deleting from explorer. – bobobobo – 2009-08-08T16:08:25.373

Well, its alright. Works better than del /f /s /q, anyway. – bobobobo – 2009-08-08T16:15:41.583

3Well, this is MUCH better than using Explorer to delete, actually. It continues to delete, even if a delete or two will fail (where explorer chokes and aborts) – bobobobo – 2009-08-08T20:14:09.113

2

The best practical solution is probably to move the folder out of the way somewhere (e.g. the Recycle Bin) and then start deleting it. It'll take ages, but at least it'll be out of the way.

I'm pretty sure the time required to delete all those files is an inherent requirement of the task, not an inefficiency in the implementation of deletion.

Ryan C. Thompson

Posted 2009-08-07T07:00:33.420

Reputation: 10 085

1

Do you have short file name generation enabled? If so, do you really need it? Removing a file is only a metadata opeartion. So if you've got twice the number of names to remove, the amount of work is significantly higher.

MSalters

Posted 2009-08-07T07:00:33.420

Reputation: 7 587

1

Install Cygwin and use rm -r. But that's likely to be overkill.

CarlF

Posted 2009-08-07T07:00:33.420

Reputation: 8 576

That's exactly the same as rd /q /s ... – Joey – 2010-03-17T09:28:59.350

Too much task, and almost equivalent speed! – adeelx – 2010-04-04T03:51:43.300

While waiting for a large collection of files to delete I had enough time to install Cygwin and try rm -rf. About 10 minutes in I aborted and went back to del /f/s/q because under Cygwin, disk usage was only 50%. Under del, disk usage is pegged at 100%. So I figure it must be deleting faster. CPU usage is <1% for both methods. I'm not sure on the file number because I closed the Windows File Deleter after it spent 10 minutes calculating. – Derek Ziemba – 2018-04-18T04:18:34.757

1It's no worse than the several other install-tool-X-and-use-it answers here, and shouldn't be voted down if they are not being. – JdeBP – 2011-06-01T22:02:25.573

2Is cygwin rm really as fast as cmd.exe builtin rd? Because the cygwin emulation of posix interface is rather slow as it requires doing some extra work, that rm does not actually appreciate. – Jan Hudec – 2013-06-18T11:59:14.147

0

This is provided to compliment and augment the steps taken in the previously mentioned answers.

The given two methods above seem quite effective, but to determine the performance would be difficult unless they are benched against the exact same conditions:

  • Physical Hard Drive Device & Model
    • Preferably, ATTO/ Crystal Disk benchmarked before cloning the data on them
  • Same Partition/ Volume Location & Geometry
    • The way to do this would be defrag the sample partition/ folders and clone them exactly as they are 2 exactly the same types of Hard Drives (from previous step)
  • Then execute the configurations via Batch File and use Powershell Cmdlet to log and measure their performance.

    • Ideally, add some mechanisms to prevent any caching biases due to recency of similar activity.
  • The following are some sample folders (named as total size) that I experimented on but was not able to get any conclusive performance / results.

r.bat

rmdir /s/q 3.2G

rd.bat

del /f/s/q 3.3G > nul
rmdir /s/q 3.3G
  • I'd have posted the output Log files, but I believe the variance is huge due to sizes & distribution of folders files etc.

Powershell:

PS S:\T> Measure-Command { S:\T\rd.bat } > rdlog.txt

PS S:\T> Measure-Command { S:\T\r.bat } > rlog.txt

PS S:\T> Measure-Command { S:\T\rd.bat } > rd_1.7G_log.txt

PS S:\T> Measure-Command { S:\T\r.bat } > r_1.8G_log.txt

PS S:\T> Measure-Command { S:\T\r.bat } > r_2.4G_log.txt

PS S:\T> Measure-Command { S:\T\rd.bat } > rd_2.7G_log.txt

PS S:\T> Measure-Command { S:\T\r.bat } > r_3.2G_log.txt

PS S:\T> Measure-Command { S:\T\rd.bat } > rd_3.3G_log.txt

Alex S

Posted 2009-08-07T07:00:33.420

Reputation: 811

fails with names > 256 characters – None – 2019-07-19T00:50:50.007

0

Have you tried either of these two apps?

Be sure to set the number of overwrites to 0 if you want fastest performance. Do this by clicking options then change the value at the bottom of the dialog.

Enter the path to delete in the Source field and then click Delete.

These apps do not put the files in the recycle bin. Use with care.

Stephen Kellett

Posted 2009-08-07T07:00:33.420

Reputation: 1

0

The answer from Hugo is the fastest but it does not remove hidden or system files in the first pass, so if you want a more complete solution use the parameter /a:

del /f /s /q /a foldername > nul
rmdir /s /q foldername

About the BAT file by Magallanes, be very carefull with takeown and icacls, cause hardlinks share the same permissions and owners, so if there are unsaturated hardlinks in the folder you want to remove, using takeown and icacls will change permissions and owners of files outside the folder.

Using the tool ln.exe you can list all the files hardlinked inside the folder and know if they are hardlinked to files outside the folder (unsaturated hardlinks):

ln.exe --enum foldername > HardlinksList.txt

liamZ

Posted 2009-08-07T07:00:33.420

Reputation: 101

-1

Deleting folder is faster than deleting multiple files.

So if you prefer to use mouse instead of command prompt, you can create directory, move files there and delete it (with Shift+Del as you said).

danijelk

Posted 2009-08-07T07:00:33.420

Reputation: 107

2The directory should be created on the same partition where the files reside. This will make the move action instantaneous. If the new folder and the files are on separate partitions, then the whole copy operation will last a lot. – None – 2009-08-07T07:15:12.670

5Where do you get the idea that deleting a folder is faster than deleting the files within? Deleting a folder contains as its first step to delete all files in that folder, so it can't be faster. Moving to the recycle bin is an entirely different matter, though, but that's not the question here. – Joey – 2009-08-07T22:40:56.027