Deleting large amount of files in Windows is slow

Question

I have a Windows XP box with an NTFS disk and deleting large amounts of files is extremely slow. If I select a folder that contains a large number of files in a tree of folders and delete (using shift-del to save the recycle bin) it takes time that seems to be directly proportional to the number of files within the folder before it even pops up the confirmation box. It then takes an even longer time to delete each file in the folder.

Is there a way to delete a folder in Windows and not having the time taken proportional to the number of files within it?

Answer 1

Is there a way to delete a folder in Windows and not having the time taken proportional to the number of files within it?

I don't think so, but some methods are clearly much quicker than others.

The worst way is to send to Recycle Bin: you still need to delete them. Next worst is shift+delete with Windows Explorer: it wastes loads of time checking the contents before starting deleting anything.

Next best is to use rmdir /s/q foldername from the command line. del /f/s/q foldername is good too, but it leaves behind the directory structure.

The best I've found is a two line batch file with a first pass to delete files and outputs to nul to avoid the overhead of writing to screen for every singe file. A second pass then cleans up the remaining directory structure:

del /f/s/q foldername > nul
rmdir /s/q foldername

This is nearly three times faster than a single rmdir, based on time tests with a Windows XP encrypted disk, deleting ~30GB/1,000,000 files/15,000 folders: rmdir takes ~2.5 hours, del+rmdir takes ~53 minutes. More info here.

This is a regular task for me, so I usually move the stuff I need to delete to C:\stufftodelete and have those del+rmdir commands in a deletestuff.bat batch file. This is scheduled to run at night, but sometimes I need to run it during the day so the quicker the better.

Answer 2

Is there a way to delete a folder in Windows and not having the time taken proportional to the number of files within it?

Well, yes, format the partition. I'm a bit surprised nobody suggested that in the previous 9 years.

It's pretty radical, but if you anticipate doing this frequently for a specific folder, it might be worthwhile creating a separate partition for it.

---

If that's too radical, the other answers are your only hope. There is a good explanation why on serverfault. It's for linux and XFS filesystems, but the same logic applies here. You can't improve much on build-in OS functions.

However, if you know the paths to all the files you wish to delete, then you can save on calls that list the directory contents and call remove directly, saving some overhead. Still proportional to the number of files though.

Personally, I like some from of progress report to ensure myself that the program didn't die. So I like to delete stuff via python. For example, if all the files are in one directory without sub-directories:

import tqdm
import sys
import os

location = sys.argv[1]
directory = os.fsencode(location)

with os.scandir(directory) as it:
    for dir_entry in tqdm.tqdm(it):
        try:
            os.remove(dir_entry.path)
        except OSError:
            pass  # was not a file

This deletes about 250 files/s on my 12 year old SEAGATE ST3250620NS. I'd assume it will be much faster on your drive.

However, at this point it's just micro-optimization, so won't do very much unless you have millions of files in one directory. (like me, lol, what have I done D:)

Answer 3

I used Hugo's original answer to create a .bat file that I use when deleting NPM projects. I added a path variable and only have to copy and paste the path once. Double-click the .bat file and it does all the work - no need to type everything.

set path="FOLDER_PATH"
del /f/s/q %path% > nul
rmdir /s/q %path%

Example usage:

set path="C:\Projects\My React Project"
del /f/s/q %path% > nul
rmdir /s/q %path%

Answer 4

Make sure you're not backing up files to the cloud and trying to delete them at the same time!

With many cloud backup solutions files will get locked while they are being backed up and then you have to wait for them to be backed up.

If you're having this issue with say a temp directory (or something that doesn't need backing up) make sure that temp directory isn't selected in your backup set.

Answer 5

I've found that folders with several layers of directories tend to really slow down Window's ability to remove them quickly. I was working on a project where it took 5 levels to get to the node_modules folder, which is always a beast to delete, even with

del /f/s/q foldername > nul
rmdir /s/q foldername

What I end up doing in this situation is navigating down to the node_modules folder or whatever directory has the deepest levels and just start selecting and deleting about a dozen or so directories at a time. If I get multiple deletes going, this forces the Recycle Bin to work in parallel processes rather than the single thread I believe it uses, dramatically speeds up the process.

Once my deepest directory is empty, I go up a few levels and do the same thing. This has cut down deletes that have taken me over an hour to just a few minutes.

Its a very manual process and could likely be scripted with some success, but its what's worked for me

Answer 6

Install gnutools for windows and run:

find YOURFOLDER -type d -maxdepth 3 | xargs rm -Rf

Answer 7

did you try using command prompt

rmdir /s /q foldername

all large file operations in GUIs are slow - mostly because visual feedback (progress bar) has to be repainted many times