Deleting many directory entries will take longer than deleting a single directory of the net size. When deleting many files, it takes many commands instead of a single when a single file is deleted.
Now, when one big file is deleted, the filemapper is told to unallocate the memory of that big file, but the 1s and 0s of the file are not reset. The mapper is just told it is free to write new stuff over the old 1s and 0s. Basically, it is one command, that takes time to clear up all of the memory allocations of the fragmented file. When you delete a million small files, you are sending a million commands to the mapper, that has to do each one at a time.
So the solution is to use software to wipe a drive of deleted files and the 1s and 0s left behind. It will take one single command to wipe the space. Use the software only if you think the files are not needed to be recovered later.
12What type of storage device are the files on an HDD or SSD? If we are talking about a HDD what is the cache size? – Ramhound – 2019-09-22T14:26:34.100
7
A number of options are explored at https://superuser.com/questions/416071/how-to-delete-huge-number-of-files-on-windows for speedily deleting a folder with many files on Windows.
– Gary – 2019-09-22T15:01:29.327@Ramhound It is a HDD with 7200 RPM, Drive Controller: Serial ATA 6Gb/s @ 6Gb/s, but for cache buffer size HWinfo says N/A for some reason. – Kagaratsch – 2019-09-22T21:12:46.463
2@Kagaratsch what is the source of your batch...if you could get each batch as an ISO file then you could use it in a virtual drive and on finishing you would only have to eject and delete a single ISO file. – Elmo – 2019-09-23T05:30:13.253
2Maybe you can just store all your files in a disk image which you mount to a drive letter when you need them and delete it, when you don't need the files anymore – Benj – 2019-09-23T13:06:08.243
3@Kagaratsch: is it an external hard drive, with "optimize for safe removal" or whatever Windows calls it? That would probably make Windows use a lot of I/O barriers to make sure metadata is on disk before sending another write. Or maybe disable write cache entirely. – Peter Cordes – 2019-09-23T19:04:54.850
2If you run
devmgmt.msc
and right-click to get "Properties..." on that drive... what do you see under "Policies" (second tab from the left)? If there's no tick on both checkboxes, that's bad for performance. Note that while enhancing performance, they both reduce safety a bit insofar as suprise-disconnecting the drive without telling Windows first will most probably result in data loss. If you don't plan to hot-plug, it's reasonably safe. – Damon – 2019-09-23T19:47:50.4831Also, is this an NTFS drive (Properties -> File system)? It's not too uncommon to have external drives as FAT, which is considerably slower than NTFS in many cases. – Luaan – 2019-09-24T07:00:46.577
1@Damon It is actually a HDD built into my PC, is it safe to enable this setting in this case? – Kagaratsch – 2019-09-24T11:56:02.423
2Yes (with a tiny "if"). If power failure is a common issue and no UPS is present (but then, in that constellation you are always in trouble), it's not safe. Otherwise, just abstain from opening the case and pulling off cables mid-operation (which I assume you didn't plan to do!), and you're good to go. – Damon – 2019-09-24T17:07:12.400
@Luaan: Not possible under Windows unless the drive is 32 GiB or smaller (modern versions of Windows refuse to format drives larger than that as FAT16B or FAT32). – Sean – 2019-09-25T02:48:16.917
Try wsl?
– mcalex – 2019-09-25T09:32:18.487