Improving NTFS performance for small files

0

0

I would like to know if there are things I can do to improve the performance of NTFS when working with lots of tiny files, such as is the case when working with Git, Subversion, NPM or other tech that creates a gazillion small files. I have read other threads on SU that explains a bit why working with small files on NTFS is so slow, but I would like to know how I can improve it, if at all.

For instance, deleting a specific large node_modules directory created by the development tool NPM takes a from an instant to a few seconds at most on most Linux filesystems, but it takes 35 seconds when done using bash running in Windows Sybsystem for Linux (WSL). I know it is not WSL in itself that is the problem, as I can remember the issue being even worse when using Explorer to delete folders. I remember firing up bash on Cygwin or MSYS2 just to save time doing this a few years ago, perhaps because the GUI operation needs to do a lot of bookkeping due to needing to support "Move to paper basket" functionality (essentially traversing the entire file system).

I know from other situations that disabling Windows Defender (the Anti-Malware protection) from working on certain directories can boost performance a lot, since it seems to intercept file system calls. The package manager Chocolatey was severely handicapped by this the last time I used it, for instance, making unzipping a file take 15 minutes, compared to a few seconds when disabling Defender. Others report 5x improvement in file creation times on WSL when excluding it from being scanned (though might not be so wise). My own experiements with disabling Defender shows a small improvement (10%) for deleting files, but a 6x improvement in file creation (40s vs 4m14s)!

So let's just assume Windows Defender has had the WSL folder added to its exclude list when listing possible improvements.

oligofren

Posted 2020-02-24T09:14:51.387

Reputation: 842

are you using WSL or WSL2? In WSL performance is very bad because additional POSIX metadata must be stored in the file and special handling is done for them. In that case using git on Windows will be a lot faster than git on WSL. But WSL2 runs inside a VM on ext4 so you won't have such issues

– phuclv – 2020-02-27T02:52:42.967

@phuclv I was running WSL, but now I have joined the Insider program to get access to WSL2. I ran the upgrade command, so now I believe I am on WSL2... The performance is now pretty ok, but copying a huge directory tree wasn't really faster than what standard WSL was after disabling Windows Defender, so ... – oligofren – 2020-02-27T06:22:26.190

are the directories inside the VM? If yes then it's quite strange since now NTFS is only used to store the ext4 image. To make sure that you're on wsl2 run wsl --set-version <Distro> 2

– phuclv – 2020-02-27T06:25:58.880

No answers