Windows with a better filesystem

From my observations as a Java developer working on Windows workstations, NTFS is slow compared to Linux filesystems. Question is, is there anything in the NTFS driver that can be manually tuned, for example give it more memory for cache? Enable some experimental algorithms? If that's not available, is there perhaps another filesystem that can be used on Windows, maybe even commercial, that's faster than NTFS?

To be clear, I'm not looking to improve compilation speeds for Maven projects, I'd like to get an overall improvement for the OS. I get a feeling that NTFS is long outdated and slow compared to Linux filesystems. It strikes me as weird that the most popular OS on planet has only one filesystem which still requires manual defragmentation. Perhaps there is an alternative?

Update: Here is what's slow according to my observations. I'm building/packaging a project, which means lots of read/write operations on disk. The build system is cross-platform (Java, Maven), so I can perform exactly the same actions when booted to Ubuntu, for example.

On Linux my builds are at least 1/3 faster. Hence the question about filesystem. I'm sorry if it's misplaced.

Yuri Ushakov

Posted 2011-02-02T08:25:24.977

Reputation: 243

1@sinni800 That's just rubbish, sorry. Whilst I admit that as a Desktop OS, Linux is far from perfect, as a Server OS it is far better than Windows for nearly all workloads. There is no question of the 96.6% of top 1 million websites using a beta quality operating system. – mjaggard – 2016-09-09T10:55:14.060

7In what way is NTFS slow? How are you measuring its speed to know that it is slower than any other filesystem for exactly the same operations? NTFS is the only natively supported filesystem (apart from FAT which has limitations) that Windows itself can boot from, though do I believe that you may be able to get an ext3 filesystem driver for non-boot partitions. – Mokubai – 2011-02-02T08:38:23.473

Yeah, please enlighten us, how is NTFS slow? – JL. – 2011-02-02T08:51:24.030

Rather show us a benchmark of how slow it is with the specs of your system :-) – Ivo Flipse – 2011-02-02T08:54:16.270

1But what is the question here? Is it how you can speed up NTFS or whether Windows can use a different file system? – Ivo Flipse – 2011-02-02T08:57:18.140

1@ivo-flipse: I'm sorry for causing confusion here, I'm interested in learning something about both questions. Like, is there no proven alternative to NTFS on Windows? And if not, would it be possible to generally increase NTFS' performance, like give more memory to some caches or enabling some experimental mechanisms? – Yuri Ushakov – 2011-02-02T09:14:36.567

Do you actually think about enabling experimental mechanisms on a RTM Windows system? Sorry, this is not open source where almost all the time everyone is a beta tester :). (Not to talk bad about open source. But you have to admit that to be maybe a little true :) – sinni800 – 2011-02-02T11:39:47.540

I'm not so sure if it's an NTFS issue or a kernel I/O subsystem issue in general. I've noticed that Linux tends to handle a lot of small file much better than Windows (which makes sense, given how Linux uses lots of small files much more than Windows), and I've noticed Linux is faster at directory traversal, but I haven't noticed the throughput for e.g. copying large files to be better. – user541686 – 2012-10-16T07:37:49.043

Answers

While I'd love to see something like ZFS available for Windows hosts, NTFS isn't a horrible filesystem. It supports most "modern" filesystem features (extended attributes, journaling, ACLs, you name it), but it's hampered by Explorer and most other apps not supporting any of these.

One thing that will absolutely kill its performance is having "too many" entries in a directory. Once you pass a couple thousand entries in one directory, everything slows to a crawl. Literally the entire machine will halt waiting for NTFS to create or remove entries when this is happening.

I used to work with an app that generated HTML-based docs for .NET assemblies; it would create one file per property, method, class, namespace, etc. For larger assemblies we'd see 20+k files, all nicely dumped into a single directory. The machine would spend a couple of hours during the build blocked on NTFS.

In theory, Windows supports filesystem plugins, which would make native ZFS, ext3 or whatever (even FUSE) possible. In practice, the APIs are undocumented, so you're completely on your own.

Now, since you're doing Java development, could you install a different OS on your machine, or use a VM on top of Windows?

Also, you might want to try some platform-independent filesystem benchmarks (iozone, bonnie... there are probably more modern ones I don't know off the top of my head, maybe even a few written in Java) to see if it's actually the filesystem holding you back, or if it's something else. Premature optimization and all that...

chrish

Posted 2011-02-02T08:25:24.977

Reputation: 826

The Installable File System has been documented at least since 2000. Lack of plugins was always down to difficulty and lack of interest. It's still difficult, but FUSE and FSDK are now available. – user165568 – 2019-06-07T07:38:14.437

I have used Ext2 on Windows for years. Works fine! – oligofren – 2020-02-24T09:18:20.010

Turn off Last Access time
Turn off Short File names
Turn off Delete Notification
Turn off Indexing
Disable journeling
Disable shadow copy and previous versons and quotas and shares.
Enable bypass traverse checking

I guess that the real solution would be to rewrite your build system so that it used the native windows file system API, instead of the unix API (fopen etc) itself underneath the portability framework. But that's not going to happen, so basically you are stuck with whatever level of performance they thought was acceptable.

Often when using systems not originally written for Windows, you find that directory traversal has been handled very poorly, so make sure you have a very flat directory tree.

user165568

Posted 2011-02-02T08:25:24.977

Reputation: 421

1This would be a better answer if it gave some more information. 1. Some measure of how much time might be saved for each and 2. How to do these changes. – mjaggard – 2016-09-09T10:40:20.870

Yeah, how do you do this? This is all fine and dandy on Linux, just modifying the fstab, but on Windows? – oligofren – 2020-02-24T09:19:35.130

There is one file system which is supported by new Windows OS AND is faster than NTFS. It's exFAT. There's a possibility to use it for the system drive. But it's unknown what complications it might have.

You can certainly use it for other partitions though. It's faster with random read/write operations. Perfect for an SSD for example.

sinni800

Posted 2011-02-02T08:25:24.977

Reputation: 3 048

The reason that test said exFAT was faster was probably because they didnt test it on a directory containing >32768 entries. Also, newer versions of Windows could have slowed down further due to internal changes. God (MS) knows what happens in a opendir() (or equivalent) call. – MattBianco – 2017-05-10T12:53:11.993

1Tried it just now, and exFAT seems slower. First build on NTFS: 36.083s, second: 19.884s. First build on exFAT: 41.160s, second: 27.291s. – Yuri Ushakov – 2011-02-02T12:42:14.333

Huh. exFAT is the only other file system that is now natively supported by Windows and COULD be used as the system drive (permissions are present). I saw a test once, online, and it said exFAT was faster with random reads. – sinni800 – 2011-02-02T13:01:35.567

Do you have an anti virus program running that's doing on access/write checking for your project folder?

Compiling involves reading and writing lots of small files in quick succession, which can overwhelm the virus scanner. Add your project folder to the list of excluded folders and see if that improves the situation.

On Linux you (probably) don't have any anti virus software....

ChrisF

Posted 2011-02-02T08:25:24.977

Reputation: 39 650

Defender is off, AV is shutdown (or uninstalled completely, as case may be). – Yuri Ushakov – 2011-02-02T11:57:07.487

@Yuri - oh well. – ChrisF – 2011-02-02T11:58:52.020