How does NTFS compression affect performance?

Question

I've heard that NTFS compression can reduce performance due to extra CPU usage, but I've read reports that it may actually increase performance because of reduced disk reads. How exactly does NTFS compression affect system performance?

Notes:

• I'm running a laptop with a 5400 RPM hard drive, and many of the things I do on it are I/O bound.
• The processor is a AMD Phenom II with four cores running at 2.0 GHz.
• The system is defragmented regularly using UltraDefrag.
• The workload is mixed read-write, with reads occurring somewhat more often than writes.
• The files to be compressed include a selected subset of personal documents (not the full home folder) and programs, including several (less demanding) games and Visual Studio (which tends to be I/O bound more often than not).

Answer 1

I've heard that NTFS compression can reduce performance due to extra CPU usage, but I've read reports that it may actually increase performance because of reduced disk reads.

Correct. Assuming your CPU, using some compression algorithm, can compress at C MB/s and decompress at D MB/s, and your hard drive has write speed W and read speed R. So long as C > W, you get a performance gain when writing, and so long as D > R, you get a performance gain when reading. This is a drastic assumption in the write case, since Lempel-Ziv's algorithm (as implemented in software) has a non-deterministic compression rate (although it can be constrained with a limited dictionary size).

How exactly does NTFS compression affect system performance?

Well, it's exactly by relying on the above inequalities. So long as your CPU can sustain a compression/decompression rate above your HDD write speed, you should experience a speed gain. However, this does have an effect on large files, which may experience heavy fragmentation (due to the algorithm), or not be compressed at all.

This may be due to the fact that the Lempel-Ziv algorithm slows down as the compression moves on (since the dictionary continues to grow, requiring more comparisons as bits come in). Decompression is almost always the same rate, regardless of the file size, in the Lempel-Ziv algorithm (since the dictionary can just be addressed using a base + offset scheme).

Compression also impacts how files are laid out on the disk. By default, a single "compression unit" is 16 times the size of a cluster (so most 4 kB cluster NTFS filesystems will require 64 kB chunks to store files), but does not increase past 64 kB. However, this can affect fragmentation and space requirements on-disk.

As final note, latency is another interesting value of discussion. While the actual time it takes to compress the data does introduce latency, when the CPU clock speed is in gigahertz (i.e. each clock cycle is less then 1 ns), the latency introduced is negligible compared to hard drive seek rates (which is on the order of milliseconds, or millions of clock cycles).

---

To actually see if you'll experience a speed gain, there's a few things you can try. The first is to benchmark your system with a Lempel-Ziv based compression/decompression algorithm. If you get good results (i.e. C > W and D > R), then you should try enabling compression on your disk.

From there, you might want to do more benchmarks on actual hard drive performance. A truly important benchmark (in your case) would be to see how fast your games load, and see how fast your Visual Studio projects compile.

TL,DR: Compression might be viable for a filesystem utilizing many small files requiring high throughput and low latency. Large files are (and should be) unaffected due to performance and latency concerns.

Answer 2

You have a quite slow disk, so your question does have merit. NTFS compression is processor-intensive and is tuned for speed rather than compression efficiency.

I would expect that you would see a (very) small improvement for read operations. However, when accessing a file residing in the system cache you will have a performance hit, since it will have to be decompressed again on every access.

You will of course see that write operations will be slower because of the additional compression.

Copying files on this same NTFS disk requires decompression and compression, so these will suffer the most.

NTFS Compression can also increase fragmentation significantly, but this is not a problem for most 'typical' computers under 'typical' work loads.

Many types of files, such as JPEG images or video or .zip files, are basically uncompressable, so these files will be slower to use and without any space saved.

Files smaller than one disk cluster (typically 4K) are not compressed, as there is no gain. However, even smaller cluster size is sometimes advised when compressing the entire volume.

NTFS compression is recommended for relatively static volumes or files. It is never recommended for system files or the Users folder.

But as hardware configuration varies from one computer model to another, depending on disk, bus, RAM and CPU, only testing will tell what the exact effect of compression will be on your computer model.

Answer 3

I explained it here in the Wikpedia entry for NTFS:

---

NTFS can compress files using LZNT1 algorithm (a variant of the LZ77 [23] ). Files are compressed in 16-cluster chunks. With 4 kB clusters, files are compressed in 64 kB chunks. If the compression reduces 64 kB of data to 60 kB or less, NTFS treats the unneeded 4 kB pages like empty sparse file clusters—they are not written. This allows not unreasonable random-access times. However, large compressible files become highly fragmented as then every 64 kB chunk becomes a smaller fragment. [24][25] Compression is not recommended by Microsoft for files exceeding 30 MB because of the performance hit.[citation needed]

The best use of compression is for files that are repetitive, written seldom, usually accessed sequentially, and not themselves compressed. Log files are an ideal example. Compressing files that are less than 4 kB or already compressed (like .zip or .jpg or .avi) may make them bigger as well as slower.[citation needed] Users should avoid compressing executables like .exe and .dll (they may be paged in and out in 4 kB pages). Compressing system files used at bootup like drivers, NTLDR, winload.exe, or BOOTMGR may prevent the system from booting correctly.[26]

Although read–write access to compressed files is often, but not always [27] transparent, Microsoft recommends avoiding compression on server systems and/or network shares holding roaming profiles because it puts a considerable load on the processor.[28]

Single-user systems with limited hard disk space can benefit from NTFS compression for small files, from 4 kB to 64 kB or more, depending on compressibility. Files less than 900 bytes or so are stored with the directory entry in the MFT.[29]

The slowest link in a computer is not the CPU but the speed of the hard drive, so NTFS compression allows the limited, slow storage space to be better used, in terms of both space and (often) speed.[30] (This assumes that compressed file fragments are stored consecutively.)

---

I recommend compression only for files which compress to 64KB or less (ie 1 piece). Otherwise, your file will consist of many 64K or less fractions.

MyDefrag does a better job of defragging.

Answer 4

It will make operations slower. Unfortunately, we cannot measure exactly how much or how little it will affect your system. When a file that is compressed gets open, it takes processor power to uncompress the file so the system can use it; when you are done with it and hit Save, it uses more processor power to compress it again. Only you can measure the performance though.

Answer 5

anybody who sees this today should be aware that, in the case of video games, yes even regularly patched ones, enabling compression on the drive or folder can decrease load times, even on slower cpus of today, and even on ssd's (other then the fastest ones most people dont have), you need to defrag regularly though, and i strongly recommend buying perfect disk, once you use its "Smart agressive" defrag, AFTER compression, leave its auto prevention of frangmentation feature enabled, it will keep an eye on activity and auto-optimize to avoid fragmentation, at very little to no perf hit(testd this all the way back to old first gen quads of both amd and intel on modern windows recently infact)

many game files compress insanely well, some games have files that take up disk space, despite being mostly blank...one game i compressed a while back went from 6gb in one of its folders to under 16mb..... (wish i was kidding...talk about wasted space and wasted I/O....)

compressed a buddies steam folder a while back, took it 4 days to compress(its on a 4tb drive and it started 3/4 full), when it was done....he was using around 1/3 of the drive total, defrag took another day(but, it started out horribly fragmented because, he had never done a defrag on it, ever..despite multi mmo's on it...and shitloads of steam/uplay/origin/etc games on it...)

DO NOT compress you pictures/images folders, it wont do any good, and will just make accessing them slower on slow systems(wont even notice on a 1/2 decent rig though...)

i have compressed my drives on every system since nt4, BUT, selectively, i will actually decompress folders where compression does more harm then good, its the "best practices" we came up with way back in the day as gamers, geeks, "it"guys(before that was a term), and, its stayed true, honestly, i wish they had a more fine tuned way to compress drives/data, there use to be a tool that wasnt free but affordable, that got you much better compression results without compressing any data that shouldnt be compressed....

anyway, even many older dual core systems actually benefit overall if you 1. run ccleaner 2. run chkdsk /f from elevated command prompt (type y then restart and let it run the check) 3. compress the drive. 4. defrag with either mydefrag or better, perfect disk, this will take time.. 5. fine any folders containing large files or pictures/other content that dosnt compress well/at all, decompress the folder or just the files, my exp here is, you rarely have to defrag after this part of the process but, its best to check.

i get why some people are against compression, but, having tested it, when used properly, ssd or hdd, and especially slow old hdd's and ssd's, compression when used properly can seriously help not only save space but, performance, even most older dual cores can deal with the average compress/decompress cycles faster then the drive in those systems can move, having tested this, first gen and cheaper older design ssd's, can benefit from compression, not as much as slower hdd's in most cases but, a buddy has a netbook thats got a VERY slow, hard to replace ssd in it, as well as a much easier to replace easy to access ssd slot, but, the stupid thing CANNOT boot from the added ssd without removing the other one physically... (horrible bios, but...for what the unit is, its actually nice, more powerful then it looks..outside the slow ssd thats installed in such a way you have to take the whole thing apart to get to it......), compressing that drive and just having windows and the most basic of apps(like office) on the slow ssd actually sped it up, even in read/write, because its cpu actually ends up waiting for the damn ssd..it dosnt for the faster one he insalled...i suggested just putting the boot loader on the internal ssd and the os on the added but..hes hoping to eventually kill the stupid thing by using most of it for the page file....(its 128gb but, ungodly slow, like i have usb3 flash drives that have better write speads....that cost all on sale at newegg/amazon......)

i STRONGLY suggest compressing at least your games drive/folder...my god the dif that can make on even fast systems!!!

Answer 6

Windows compresses non-recently used data in RAM, with even SSDs being a fraction of the speed I'd guess the performance hit is a non-issue. I'm more concerned about compressed blocks that develop a 1-2 bit error and can't recover some or all of data... or a dictionary error in worst case. Anything that produces a disk that's non-readable on alternate OSes and potentially lowers reliability isn't worth the extra speed it (might) bring, IMHO. Videogame texture pack files and such are usually already compressed, so I don't see how layering another set of compression will improve things. I'd like to see an OS that supports marking files as linear layout on the disk geometry so random r/w isn't used. It speeds things up even on SSDs for certain use cases. My other problem with compression is that since images and movies are already compressed, as are MS Office docs and tons of other formats, you're stuck marking files as compressible and micromanaging it. For a linux source tree or large open source project it could help a lot since compression is usually optimal on text files.

Answer 7

Microsoft Windows NTFS compression should not be used for anything other than log files or, generally speaking, text files or otherwise highly compressible files.

Consider this: historically I have seen performance of file compression stuck at 20-25 MiB/s. This is the normal speed for zipping a file with one 2.4-3.0Ghz processor. NTFS Compression is not multithreaded. This is a huge problem!

Consider decent speed for a hard drive nowadays is 100 MiB/s. If you don't get around 4-5x compression your are massively loosing performance both in reading and in writing. This is what happens.