Performance implications of a large Documents folder

1

1

I have over time noticed a gradual degradation in computer performance, I cannot figure out what it is, and it being my office computer, I cannot run the usual tools to diagnose this problem (no admin privileges)... IT department is always too "busy" to look into it.

The computer runs fine with CPU intensive operations (compiling, image manipulation...etc.) but anything that requires more than the usual amount of disk activity (i.e. loading a program) takes a disproportionate amount of time to do so.

The only thing I can think of is the large documents folder that I have, and possibly the windows search indexer going nuts over it.

My documents folder has ~900,000 files, 73,000 folders and is 140GB large.

Moving those files to another location will take a couple hours (yes, it's slow, that's my problem...) and, in total, take a day to re-setup my system into working condition...

So before I proceed with that, I would like to shoot out this quick question to see if I would be wasting my time doing that :/

Thanks in advanced

Populus

Posted 2013-10-15T18:48:51.200

Reputation: 113

I have 3x as many files at around 500GB. You need your IT department to look at the problem. 900,000 is nothing. If you have files you can delete do so, if you have not backed up files you should do so, but those numbers are not alarming. – Ramhound – 2013-10-15T18:52:16.017

Overall, anytime you use your Operating System hard drive as a storage hard drive, you will see a decrease in performance. I highly recommend just biting the bullet and switch to an external disk for file storage as large as this. A quick fix might involve something as simple as running a disk defrag. I had a user complaining that their phone is being incredibly slow with EVERYTHING, and in fact it was. I moved about 1000 pics from internal storage to external micro SD and everything sped right up instantly. – MonkeyZeus – 2013-10-15T18:55:01.230

@Ramhound are your files in your "My Documents" folder? I understand that windows indexes that folder, and it may be a cause of the slow downs. – Populus – 2013-10-15T18:55:21.483

@MonkeyZeus my computer defrags weekly (scheduled task), I will not have access to an external drive. I do have network drives that I can use, but they are not fast enough for what I do. – Populus – 2013-10-15T18:56:47.847

Can you use Task Manager to look for programs that use your resources? I sometimes have explorer or other tasks using up almost a core for no specific reason. – Doktoro Reichard – 2013-10-15T18:57:40.327

CPU usage is low even during slow downs. I am going to try turning off search indexing on the Users folder for a few days first... – Populus – 2013-10-15T19:03:35.017

"Gradual degradation in computer performance" is usually due to a full %temp% folder in my experience. Could you check how many files are in the root of that directory? Also, how many files are typically in a large directory in your Documents-folder? If you have all your files distributed over those sub-folders this should not be a problem but if you have many thousands of files in one directory it can be a problem. – Rik – 2013-10-15T19:07:31.057

@Populus If CPU usage is low even during slow-downs, then how much memory are you using? My guess, now that you said that, might be related to virtual memory issues... On a related note, are you using a SSD? How much free space does the drive have? Is it a SATA or a PATA? – Doktoro Reichard – 2013-10-15T19:08:55.610

@populus of course where else would I place documents? I place all files on the system disk also – Ramhound – 2013-10-15T19:14:44.520

@DoktoroReichard I have 8GB of ram, I usually end up using about 70% of my RAM (as indicated by task manager). I have 6% free disk space right now... And the hard drive is this one: http://www.seagate.com/em/en/internal-hard-drives/desktop-hard-drives/desktop-hdd/?sku=ST3500413AS

– Populus – 2013-10-15T19:15:17.740

If you have access to it then you can try and use the Performance Monitor built into windows. It is a bit advanced but it should be able to clearly tell you what is hogging your hard drive resources. – MonkeyZeus – 2013-10-15T19:16:24.263

16% free disk space is the clear culprit here. If other people used this computer before you then their profiles are probably still on there and using GBs of space. Getting above 10% free space should be your top priority at this moment. Do you use Microsoft Outlook? because the OST, PST, and archive files could be taking up huge space. Do you clear your internet cache? In the past I have freed up 3gb of space by clearing cache. Does your recycling bin have a bunch of crap in it? Permanently delete the contents of your recycling bin. – MonkeyZeus – 2013-10-15T19:21:03.393

Also your hard drive has sub-par performance specs so that could also be an issue but possibly an issue you cannot solve easily. – MonkeyZeus – 2013-10-15T19:22:42.370

@MonkeyZeus Thanks for advice. Yes, unfortunately I cannot change the hardware of my machine... Shame, as the rest of the machine is decent with the Windows Experience Index of 7.7 (CPU), 7.7 (RAM), 6.1 (GFX) but the hard disk scores 5.9 (which is OK I guess, but it's a bottleneck) – Populus – 2013-10-16T14:02:43.940

Sounds like a beast of a machine but a 5.9 for the hard drive is pretty much what you'll see with most hard drives. I don't know how well the Experience Index is able to actually assess HDDs but from a user experience perspective an SSD is the ultimate answer. – MonkeyZeus – 2013-10-16T14:46:54.410

Answers

2

The problem, as @MonkeyZeus stated and as I tried to figure out, is free space.

Although you can run almost any OS with 0 KB of free space in the main disk (been there, done that, on both Vista and OSX), the system will suffer. Why?

There exists a reserved space called virtual memory (or paging file or swap partition), that serves as a pool of space to be used as memory for almost any application that exists. Why is it used instead of regular RAM? Performance: it is better to have the processes that use more memory more frequently on RAM rather than latent, check-once processes.

That space is somewhat variable, and the OS will try to achieve a balance (when in need) between used and virtual disk space. If programs can't store information on the disk, the only place that remains is the RAM. That will end up bottlenecking your entire system.

As you use somewhat intensive programs, that could be the main cause for your lack of performance.

The solution, as stated before, is to try and clear space. Outlook files, temporary files, cache files, duplicate files, all contribute to the lack of space. There is no one-size fits-all rule for these kinds of situation but clearing temporary folders is a good start.

Doktoro Reichard

Posted 2013-10-15T18:48:51.200

Reputation: 4 896

As suggested by @MonkeyZeus and Doktoro Reichard I have deleted a bunch of useless backups that I had. Now my free space is at 26%. Will see if performance is better over the rest of the day and the next. Will award answer if it gets better :P – Populus – 2013-10-15T20:37:37.027

Also, if it is still slow with 26% free try defragmenting. A disk which has had 6% free would probably be heavily fragmented. – Rik – 2013-10-15T20:52:10.770

Adding to what @Rik said, defragmenting can only be done if there exists some free space, in order for the program to move files while reordering. You did said you had defrag set up as a scheduled task. As explained, that could not be efficient if your disk was full. – Doktoro Reichard – 2013-10-15T21:16:59.687

My computer has it set to defrag every week, so that isn't a problem at all. – Populus – 2013-10-15T21:24:10.400

I have read it before, so that's why I warn you. Defragmentation is only optimal if you have space in which the program can move files. Just because you defrag every week it means nothing if the program can't optimize it's functions due to a lack of space. – Doktoro Reichard – 2013-10-15T21:53:09.737

@DoktoroReichard thanks for advice, things seem to be running smoother now, I'll assess it over the rest of today and give you the points :) – Populus – 2013-10-16T14:03:54.743

Asked: 2013-10-15T18:48:51.200

Viewed: 84 times

Active: 2013-10-15T19:29:52.430