High disk reponse time every Saturday starting at 8 am

0

1

I have a server running Microsoft windows server 2012 R2 and it has a very odd problem. Every week on Saturday starting around 8 am the disk reponse time goes up to over 1000ms, every other day of the week this problem does not seem to surface.

I've been trying to find the cause of this issue for a few weeks now but I am not able to pinpoint it.

The server is a:

Dell PowerEdge R410
2x 2.26 Ghz (6 core) Intel Xeon L5460
2X 16 GB DDR 3 
Perc h700 raid controller
4x 3TB 7.2K 3.5" 
Raid 5 (3 disk) + 1 disk hot spare

The server is running the following roles/software:

Active Directory
IIS 
Cache

(Cache is a software package by intersystems)

If any more information is required please let me know

The problem is surfacing at the moment of writing, any help our tips would be highly appreciated.

Edo Post

Posted 2018-12-29T08:47:18.040

Reputation: 97

Answers

2

Sounds like you are using the Dell PERC Patrol Read [archive] feature.

From the article:

Patrol Read automatically runs every seven days on configured SAS and SATA HDDs.

If you have low I/O on Saturday at 8 AM, Patrol Read would run at a higher priority and may cause the disk response time to spike. Supposedly, if you increased your I/O usage, Patrol Read would back off to let you perform your I/O activity:

Patrol Read adjusts the amount of controller resources dedicated to Patrol Read operations based on outstanding disk I/O. For example, if the system is busy processing I/O operation, then Patrol Read uses fewer resources to allow the I/O to take a higher priority.

If you want consistent performance, you can try turning Patrol Read off, but in the future, bad sectors may begin to lurk in infrequently accessed areas of affected disks, which may hurt your RAID 5 recovery chances.

As for how to configure Patrol Read:

Patrol Read Mode can be set in BIOS configuration utility and UEFI RAID Configuration Utility.

Deltik

Posted 2018-12-29T08:47:18.040

Reputation: 16 807

Looks like patrol read was indeed running and starting at Saturdays at 8am sharp. I stopped it for now, going to keep monitoring the response times (highest at the moment is 15 ms which is great) and change the sceduale to start at Saturday night instead. It looks like the small file acces Cache does (up to 8kb per file io) was not registering with patrol reads "Activity" and it just kept going. It was a feature I had no knowledge of, but it sure seems like it was causing the disk slowdown. You my friend, are a hero – Edo Post – 2018-12-29T09:42:55.810

Also, i was able to configure patrol read through the perc cli – Edo Post – 2018-12-29T10:07:30.623

Asked: 2018-12-29T08:47:18.040

Viewed: 293 times

Active: 2018-12-29T09:10:19.993