perfmon reporting higher IOPs than possible?

Question

We created a monitoring report for IOPs on performance counters using Disk reads/sec and Disk writes/sec on four servers (physical boxes, no virtualization) that have 4x 15k 146GB SAS drives in RAID10 per server, set to check and record data every 1 second, and logged for 24 hours before stopping reports.

These are the results we got:

Server1 Maximum disk reads/sec: 4249.437 Maximum disk writes/sec: 4178.946

Server2 Maximum disk reads/sec: 2550.140 Maximum disk writes/sec: 5177.821

Server3 Maximum disk reads/sec: 1903.300 Maximum disk writes/sec: 5299.036

Server4 Maximum disk reads/sec: 8453.572 Maximum disk writes/sec: 11584.653

The average disk reads and writes per second were generally low. I.e. for one particular server it was like average 33 writes/sec, but when monitoring in real-time it would often spike up to several hundreds and also sometimes into the thousands.

Could someone explain to me why these numbers are significantly higher than theoretical calculations assuming each drive can do 180 IOPs?

Additional details (RAID card): HP Smart Array P410i, Total cache size of 1GB, Write cache is disabled, Array accelerator cache ratio is 25% read and 75% write

score 7 · Answer 1 · answered Sep 05 '12 at 19:42

7

Each drive can do 180 random IOPs. Is your workload totally random? I bet what you're seeing is sequential reads/writes.

answered Sep 05 '12 at 19:42

MikeyB

38,725
10
102
186

Damn, +1, beat me to it. Sequential reads and writes don't have any seek delay, so they're much faster. – HopelessN00b Sep 05 '12 at 19:48

score 4 · Answer 2 · answered Sep 05 '12 at 19:11

4

I would assume it is caching. You said write cache is disabled, but I see the "Array accelerator cache", I'm not familiar with that - but memory caching would explain bursts of high throughput.

answered Sep 05 '12 at 19:11

Kyle Brandt

82,107
71
302
444

score 2 · Answer 3 · edited Apr 13 '17 at 12:14

If you're benchmarking using realistic activity patterns, and the application performance is acceptable and within the specs of the hardware, then you're in good shape. However, understanding your read/write patterns and the capabilities of your storage system is also important.

You're using an HP Smart Array controller, so there are a number of factors involved in its performance.

1). You have the write cache, which is just the physical disk cache on each drive. Maybe 8-32 megabytes. This is probably disabled in your setup.

2). You also have the battery-backed write cache (BBWC) or flash-backed write cache (FBWC) on the actual controller. This is either 512MB or 1GB, protected by a non-volatile cache mechanism. This appears to be enabled.

3). The cache ratio you described in your question is the percentage of the above dedicated to reads and writes. It's denoted by the "Array Accelerator" terminology.

By having the array accelerator enabled, you're going to have low-latency writes committed to cache before going to disk. Basically, your application can say, "yeah, I wrote that" because the storage system says, "it's written" and can coalesce writes and commit them to rotating disk in sequential batches.

You have a 384MB or 768MB write buffer, based on your current settings, so that accounts for the high IOPS figures during your test. You have a small amount of read cache available as well. If your working set of data is small enough, you could be working entirely in cache, which is far faster than disk.

Here's the output of a Smart Array P410 configuration on a ProLiant DL380 G7. As you can see, there's a lot involved in the basic setup, and there are a few optimizations. I think you may have only disabled one small item, leaving the rest in place.

Smart Array P410i in Slot 0 (Embedded)
   Bus Interface: PCI
   Slot: 0
   Serial Number: 500143801664FE50
   Cache Serial Number: PBCDF0CRHZV1JS
   RAID 6 (ADG) Status: Disabled
   Controller Status: OK
   Hardware Revision: C
   Firmware Version: 5.14
   Rebuild Priority: Medium
   Expand Priority: Medium
   Surface Scan Delay: 15 secs
   Surface Scan Mode: Idle
   Queue Depth: Automatic
   Monitor and Performance Delay: 60  min
   Elevator Sort: Enabled
   Degraded Performance Optimization: Disabled
   Inconsistency Repair Policy: Disabled
   Wait for Cache Room: Disabled
   Surface Analysis Inconsistency Notification: Disabled
   Post Prompt Timeout: 0 secs
   Cache Board Present: True
   Cache Status: OK
   Cache Ratio: 25% Read / 75% Write
   Drive Write Cache: Enabled
   Total Cache Size: 1024 MB
   Total Cache Memory Available: 912 MB
   No-Battery Write Cache: Enabled
   Cache Backup Power Source: Capacitors
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: OK
   SATA NCQ Supported: True

perfmon reporting higher IOPs than possible?

3 Answers3