What would cause IO Wait on a SAN?

Question

I understand IO wait when I see it on a server, it means that the CPU is blocked while I wait for the IO to catch up [source].

I am trying to understand why a SAN stats would show a high IO wait - does this indicate that the SAN CPU is blocked by the SAN disk or is it something else?

Where are you seeing IO wait? On the server? Or in the SAN management software? — longneck, Jan 28 '13 at 13:40
`it means that the CPU is blocked` - no it doesn't. It means that there is an IO request outstanding - which might mean that a process is blocked if it explicitly calls sync() or is writing using O_DIRECT. — symcbean, Jan 28 '13 at 14:35
I think I may have answered the wrong question- you're saying your SAN is complaining about wait time? Is it saying that it thinks its hosts are waiting too long to get answers to their IO? — Basil, Jan 28 '13 at 14:36
What kind of SAN is this specifically? Is this an actual array, or a server functioning as a SAN target? — pauska, Jan 28 '13 at 15:03
@Basil Yes - the SAN is complaining. Nothing to do with the hosts. — Robert MacLean, Jan 28 '13 at 16:06
Which array? What management software, and what's the exact message? — Basil, Jan 28 '13 at 16:55

score 8 · Accepted Answer · answered Jan 28 '13 at 13:45

8

A SAN has a much higher IO latency than a local disk due to the fundamental laws of physics. So if your application is doing lots of small writes and fsync() after each, you'll see a lot of iowait.

For example, here are two mysql replicants of the same dataset containing many small transactions, you'll see the slave on the SAN is spending a lot more of its time doing IO.

San: enter image description here

Local:

enter image description here

answered Jan 28 '13 at 13:45

Dennis Kaarsemaker

18,793
2
43
69

Is that Graphite for the graphing? – Tom O'Connor Jan 28 '13 at 14:06
Yeah, graphite with [a plugin I wrote to graph mysql replication load](http://www.kaarsemaker.net/blog/2012/09/27/monitoring-replication-load-graphite/). – Dennis Kaarsemaker Jan 28 '13 at 14:10
What's the effect of SAN controller cache on this? – ewwhite Jan 28 '13 at 14:35
One very unsafe, very bad and very data unfriendly way to make a program stop using `fsync` is to preload libeatmydata http://www.flamingspork.com/projects/libeatmydata/ It will possibly make you lose data (hence the name) but can be useful in some cases. – Jens Timmerman Feb 01 '13 at 11:40

score 5 · Answer 2 · answered Jan 28 '13 at 14:28

SAN wait time could mean that your storage is the bottleneck. It could also be server settings or the connection between your servers and the storage, but much more frequently, when I see wait time for a SAN disk, it's simply a busy SAN.

First, check the performance on the disks backing the volume. You're looking for spikes in IO/s or MB/s reads or writes, and potentially a spike in cache utilization. Try to look only at the hardware involved in the volume you're investigating. Also, look back and forward in time a little to see if there have been higher spikes that didn't cause issues. If so, then the storage hardware is unlikely to have been the problem. Corrective action for hardware bottlenecking on the storage could include migrating this volume to another pool or RAID, or increasing the number of spindles or cache.

Secondly, check the queue depth settings on the server. If you have a very high queue depth, your server will see higher latencies during periods of heavy utilization. Queue depth is a way for the storage to tell the server to throttle their IO allowing the storage to catch up. 32 is a good average number that would be supported by most server OSs and most storage devices I've seen. I've seen higher and lower work as well, but if it's set to 1024 or something, that could explain high wait times. In a situation where the queue depth is very high, the server queues up everything it wants to do, and then the storage does it as fast as it would have if the queue depth were a lot lower. Since the server measures wait time from when something comes into the queue and goes out of the queue, the wait time would go up.

Lastly, check the error logs for the server. Ensure that there's no transfer level issues (like disk timeouts or path failures). If there are, you'd want to look into the switch.

score 1 · Answer 3 · answered Jan 28 '13 at 13:42

1

It's measured no differently than on a server: there are more IO requests coming in than can be dealt with by the hardware resources available.

answered Jan 28 '13 at 13:42

EEAA

108,414
18
172
242

score 1 · Answer 4 · answered Jan 28 '13 at 14:27

High IO wait as reported by the SAN management software either means SAN hardware can't keep up with the demands of your SAN clients. This is either because your hardware just doesn't have the capacity for your load, or it could be something is failing and under-performing.

A slowly failing drive causing poor performance is actually pretty common, especially in RAID5 setups. Pull the SMART logs for all of your drives and I'll bet you find a drive with a very high number of corrected errors. (Correcting those errors takes time. If an individual error is corrected within a certain amount of time, then the RAID controller does not log an error. But stack up a lot of those errors and that adds up to a lot of time. And that's how you get poor performance.)

What would cause IO Wait on a SAN?

4 Answers4