250

As much as I have read about iowait, it is still mystery to me.

I know it's the time spent by the CPU waiting for a IO operations to complete, but what kind of IO operations precisely? What I am also not sure, is why it so important? Can't the CPU just do something else while the IO operation completes, and then get back to processing data?

Also what are the right tools to diagnose what process(es) did exactly wait for IO.

And what are the ways to minimize IO wait time?

BenMorel
  • 4,215
  • 10
  • 53
  • 81
Peter Krumins
  • 3,435
  • 4
  • 21
  • 18

7 Answers7

124

I know it's the time spent by the CPU waiting for a IO operations to complete, but what kind of IO operations precisely? What I am also not sure, is why it so important? Can't the CPU just do something else while the IO operation completes, and then get back to processing data?

Yes, the operating system will schedule other processes to run while one is blocked on IO. However inside that process, unless it's using asynchronous IO, it will not progress until whatever IO operation is complete.

Also what are the right tools to diagnose what process(es) did exactly wait for IO.

Some tools you might find useful

  • iostat, to monitor the service times of your disks
  • iotop (if your kernel supports it), to monitor the breakdown of IO requests per process
  • strace, to look at the actual operations issued by a process

And what are the ways to minimize IO wait time?

  • ensure you have free physical memory so the OS can cache disk blocks in memory
  • keep your filesystem disk usage below 80% to avoid excessive fragmentation
  • tune your filesystem
  • use a battery backed array controller
  • choose good buffer sizes when performing io operations
pQd
  • 29,561
  • 5
  • 64
  • 106
Dave Cheney
  • 18,307
  • 7
  • 48
  • 56
  • 12
    Don't forget "make sure your backend storage is fast enough to keep up with your I/O load." – jgoldschrafe Jan 25 '12 at 20:56
  • 2
    @Dave Cheney, And when my process is idle that is because it is waiting on IO. So what is the difference between IOWait and idle? – ctrl-alt-delor Jul 10 '12 at 09:07
  • 5
    When in IOwait, the process is in "uninterruptible sleep" meaning it can't be killed, to avoid the risks of corrupting data on disks. a normal idle process is really doing nothing, so there is less risks killing it. – mveroone Jan 02 '14 at 08:18
  • 2
    On top practically it means your IO is too slow. "Make the server faster" is different whether you are CPU limited or your CPU is starving because someone decided the slow notebook disc is enough to run a database server and the IO load makes the CPU only use 2% of what it can, waiting like crazy for the IO to finish. – TomTom Nov 07 '14 at 09:40
  • 5
    OMG I cannot believe the top 3 answers here are so **wrong**. The answer below by haridsv is correct. No cpu is "waiting" for the io to complete. Yes, some io can be blocking - often there is a good reason for this, and under some circumstances all io can be blocking. But you can also see iowait occurring for completely asynchronous operations. – symcbean Sep 17 '16 at 22:59
  • @symcbean I really don't see anyone having claimed that any CPU hardware unit is actually waiting in a blocked state. Only that the CPU might be idle because if there is currently no work ready to be executed, then it is in fact "waiting" for new work to become ready for execution. – Oskar Berggren Jan 08 '22 at 14:31
62

Old question, recently bumped, but felt the existing answers were insufficient.

IOWait definition & properties

IOWait (usually labeled %wa in top) is a sub-category of idle (%idle is usually expressed as all idle except defined subcategories), meaning the CPU is not doing anything. Therefore, as long as there is another process that the CPU could be processing, it will do so. Additionally, idle, user, system, iowait, etc are a measurement with respect to the CPU. In other words, you can think of iowait as the idle caused by waiting for io.

Precisely, iowait is time spent receiving and handling hardware interrupts as a percentage of processor ticks. Software interrupts usually are labled separately as %si.

Importance & Potential misconception

IOWait is important because it often is a key metric to know if you're bottlenecked on IO. But absense of iowait does not necessarily mean your application is not bottlenecked on IO. Consider two applications running on a system. If program 1 is heavily io bottlenecked and program 2 is a heavy CPU user, the %user + %system of CPU may still be something like ~100% and correspondingly, iowait would show 0. But that's just because program 2 is intensive and relatively appear to say nothing about program 1 because all this is from the CPU's point of view.

Tools to Detect IOWait

See posts by Dave Cheney and Xerxes

But also a simple top will show in %wa.

Reducing IOWait

Also, as we are now almost entering 2013, in addition to what others said, the option of simply awesome IO storage devices are affordable, namely SSDs. SSDs are awesome!!!

Grumpy
  • 2,939
  • 17
  • 23
45

I found the explanation and examples from this link very useful: What exactly is "iowait"?. BTW, for the sake of completeness, the I/O here refers to disk I/O, but could also include I/O on a network mounted disk (such as nfs), as explained in this other post.

I will quote a few important sections (in case the link goes dead), some of those would be repetitions of what others have said already, but to me at least these were clearer:

To summarize it in one sentence, 'iowait' is the percentage of time the CPU is idle AND there is at least one I/O in progress.

Each CPU can be in one of four states: user, sys, idle, iowait.

I was wondering what happens when system has other processes ready to run while one process is waiting for I/O. The below explains it:

If the CPU is idle, the kernel then determines if there is at least one I/O currently in progress to either a local disk or a remotely mounted disk (NFS) which had been initiated from that CPU. If there is, then the 'iowait' counter is incremented by one. If there is no I/O in progress that was initiated from that CPU, the 'idle' counter is incremented by one.

And here is an example:

Let's say that there are two programs running on a CPU. One is a 'dd' program reading from the disk. The other is a program that does no I/O but is spending 100% of its time doing computational work. Now assume that there is a problem with the I/O subsystem and that physical I/Os are taking over a second to complete. Whenever the 'dd' program is asleep while waiting for its I/Os to complete, the other program is able to run on that CPU. When the clock interrupt occurs, there will always be a program running in either user mode or system mode. Therefore, the %idle and %iowait values will be 0. Even though iowait is 0 now, that does not mean there is NOT a I/O problem because there obviously is one if physical I/Os are taking over a second to complete.

The full text is worth reading. Here is a mirror of this page, in case it goes down.

Community
  • 1
haridsv
  • 131
  • 2
  • 5
  • Will the CPU that is waiting for is also responsible for the I/O interrupt once the I/O is complete by the driver. – Srikan Sep 06 '20 at 20:11
  • @Srikan I think CPU is only involved in the handling not in the generation of interrupts. – haridsv Sep 08 '20 at 08:18
39

iowait

iowait is time that the processor/processors are waiting (i.e. is in an idle state and does nothing), during which there in fact was outstanding disk I/O requests.

This usually means that the block devices (i.e. physical disks, not memory) is too slow, or simply saturated.

You should hence note that if you see a high load average on your system, and on inspection notice that most of this is actually due to I/O wait, it does not necessarily mean that your system is in trouble - and this occurs when your machine simply has nothing to do, other than than I/O-bound processes (i.e. processes that do more I/O than anything else (non-I/O-bound system calls)). That should also be apparent from the fact that anything you do on the system is still very responsive.

tools

  • sar (from the sysstat package, available on most *nix machines)
  • iostat
  • sarface (a front-end to sar)
Xerxes
  • 4,133
  • 3
  • 26
  • 33
  • 8
    Note that strictly speaking, that definition of I/O wait time is only valid on single processor systems. It needs to be somewhat refined for multi-processor systems: http://veithen.blogspot.be/2013/11/iowait-linux.html – Andreas Veithen Nov 18 '13 at 22:40
  • 1
    To what extent `iowait` affects `Load Average`? Say, 100 threads waiting for network, will LA be 100? – Ivan Balashov Feb 20 '17 at 12:01
1

For Solaris, I use DTrace to look at what the processes are doing if I need to see what I/O operations are running. For Linux, there's a similar program called systemtap which provides a similar level of exposure to the kernel and process calls.

One example I used when learning DTrace was to compare a cp command to a dd command. You can see that dd does a lot more reads for the write, while cp does not, mostly because of the buffer size dd uses by default (if I'm remembering correctly).

Milner
  • 935
  • 7
  • 17
0

What kind of IO operations will depend on your applications and setup.

It is important as in some cases the CPU can't get the data or instructions that it needs to continue. In some cases it can continue, but it will depend on what apps are running as to what it can do. If you have a single threaded application which does lots of disk access then you will need to wait.

To minimise the IO time, buy more and faster memory, get faster disks, defrag the disks you have.

If it is an in house application which is the bottleneck see if it can be optimised to read in bigger blocks or to do IO asynchronously.

Jeremy French
  • 665
  • 3
  • 12
  • 25
0

using ps aux can print process STAT
if stat is D or Ds, the process is in Uninterruptible sleep (usually IO)
when a process enter Uninterruptible sleep, nr_iowait of runqueue is added, and if nr_iowait > 0, the idle time of cpu is counted to iowait

vmstat also shows how many process blocks
r: The number of processes waiting for run time.
b: The number of processes in uninterruptible sleep.

http://bencane.com/2012/08/06/troubleshooting-high-io-wait-in-linux/

Singo
  • 101
  • 2