High Disk write not listed under iotop

Question

I have problems with high I/O on my Debian 7.3 server (up to 7 seconds average I/O wait). I tried to find out what's causing this by using iotop -o. Now I see sonething strange: iotop tells me on the top 'Total DISK WRITE: 32.46M/s'. But in the tasklist I can't see any process writing so much on the disk. The highest are two mysql-processes with 90K/s and 8K/s and a kjournald Process. All three are listed with 99,99% IO. # If I add all the writing speed I come nowhere near to 32MB/s.

Screen of iotop:

iotop Screenshot

What's wrong here? I think 32.46MB/s writing is correct, because at this time my webserver has a high page load time, probably caused of high I/O. But why are the 'big task' which use this much capacity, is not listed in the list below?

Edit: I can also sort it by disk write and i also get strange results like this:

enter image description here

You have it sorted by _I/O wait_. Which shows the way to the problems. What does your storage look like? — Michael Hampton, Feb 08 '14 at 19:26
I can sort it by what I want and I also get wrong data, see edit above. I've a 500GB SATA HDD. — Lion, Feb 10 '14 at 14:16

score 8 · Answer 1 · edited Apr 13 '17 at 12:14

if you install newer iotop (like 0.6.x from wheezy-backports), it will show you both "total" and "actual" values. And man page gives the explanation

In addition, the total I/O bandwidth read and written during the sampling period is displayed at the top of the interface. Total DISK READ and Total DISK WRITE values represent total read and write bandwidth between processes and kernel threads on the one side and kernel block device subsystem on the other. While Actual DISK READ and Actual DISK WRITE values represent corresponding bandwidths for actual disk I/O between kernel block device subsystem and underlying hardware (HDD, SSD, etc.). Thus Total and Actual values may not be equal at any given moment of time due to data caching and I/O operations reordering that take place inside Linux kernel.

Not that the answer completely satisfies me, but at least it points to the possible reasons for discrepancy. It might be some kernel thread, like FS metadata/journals, or short-lived processes which started, did IO, and died before iotop seen them them.

There are some related questions here and here. For debugging, /proc/sys/vm/block_dump might be useful to find the source.

High Disk write not listed under iotop

1 Answers1