Read IO Wait Increase

Question

While troubleshooting a performance issue on a Debian Linux box running ext3 on software RAID 1 I redirected the output of a process writing useless temporary log files to /dev/null (ln -s /dev/null/ foo) which has reduced write operations by around 3MB/s. However the change seems to have also lead to a big increase in Write and Overall IOwait times which I'm at a loss to explain and would really appreciate some input on...

iostat: http://oi45.tinypic.com/5l2mx0.jpg

write latency (change applied at ~3am): http://oi46.tinypic.com/33be1kw.jpg

Update 18.04.13:

Write IOwait remains higher than before but I've applied the following all of which have been at least somewhat beneficial...

Remounted the filesystem with noatime option.
Changed to deadline ioscheduler.
Made vm.swappiness 30 (host is home to ~1GB of database, a CMS and streaming audio).
Turned readahead down to 256 down from 4096 for sda and sdb.
Turned readahead up to 4096 from 256 for /dev/md2 (the main data partition).

Update 19.04.13:

Converted the main filesystem from ext3 to ext4

«I redirected the output … which has reduced read operations» — output reduced read? O_o — poige, Apr 17 '13 at 03:36
Could you post an iostat before and after you route output to /dev/null? — Gazzonyx, Apr 17 '13 at 05:24
Run `pidstat -d 1 5` and either share the output or see what processes are causing the writes. — Matthew Ife, Apr 19 '13 at 08:29
If that 3MB/s was a significant portion of all write operations, you may be seeing the difference between latency of sequential writes going to the writeback cache (for the log file) and random writes with proper cache flushes (which your database hopefully performs). — Sergey Vlasov, Apr 20 '13 at 17:48

score 0 · Answer 1 · answered Apr 20 '13 at 16:58

Logically, the only thing that can increase response time (I/O wait) is increasing the number of operations that have to be carried out on the "same" thing.

Hypothesis: the I/O to /dev/null causes more operations at the driver level than the (possibly more buffered) I/O to a disk

Test: try switching it back to a disk file. If response time falls, there is something to do with /dev/null that takes longer than it should. If response time stays the same, then something happened at the same time as the change that affected response time.

--dave

Read IO Wait Increase

1 Answers1