very specific iostat output needed in realtime

Question

Is there anyway to get the output of iostat -m and add only the column await from the extended output in realtime? I know this question sounds nutso but I need to be able to get all those column in realtime without outputing to a file or running iostat twice since that would give misleading results. This is centos 6.2 iostat 9.0.4. kernel 2.6.36.4. So to summarize I need this.

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               0.56         0.00         0.00        324        485

and I need to add await to it so it looks like this.

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn   await
sda               0.56         0.00         0.00        324        485   10.72

Am I dreaming? I'm I looking at this the wrong way? Do I need to compile my own iostat?

Thanks in advance.

ewwhite · Accepted Answer · 2012-09-10T04:56:16.390

I'm not sure how to get the iostat formatting you're looking for. The tool I use for this type of monitoring, though, is customizable to the point where you could display the relevant fields. It's also excellent for generating data for replay and graphing purposes.

I use Collectl monitoring (available in CentOS via yum), and the disk check flags should give you what you need:

collectl -sD --dskfilt sda

Sample output (you can change the units and columns printed):

collectl -sD --dskfilt sdb
waiting for 1 second sample...

# DISK STATISTICS (/sec)
#          <---------reads---------><---------writes---------><--------averages--------> Pct
#Name       KBytes Merged  IOs Size  KBytes Merged  IOs Size  RWSize  QLen  Wait SvcTim Util
sdb              0      0    0    0     336      0   52    6       6     0     0      0    1
sdb             16      0    1   16     389      1   52    7       7     0     0      0    3
sdb              0      0    0    0    1236      9  109   11      11     0     0      0    0
sdb             16      0    1   16     676      3  140    5       4     0     1      1   14
sdb              0      0    0    0      64      1   16    4       4     0     2      1    2

oh boy. this might work for me. I need to see if the developers can work with this and if I can get the green light to install the pkg. Thanks. I'll update. — egorgry, Aug 23 '12 at 14:39

gWaldo · Answer 2 · 2012-08-23T14:47:17.020

1

This seems like something that should be monitorable with munin, graphite, etc.

EDIT: If your system has watch, you can set the iostat to automatically run a given (or infinitely-reapeating) number of times, with a specified gap between runs! awk out the fields that you actually want!

edited Aug 23 '12 at 14:47

answered Aug 23 '12 at 14:33

gWaldo

11,887
8
41
68

This is for a custom embedded solution that I can't use munin or any other tools outside of the standard systat utilities. My hands are kinda tied here. – egorgry Aug 23 '12 at 14:37
is plain-old RRD/RRDTool included in your system? – gWaldo Aug 23 '12 at 14:39
Unfortunately, no. It's pretty minimal. – egorgry Aug 23 '12 at 14:42
LOL... Sorry, then; I've got nothing. – gWaldo Aug 23 '12 at 14:45
Oh, I lied! See edit. – gWaldo Aug 23 '12 at 14:47
'watch' command is pretty basic but it wont fit here because it will run iostat once each few seconds and as we know, you have to have more than one samples to gain representative data – wojciechz Aug 23 '12 at 15:06
`watch` is pretty configurable, including the interval and duration – gWaldo Aug 23 '12 at 15:39

wojciechz · Answer 3 · 2012-08-23T14:41:19.887

1

Why not creating a wrapper in bash ?

One-liner version would look like :

iostat 1 2 | grep -w sda | tail -1 | awk {'print $7'}

This will run iostat twice, and return the right value of await (which is 7th column)

You can do the same with vmstat by passing the right column to awk to get whole CPU iowait. For example :

vmstat 1 2 | tail -1 | awk {'print $16'}

Note that iostat and vmstat is different on each linux distribution and you have to pick the right column.

edited Aug 23 '12 at 14:41

answered Aug 23 '12 at 14:39

wojciechz

538
3
11

I've thought of that and I'm not sure the results are accurate enough since there will be a small delay between runs. – egorgry Aug 23 '12 at 14:41
It depends on your needs but 2s delay would be quite realtime for such parameter. – wojciechz Aug 23 '12 at 14:45
Can you define what you mean by "realtime"? Are you looking for per second stats or something else? – Matt Aug 23 '12 at 15:26

score 1 · Answer 4 · answered Aug 24 '12 at 12:46

1

rereading the original question if the author is looking for real-time info, I agree collectl is the way to go. that's why I wrote it. ;)

if there is some specific thing you're trying to do with the data perhaps I could help but I'd need to know more. as for how real-time you want to be, you can choose any sub-second monitoring interval you like.

-mark

answered Aug 24 '12 at 12:46

Mark J Seger

161
1
6

Collectl has been very well received by the developers and it looks like we are waiting on the final approval to install the pkg. I see no reason it won't get approved. Even if it isn't I'm really impressed and happy to have it in my toolbox for the future. I work for a hardware company and we are graphing a lot of data for an SSD array with very high performance. We will likely retool everything around collectl including infiniband and ethernet. Thanks for such a great tool. – egorgry Aug 24 '12 at 13:38
glad to hear you're enjoying it. have you discovered colmux yet? basically it talks to multiple collectl's across a cluster and provides output like top, but unlike top which only shows top processes, with colmux you can show top-anything! for example, with colmux I was able to find a few dozen misconfigured servers which were using 10GB+ of system buffers out of several thousands in just a few minutes. – Mark J Seger Aug 24 '12 at 14:27

very specific iostat output needed in realtime

4 Answers4

Linked