iostat taking lot of time to return

Question

I have a solaris 10 server with a lot of SAN disks (close to 1000 across all paths). iostat -En takes close to 5 min to return. Is there a way to figure out what is causing the delay? truss hasn't helped much.

Answer 1

dtrace is definitely the tool you are looking for. you can use a relatively simple script to find out which syscall is exactly most time-consuming.

Something like this:

#!/usr/sbin/dtrace -s

syscall:::entry
/execname == "iostat"/
{
    self->ts = vtimestamp;
}

syscall:::return
/self->ts/
{
    @[probefunc] = sum(vtimestamp - self->ts);
    @["- all syscalls -"] = sum(vtimestamp - self->ts);
    self->ts = 0;
}

profile:::tick-1sec
/i++ >= 59/
{
    exit(0);
}

Plus, you can use the DTrace Toolkit, may be you will find something ready to use and more appropriate there.

Once you will determine the stucking syscall, you will be able to do more investigations around it.