How to to measure and monitor I/O performance

Question

We have a linux cluster with worker nodes sharing a common remote file system. We want to optimize the I/O, in particular when the workers read from the remote fs.

The starting point is to measure the I/O performance. The cluster is used by many user and their jobs are doing intensive reading.

I would like to monitor the performance many times during a day to see how they change (night/day/workdays/before and after hardware upgrade,...).

The question is how to measure the performance. If possible I would like to decouple the network performance from the disk performance. I want to test many different kind of reading, big files, small, sequential, random, access, ... avoid caching, ...

Do you have experience with this kind of test? Do you know some free tools, library, easy to interface with other utilities (visualization, ...)

By now the most smart thing I can do is to use dd

Answer 1

I've used iozone in the past to test disks. It has a lot of options to configure exactly the way your live workload is going to look like. To be any more specific, one would need to know exactly what kind of load you will have. I suggest taking a look at the iozone man page.

The network is roughly the same problem. Different kind of workloads require different tests. Depending on the protocol you may have to use radically different approaches. If you just want to test raw network performance, you can use hping to flood your network with packets.

If you for example would want to test a HTTP server, you could use siege or jMeter to do that.

If you on the other hand wish to monitor the server while in production, you will need to employ some kind of metering. The most popular Linux metering tools are probably munin, Cacti and collectd, but there are a ton of tools out there. Most of them take plugins that are easily written in a shell script or your favorite programming language, but they also contain a number of plugins to monitor IO workload, etc.