Benchmarking hosting providers IO with Bonnie

Question

Ok, because of a bunch of projects I'm working on I've access to dedicated Servers on a 3 hosting providers.

As an experiment and for educational purposes I decided to see if I could benchmark how good the IO is with each.

Bit of research lead me to Bonnie++

So I installed it on the server and ran this simple command

/usr/sbin/bonnie -d /tmp/foo 

The 3 machines in different hosting providers are all dedicated machines, one is a VPS, other two are on some cloud platform e.g. VMWare / Xen using some kind of clustered SAN for storage

This might be a naive thing to do but here are the results I found.

    HOST A

    Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                        -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
    Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
    xxxxxxxxxxxxxxxx 1G 45081  88 56244  14 19167   4 20965  40 67110   6  67.2   0
                        ------Sequential Create------ --------Random Create--------
                        -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
                  files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                     16 15264  28 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
    xxxxxxxx,1G,45081,88,56244,14,19167,4,20965,40,67110,6,67.2,0,16,15264,28,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++



    HOST B

    Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                        -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
    Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
    xxxxxxxxxxxx     4G 43070  91 64510  15 19092   0 29276  47 39169   0 448.2   0
                        ------Sequential Create------ --------Random Create--------
                        -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
                  files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                     16 24799  52 +++++ +++ +++++ +++ 25443  54 +++++ +++ +++++ +++
    xxxxxxx,4G,43070,91,64510,15,19092,0,29276,47,39169,0,448.2,0,16,24799,52,+++++,+++,+++++,+++,25443,54,+++++,+++,+++++,+++



    HOST C

    Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                        -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
    Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
    xxxxxxxxxxxxx 1536M 15598  22 85698  13 258969  20 16194  22 723655  21 +++++ +++
                        ------Sequential Create------ --------Random Create--------
                        -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
                  files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                     16 14142  22 +++++ +++ 18621  22 13544  22 +++++ +++ 17363  21
    xxxxxxxx,1536M,15598,22,85698,13,258969,20,16194,22,723655,21,+++++,+++,16,14142,22,+++++,+++,18621,22,13544,22,+++++,+++,17363,21

Ok, so first off what is the best way to read the figures and are there any issues with really comparing these numbers?

Is this in any way a true representation of IO Speed?

If not is there any way for me to test that?

Note: these 3 machines are using either Ubuntu or Debian (I presume that doesn't really matter)

Answer 1

These sites can help you with interpreting bonnie results:

http://www.textuality.com/bonnie/advice.html

http://www.issociate.de/board/post/478953/Understanding_bonnie++_results.html

http://sourceforge.net/projects/witme/files/bonnie-to-chart/

First though - I'd like to address some inconsistency here:

You've done three different test sizes and haven't shown any other system parameters so your results are kind of hard to evaluate. (What are the CPUs here? What sort of disk sub system? Why did you run three different sizes? Why did you use different versions of bonnie? What filesystems were you running on? Were you doing any filesystem mount option improvements?)

Knowing what specs matter to you depends on your application need: Video streaming needs fast read (bonnie input) performance. Video recording needs fast write (bonnie output) performance. etc.

Here are some bonnie tips/tricks that I usually use:

• Reduce your system RAM as low as you can go You can pass kernel parameters at boot to do this. mem=512MB is what I normally use. This makes sure your local OS cache effects have minimal impact on your IO testing.

• Use a decent test size to stress IO I find 5-20G to be good test ranges. Make sure your results are similar for a variety or ranges and then use the same size for all your testing.

• Don't bother with per-char tests.
  They don't reflect real world disk usage and take time to run. (just all about disk i/o will happen with blocks, not character)

• If you're running on a SAN, consider zeroing our your block layer before running tests. Sometimes there's a first-write penalty when allocating space. If you dd out your entire drive before running tests, you can know for certain that you're not hitting this. (running several iterations of a test on the same node and comparing results can also help identify if this is a problem)

• Always post your bonnie command line to help others duplicate your tests.

EC2 Hint: Several people have found running software RAID0 stripes on AWS EBS to improve IO performance.