5

Hoping someone can point me in the right direction with some iSCSI performance issues I'm having. I'm running Openfiler 2.99 on an older ProLiant DL360 G5. Dual Xeon processor, 6GB ECC RAM, Intel Gigabit Server NIC, SAS controller with and 3 10K SAS drives in a RAID 5. When I run a simple write test from the box directly the performance is very good:

[root@localhost ~]# dd if=/dev/zero of=tmpfile bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 4.64468 s, 226 MB/s

So I created a LUN, attached it to another box I have running ESXi 5.1 (Core i7 2600k, 16GB RAM, Intel Gigabit Server NIC) and created a new datastore. Once I created the datastore I was able to create and start a VM running CentOS with 2GB of RAM and 16GB of disk space. The OS installed fine and I'm able to use it but when I ran the same test inside the VM I get dramatically different results:

[root@localhost ~]# dd if=/dev/zero of=tmpfile bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 26.8786 s, 39.0 MB/s
[root@localhost ~]#

Both servers have brand new Intel Server NIC's and I have Jumbo Frames enabled on the switch, the openfiler box as well as the VMKernel adapter on the ESXi box. I can confirm this is set up properly by using the vmkping command from the ESXi host:

~ # vmkping 10.0.0.1 -s 9000
PING 10.0.0.1 (10.0.0.1): 9000 data bytes
9008 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=0.533 ms
9008 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.736 ms
9008 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.570 ms

The only thing I haven't tried as far as networking goes is bonding two interfaces together. I'm open to trying that down the road but for now I am trying to keep things simple.

I know this is a pretty modest setup and I'm not expecting top notch performance but I would like to see 90-100MB/s. Any ideas?

ewwhite
  • 194,921
  • 91
  • 434
  • 799
Justin
  • 303
  • 3
  • 10
  • Only use storage controllers with write-back caching enabled, otherwise the numerous synchronous write requests performed by ESXi will kill your storage performance: http://communities.vmware.com/thread/208767 -- you ***have*** a BBWC installed, don't you? http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01832427&lang=en&cc=us&taskId=101&prodSeriesId=3884082&prodTypeId=15351 – the-wabbit Sep 22 '12 at 19:30
  • To rule out the network, could you please run iperf between the guest and the storage node? If all is fine, could you please try NFS in async mode and see if you get better results? – 3molo Sep 22 '12 at 20:13
  • I do have a 256MB BBWC module attached to the storage controller and it is enabled. I'm trying to compile iperf on openfiler but it doesn't look like there is a suitable C compiler installed and of course, no package manager to get one. Any ideas? – Justin Sep 22 '12 at 21:42
  • [root@localhost iperf-2.0.5]# ./configure ... checking for g++... no checking for c++... no checking for gpp... no checking for aCC... no checking for CC... no checking for cxx... no checking for cc++... no checking for cl.exe... no checking for FCC... no checking for KCC... no checking for RCC... no checking for xlC_r... no checking for xlC... no checking whether the C++ compiler works... no configure: error: in `/root/iperf-2.0.5': configure: error: C++ compiler cannot create executables – Justin Sep 22 '12 at 21:43
  • @Justin Could you mark the correct answer? – ewwhite Oct 28 '12 at 14:13

1 Answers1

6

You didn't provide the version of CentOS installed as your VMWare guest... but let's assume it's CentOS 6.x.

The default CFQ I/O scheduler is a poor choice for guest virtual machines (and most systems). You'll want to modify it. I used to just recommend setting the scheduler to deadline, but there's an easier method now for RHEL/CentOS systems...

The best way to handle that and a couple of other tunables is to download the tuned framework and apply a better profile to the system.

In your case:

yum install tuned tuned-utils

Once installed, try one of the appropriate profiles. For VM's, I either do enterprise-storage or virtual-guest. Test and see which works best for you. A reboot is NOT required between profile changes.

tuned-adm profile virtual-guest

The reasoning:

Edit:

Well, that's an improvement. Make sure you have the VMWare tools installed in your guest. Other items include filesystem tuning (e.g. noatime) and such... but really, those results should not matter beyond a certain point. A dd test is not representative of realistic usage patterns.

One of the other benchmarking tools like bonnie++ or iozone would be more helpful. Grab them via yum.

As a virtual machine, you need to be concerned with small random reads and writes. That will matter far more than sequential write metrics from your dd test.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • 1
    Applying the virtual-guest profile did make a noticeable difference, I am now getting 62.1MB/s on the dd test. The enterprise-storage profile made it worse so I'll stick with virtual-guest. Anything else I can do on the guest side to improve performance? – Justin Sep 23 '12 at 13:50
  • @Justin Well, that's a noticeable improvement. See my edit above. – ewwhite Sep 23 '12 at 14:11