5

I'm connecting a RHEL5 box to a Dell EqualLogic PS5000X and seeing a maximum of 1 Gbps even while using MPIO with three gigabit NICs.

I have all three interfaces on the PS5000X and three dedicated iSCSI interfaces on the server connected to the same switch. When using more than one interface to connect to the SAN I only get 1/n traffic on each. I followed the Dell setup guide and everything seems to be working perfectly aside from my throughput numbers.

[root@testbox ~]# multipath -ll
linuxvol (36090a02840c3bd2f5ec324010000707f) dm-5 EQLOGIC,100E-00
[size=50G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=3][enabled]
 \_ 6:0:0:0  sdf 8:80  [active][ready]
 \_ 14:0:0:0 sde 8:64  [active][ready]
 \_ 15:0:0:0 sdi 8:128 [active][ready]


[root@testbox ~]# dd if=/dev/zero of=/mnt/linuxvol/testfile obs=64k count=16M
16777216+0 records in
131072+0 records out
8589934592 bytes (8.6 GB) copied, 76.4894 seconds, 112 MB/s

I can even shut down iSCSI sessions mid-transfer and watch the interface utilization on the remaining connections jump from 33% to 50% to 100% as I go from three to two to a single session. I've tried tweaking the various queue depth and round robin settings which only make a small difference.

The PS5000X is set up with 14 data plus 2 hot spare 10k RPM SAS drives. My test volume is set to RAID-10 with multiple initiators enabled. No thin provisioning, no snapshots, no other traffic on the SAN.

What am I missing? Is there some MPIO setting I need to adjust on the OS?

Update: The switch is an HP ProCurve 4104GL. It doesn't support jumbo frames. Flow control is enabled on all of the SAN ports. Spanning tree is disabled. The only devices connected to the switch are the PS5000, the test server, and one uplink port. I know traffic isn't going through the uplink because I can watch port utilization on the switch while running the test. I'm also watching the network interface graph on the PS5000 and it lines up perfectly with the switch utilization, around 33% throughput on each interface.

Austin Lundeby
  • 71
  • 5
  • When you look at the interface utilization are you talking about the interfaces on the Equallogic Array or the interfaces on the Server? Out of curiosity what type of switch are you using and how have you isolated the iSCSI traffic from everything else? – Helvick Jul 22 '10 at 20:10

3 Answers3

2

It turns out that the issue was related to the switch. When we finally replaced the HP with a new Dell switch we were able to get multi-gigabit speeds as soon as everything was plugged in.

Austin Lundeby
  • 71
  • 5
1

Just food for thought... we had Server 2003 boxes with software multipath IO on an HP ProCurve switch to Dell PS5000 SANs working fine... I will say that we had to fight with a lot of settings on both the Dell EqualLogic side and the software iSCSI side in Windows to get everything working smoothly however.

On Windows a crucial step in getting things working was switching the load balancing from "least queue" to "round robin" (this is from memory so cut me some slack if I'm not remembering the terms exactly right). Once we did that, we could see bandwidth to the EqualLogic banalcning across our three dedicated iSCSI NICs.

Make sure your iSCSI working scenario survives across reboots too! We had to tweak the order in which services start on Windows (by changing service dependencies) to get iSCSI and MPIO working "just right" after a reboot

Eric Gearhart
  • 191
  • 2
0

What are your network cards? Integrated or Cards? What is the bus limitations of their connectivity? So basically are the network interfaces built into the motherboard or are they a PCI card of some form? What is the PCI bus, PCI-X or what? What is the bus speed. The system can't push more than the speed of the interface system bus

John
  • 567
  • 2
  • 4
  • One Intel Gigabit ET dual port card (PCI Express x8 slot) and one onboard Broadcom NC373i. I know the onboard card isn't optimal, but I wanted to have 3 NICs on the server as well as the array. – Austin Lundeby Jul 23 '10 at 04:31
  • 1
    Long night yesterday. Anyways. MPIO isn't going to split the write operation from a single sequential operation. But if you submit 3 different dd commands you'll see all three interfaces in use. According to the multipath.conf you can set the rr_min_io parameter to adjust the amount of data that will be sent from an interface at a time, but it's still round-robin. If you want to be able to split the data from the write operation you may need to look at interface bonding at the network level and the switch must support this as well. – John Jul 24 '10 at 02:52
  • 1
    I also get similar throughput numbers when running iozone with 8 threads. Either way I can see the traffic spread across all three interfaces. When I crank up the rr_min_io setting to 500+ I can actually see the writes as they move between each interface but it still never pushes past 1/3 of max with all three connected. – Austin Lundeby Jul 24 '10 at 16:39