5

I've been scratching my head over this for the past 1 week and can't seem to find any solution. So, this my problem... I have a Juniper ex4500 10g switch connected to my servers over sfp+ twinax cables.

1st server (filer): Dell R620

RAM : 96GB
N/W card : Ethernet controller: QLogic Corp. cLOM8214 1/10GbE Controller (rev 58) (2 ports) on bond0 which makes this 20Gbps
OS : centos 6.7 x86_64
Storage : 24 disks connected via perc 810 on RAID 5

2st server (compute): Dell R810

RAM : 256GB
N/W card : Ethernet controller: QLogic Corp. cLOM8214 1/10GbE Controller (rev 58) (2 ports) no bond
OS : centos 6.7 x86_64
Storage : single disk 512GB 6Bb/s

3rd server (compute): Dell R810

RAM : 512GB
N/W card : Ethernet controller: QLogic Corp. cLOM8214 1/10GbE Controller (rev 58) (2 ports) no bond
OS : centos 6.7 x86_64
Storage : single disk 512GB 6Bb/s

I have a CentOS 6.7 ISO as the file that I'm using for testing the transfer speed. And this is how I started my tests. First I started with server 2 and server 3 by using scp centos 6.7.iso root@server3:~/ and the speed it is transferring is maxing out to 167MB/sec. I figured it was because the MTU is 1500 and I'm getting lower speeds. So changed the MTU on the switch to 9214 on the ports the servers are connected to and in the servers ifcfg-px files and rebooted the servers. But now the transfer speed increased only to 200MB/sec. I tested with server 1 which has a bond on it and MTU set to 9000 I still get 200MB/s.

iperf says 10g as the bandwidth but I can't see it. Does scp have so much of an overhead that it is maxing out to 200MB/s?

Anyways, if anyone of you have any idea what the problem is please let me know. it is highly appreciated.

Below are my outputs of iperf and scp.

Client connecting to teng.tofu, TCP port 5001

TCP window size: 92.6 KByte (default)
[ 3] local 10.10.113.15 port 58221 connected with 10.10.113.21 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 11.5 GBytes 9.90 Gbits/sec

SCP

100% 3926MB 186.9MB/s 00:21
D34DM347
  • 1,461
  • 2
  • 19
  • 32
Bhargav
  • 87
  • 6
  • SCP or disk is your issue. Try rsync without the encryption for comparison. – hookenz Dec 20 '15 at 19:51
  • I second Matt - SCP has a large overhead and some internal limitations buffering wise. Try setting up something simple like an NFS share on the server and see how that performs. I would imagine that works just fine and much faster. This is the typical small VMWare setup (with NFS that is) and usually it works very nicely. – MrMajestyk Dec 21 '15 at 08:10

3 Answers3

2

The limitation is most certainly the hard coded buffers in SSH and the additional hard coded buffers in the SCP client. You can get around that with the HPN ssh patch that ewwhite mentioned, but I would first consider other methods rather than modifying your ssh daemon. Or, make a second custom ssh daemon if you must.

If your traffic needs to be encrypted, there are several ways to set up a VPN between the hosts, then you can simply use traditional methods such as native rsync (rsync host::resource/path vs. rsync host:/path) or NFS. Native rsync is easier to secure if you are using an ad-hoc vpn setup such as OpenVPN, Tinc or the native IPSEC built into the kernel.

To see that the network is not likely your issue, create an NFS share, or fire up rsync as a daemon and use the native rsync protocol (not using ssh). To reach 10gb/s you will need to tune the buffer sizes in /etc/sysctl.conf (not there by default) and likely may need to adjust the default options of your gig-e drivers via /etc/modprobe.d/driver-name.conf that you would need to read up on.

Edit

If you are just looking to test your 10Gb cards, then fire up iperf3 as a server on one machine and use iperf3 as a client on the other machine. You should be able to find this in the EPEL repo if using centos/redhat or you can compile it yourself. iperf3 will show you the maximum speed your servers can potentially get not counting application overhead, then it will be up to you to tune sysctl.conf, gig module driver options and such to get higher speeds if you need it.

Aaron
  • 2,809
  • 2
  • 11
  • 29
0

Don't use traditional SSH/rsync for this.

Please try a tool like hpnSSH or something like UDR to run an unencrypted copy.

Also, I can't tell what direction you're copying, but your ingest speed of the client side may not be great. Are you reaching the upper limit of your storage?

(I won't comment on the 24-disk RAID5 setup)

Also see: Improving speed of large file transfer over high latency link

Community
  • 1
ewwhite
  • 194,921
  • 91
  • 434
  • 799
-1

SCP uses SFTS/SSH to encrypt/decript the files. It's overhead is quite high for high-speed networks. You can try to lower the encryption overhead using a weaker cipher, for example arcfour.

Try issuing the following command: scp -o Ciphers=arcfour <local> <remote>

EDIT: if you don't care about SCP/SSH performance, than you should not use SCP to benchmark your LAN. To test pure network speed, you can use netperf or iperf. If you care about NFS, simply export a NFS mount with a big file (eg: an ISO file), transfer it on the other machine and check the seconds required for the transfer.

shodanshok
  • 44,038
  • 6
  • 98
  • 162
  • I tried the above command but im still getting 175MB/s. – Bhargav Dec 20 '15 at 21:22
  • 1
    The limitation is a hard coded buffer size in SSH that is further exacerbated in SCP. Modern CPU's do not have a problem with multi-gigabit speeds with AES256. A different protocol would be required to get around this, or the sometimes flakey hpn ssh patch. – Aaron Dec 21 '15 at 01:44
  • ok. i actually don;t care if scp is not transferring at full bandwidth of a 10G network. I'm doing this only to test the 10G network... I care mostly about my NFS mount points. Is there a way that i can effectively test this? – Bhargav Dec 21 '15 at 05:31
  • I've edited my answer – shodanshok Dec 21 '15 at 08:33
  • updated my answer to reflect an iper3 network test. – Aaron Dec 21 '15 at 21:23
  • iperf shows 9.9 Gb/s. So where is the problem? Your network is working at advertised speed. – shodanshok Dec 21 '15 at 21:24