28

I have two Dell R515 servers running CentOS 6.5, with one of the broadcom NICs in each directly attached to the other. I use the direct link to push backups from the main server in the pair to the secondary every night using rsync over ssh. Monitoring the traffic, I see throughput of ~2MBps, which is much less than I'd expect from a gigabit port. I've set the MTU to 9000 on both sides, but that didn't seem to change anything.

Is there a recommended set of settings and optimizations that would take me to the maximum available throughput? Moreover, since I am using rsync over ssh (or potentially just NFS) to copy millions of files (~6Tb of small files - a huge Zimbra mailstore), the optimizations I am looking for might need to be more specific for my particular use case.

I am using ext4 on both sides, if that matters

Thanks

EDIT: I've used the following rsync options with pretty much similar results:

rsync -rtvu --delete source_folder/ destination_folder/

rsync -avHK --delete --backup --backup-dir=$BACKUPDIR source_folder/ destination_folder/

Currently, I'm looking at the same level of bad performance when using cp to an NFS export, over the same direct cable link.

EDIT2: after finishing the sync, I could run iperf and found performance was around 990Mbits/sec, the slowness was due to the actual dataset in use.

dyasny
  • 18,482
  • 6
  • 48
  • 63
  • 1
    You should add rsync to your tags. Did you check the time for the listing part of rsync ? The low throughput can be due to small files. Can you post your rsync command to check options ? – kranteg Apr 20 '14 at 18:15
  • @kranteg please see edit – dyasny Apr 20 '14 at 18:57
  • 2
    Please verify connectivity with `iperf`. – ewwhite Apr 20 '14 at 20:09
  • yup, iperf shows 991mbits/s, I guess it's te dataset that was so slow – dyasny Apr 21 '14 at 21:08
  • You cannot have good throuphput with rsync and a dataset with small files. You should definitely try tar. – kranteg Apr 22 '14 at 11:47
  • yeah, It definitely looks like tar is the next step, if not LVM snapshots and dd. One question though - I have lots of links and quite a few sparse files. Will tar perserve sparsity? – dyasny Apr 22 '14 at 14:45

3 Answers3

26

The file count and SSH encryption overhead are likely the biggest barriers. You're not going to see wire-speed on a transfer like this.

Options to improve include:

  • Using rsync+SSH with a less costly encryption algorithm (e.g. -e "ssh -c arcfour")
  • Eliminating encryption entirely over the SSH transport with something like HPN-SSH.
  • Block-based transfers. Snapshots, dd, ZFS snapshot send/receive, etc.
  • If this is a one-time or infrequent transfer, using tar, netcat (nc), mbuffer or some combination.
  • Check your CentOS tuned-adm settings.
  • Removing the atime from your filesystem mounts. Examining other filesystem mount options.
  • NIC send/receive buffers.
  • Tuning your rsync command. Would -W, the whole-files option make sense here? Is compression enabled?
  • Optimize your storage subsystem for the type of transfers (SSDs, spindle-count, RAID controller cache.)
Community
  • 1
ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • I've dumped SSH for NFS, seeing pretty much the same results. Block based transfers is what I am planning, switch to LVM snapshot based backups and dd the backups to the second server, where I'll be running ZFS for dedupe. atime is disabled on both sides. No compression is used. How do I optimize the storage subsys for this kind of transfer? The source has two RAID10 over 12x 10k SAS drives, one on the local drives, the other an MD1220. The backup server has the same disk count, but with large SATA drives, and uses RAID5. Full cache H800 and H700 controllers on both sides. 2MBps (from iftop)~ – dyasny Apr 20 '14 at 19:01
  • ~makes me think networking is the bottleneck here nonetheless. – dyasny Apr 20 '14 at 19:01
  • @dyasny Test your network with `iperf` to be sure. – ewwhite Apr 20 '14 at 19:01
  • (and you probably don't want to use ZFS dedupe here. A world of pain awaits, especially for a dataset that size) – ewwhite Apr 20 '14 at 19:02
  • the backup machine has 64Gb of RAM, it should be able to handle the size. What else might be the problem? for ZFS world of pain I mean :) the main host will not be using ZFS of course, plain old ext4 only – dyasny Apr 20 '14 at 19:06
  • Dedupe isn't good for backup datasets. But that's just *my* warning. – ewwhite Apr 20 '14 at 19:11
  • got anything I could read on this one? ZFS or opendedup was the plan, but if it's not the best plan, I'll start looking at something else... – dyasny Apr 20 '14 at 19:12
  • 1
    @dyasny Yes. [**Most of what you need to know about ZFS is here**](http://nex7.blogspot.com/2013/03/readme1st.html) – ewwhite Apr 20 '14 at 20:53
  • 1
    Make sure the destination directory structure was created by `rsync` and not by `cp`. I've seen `rsync` take **much** longer to update a remote directory tree originally created by `cp`: 88GB updated with checksumming in 1h26m instead of 3h! How you create the initial disk layout is critical to getting good update performance. The CPU time is the same; the real time can double. (The same update without checkumming runs in 13 minutes from an SSD to a 200GB Seagate). – Ian D. Allen Apr 21 '14 at 04:04
  • Disabling atime seems controversial. Can you provide any more info on that? I found a lifehacker article about it, but I'm looking for something...meatier. – mlissner Apr 26 '18 at 21:25
  • Most applications don't rely on `atime`, so it's generally safe to mount without it. – ewwhite Apr 26 '18 at 22:28
4

As you probably know copying a lot of little files (eg mailboxes using MailDir format or similar) is definitely not the best option to take advantage of high bandwith interfaces. SSH is probably not the best transport protocol for that either. I would try using tar to create a tarball on the source host prior to send it to you secondary host. 

tar c /var/mail | ssh root@secondary-host 'tar x -C /var/backups'

If you need incremental backup you may want to try the -g options of tar. If you still need to maximize throuput, try using netcat instead of ssh.

alxgomz
  • 1,600
  • 1
  • 10
  • 14
  • I've switched to NFS instead of SSH, to remove the encryption overhead, no joy – dyasny Apr 20 '14 at 19:02
  • Have you tried using tar? May be as a first step try creating a local tarbal on the primary server and then transfer it over the wire. (or test your network with iperf like @ewwhite suggeted) – alxgomz Apr 20 '14 at 19:13
  • I would, if I had local space to spare. This is pretty huge, even with a fully populated DAS box – dyasny Apr 20 '14 at 19:17
  • then try piping it over netcat or ssh (not this is as efficient though) – alxgomz Apr 20 '14 at 19:18
  • I'll be switching to block based backups later on, and I intend to pipe `dd` through `nc` then. but right now, I'm stuck with two huge backups then need to be moved off of the main host, so I can create an LVM system there – dyasny Apr 20 '14 at 19:20
1

Try teasing apart the contributing factors:

  • CPU (e.g. dd of /dev/zero piped through loopback)
  • disk I/O (e.g. dd of a large file piped to cat > /dev/null [piped to prevent short-circuiting])
  • physical network I/O (e.g. dd piped to the other machine)
  • etc.

and testing them independently.

I've had some bad experiences with Broadcom drivers, so my first suggestion is to test the usable network bandwidth with: dd if=/dev/zero bs=1m count=10k | rsh backup_host cat \> /dev/null

Scott Leadley
  • 171
  • 4