How to Find the Bottleneck when Transferring Files over WAN

Question

This question may be a little too broad, but I'm looking for some methodology to identify a transfer speed bottleneck. I'm using rsync to transfer files between 2 NAS devices (DNS-323 to be specific) that are in different locations. I'm tunneling through SSH. Transfer rates are averaging 320-370kB/s according to rsync.

I know there are tons of variables at play here, so that's why I'm wondering what my process should be for determining why the transfer is so slow. Here are some relevant details.

• Internet upload speed for NAS that is uploading is ~3.5Mbps
• Internet download speed for NAS that is downloading is ~20Mbps
• CPU usage (according to top) during transfer for both NASes is ~40-45%
• Memory usage (according to top) for rsync process on both NASes is ~160-190% (not sure what that even means)
• I'm using an AES cipher
• I'm using the OpenSSH HPN patches on both NASes

My initial assumption was that encryption overhead was the bottleneck given the weak CPUs on these devices. But, if that was the case, shouldn't I see higher CPU usage? The ssh process on both machines is only showing ~15% CPU usage.

My next inclination was that HDD I/O transfer speeds are limiting things, but the NASes are able to read/write significantly faster when doing local transfers, so that can't be it.

What else could be causing the slow transfer? Is there some kind of scientific way I can determine what the bottleneck is?

Answer 1

I typed this whole thing up, and then I realized the answer was pretty obvious. Instead of just deleting it, I figured I would share the answer in case anyone else goes down this road.

3.5Mb/s = 448KB/s

So, I'm getting between 70-80% saturation of my up-link. Factoring in traffic from other computers on the network, TCP overhead, packet loss, etc. I would say the transfer speeds I'm seeing are actually pretty decent. If someone disagrees or sees a way to improve performance, I would still love to hear it.