Making mbuffer faster

Question

I'm trying to transfer files as fast as possible across a 10 gig network connection. Up until now, I've been using rsync to move large files across the network, but I'm trying mbuffer now because I've heard its faster. The trouble is that I don't get as large an increase in performance as I'm expecting, especially since I'm using a 10 gig network connection between the two computers.

These are my current mbuffer arguments:

mbuffer -m 1G -s 512M -P 10 -i "$currentFile" | ssh $outputTarget "cat - > $outputDir/$filename" > $debugOut

Does anybody have any tips for how I can improve speeds over the network? I'm new to mbuffer and don't really know which arguments would work best for my situation. Right now I'm getting about 110 MiB/s for a 1 GB file I made out of random numbers.

Answer 1

This answer to probably far too late to help the OP, so the following is for posterity.

The bottleneck likely has nothing to do with mbuffer but with the fact that data is being sent through an ssh-encryped connection. In other words, the bottleneck is the speed at which the single-threaded ssh process can encrypt traffic on the sending host and decrypt it on the receiving host.

For your purpose, it's probably better to use mbuffer's -O option on the sender and the -I option on the receiver. That makes mbuffer send the data directly across an unencrypted TCP pipe, which will push your network as hard as the OS, drivers, and hardware will allow.

It's a two-step process because you need to run mbuffer on both the sending host and the receiving host, like this:

1. First, run mbuffer on the receiver. It will listen for a connection on the port given by the -I option (5567 is just an example; choose your own port number):

   ssh $outputTarget mbuffer -I 5567 \>$outputDir/$filename
   

2. Then initiate the transfer on the sender:

   mbuffer -i "$currentFile" -O $outputTarget:5567
   

As outlined above, the receiving mbuffer process running on $outputTarget will listen for and accept a TCP connection from anyone who connects to that socket, not just from the sending mbuffer process that is started in step 2. So you have to be mindful of the fact that mbuffer, when used in this way, is not as secure as ssh, not only because there's no encryption of the data, but also because there's no way to be certain that the intended sender is the only one initiating a connection to the receiver's listen port. Unless someone is port-scanning your receiver or otherwise knows what you are doing and is trying to exploit you, the latter caveat is rarely problematic. The lack of encryption, on the other hand, might very well be a show-stopper depending on the use case.