7

I normally use rsync's -z option to enable compression when transferring files over the internet. However, if I am on my own (idle) gigabit LAN, do I still want compression? Or will it be faster without it? What factors does this depend on? (I haven't done any benchmarks yet).

So basically, is it faster to: compress + transfer + decompress, or just transfer uncompressed?

If your network connection is slow, it's obviously better to compress... but how about if everything is running on a gigabit ethernet network?

Corey Goldberg
  • 173
  • 1
  • 1
  • 5

5 Answers5

18

During an rsync transfer are you CPU bound, or is your link saturated.

  • If your link is saturated, but your CPU is idle, then compress.
  • If your CPU is maxed, and your link isn't, then do not compress.
Zoredache
  • 128,755
  • 40
  • 271
  • 413
  • i can only access CPU utilization on one end of the rsync, but that should do. i can measure upload traffic from one end, and also see overall traffic from my router. that should get me somewhere. thanks much! – Corey Goldberg Jul 18 '14 at 18:00
  • 5
    Oh, and it is entirely possible to not be either CPU, or NETWORK bound. You could be limited by the performance of your storage subsystem. In that case, compression or not won't really make any difference. The only way to fix that is to buy faster disk. – Zoredache Jul 18 '14 at 18:02
  • 5
    This might be overly obvious but I'll point out that you should consider what type of data you're sending and if it makes sense to compress. Sending a bunch of log files, hell yes, always compress. Sending already compressed file formats like JPEG, MP3, MKV or even ZIP? Compression during transfer is probably useless. – pzkpfw Jul 31 '15 at 20:23
11

I work this quite often, migrating data between servers and during impossibly-long data transfers...

The short answer is to test with your specific data... This is really easy to do. Try a LAN transfer with compression off, then try it with compression on...

In my experience with production data sets across a few environments, on a GigE or greater connection, enabling rsync compression throttled transfers to ~40 Megabytes/second (or 33% of the theoretical maximum speed of a Gigabit link).

That rate varied with the size and number of files being transferred as well (lots of small files, for instance, lowered overall rsync throughput). The storage subsystems of the sender and receivers also mattered.

Recently, I've been using a modified rsync for LAN transfers. By using the UDR library with rsync (allow UDP transfer), and disabling compression and encryption, I can get wire-speed rsync transfers with mixed file types, regardless of the number of files.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • I would assume that this could be improved by reducing the compression level to 1, or perhaps slightly higher. I've yet to find tests which discuss this option. – William T Froggard Jan 14 '18 at 16:51
  • @WilliamTFroggard with --compress-level=1 I'm finding the gigabit connection mostly isn't saturated - getting about 50 megabytes/second, so unless your data is more compressible than mine it'll be best to run without any compression. – JosephH Aug 15 '22 at 19:39
2

Like Zoredache said, if your link isnt saturated no -z. Also, something else you can consider to help manage long transfers is the bwlimit flag.

pokemaster
  • 21
  • 3
  • 1
    According to the manual, `bwlimit` flag `allows you to specify the maximum transfer rate for the data sent over the socket`. Could you explain why this functionality helps manage long transfers? – Samuel Li Sep 11 '18 at 23:33
0

Here is linked quetion "Does compression option -z with rsync speed up backup": https://unix.stackexchange.com/questions/188737/does-compression-option-z-with-rsync-speed-up-backup

Alexred
  • 68
  • 4
0

A few details to complement the answer from Zoredache:

  • Use top on the machines to check CPU usage. Particularly on the sending machine. Use ssh to get at the sender if it is remote.

  • Use rsync with the --progress option if you have enough big files to make it show meaningful speeds. If the files are mostly small, it may not tell you much about the network speed. In that case ...

  • Use nload on one of the machines to see the network speed

And if in doubt, don't compress.

Most big files are usually already compressed (movies, pictures, installers, ...) so trying to compress them more is a waste of time.

Samuel Li
  • 103
  • 2
mivk
  • 3,457
  • 1
  • 34
  • 29