Best way to transfer files over a LAN between two Linux computers

Question

I want to transfer files (a music folder) between two Linux computers. After searching for the best way to do this, I've seen that there are lots of ways of doing this. I know this has been asked a lot, everywhere and all the time. The main problem with this is that there is no clear, recent consensus on one best way to do this task in 2011 for Linux beginners (even depending on some parameters).

So in the spirit of the Stack Exchange websites, I want this not to be related to my particular situation, but more of a guide to others as well on how to transfer files between two Linux computers over a local network. I think a wiki would be useful for many.

Here's what I found so far:

• ssh
• sshfs
• scp
• sftp
• nfs
• samba
• giver

What is the easiest? Most flexible? Simplest? Best solution? What are the pros and cons of each? Are there other (better) options? What are the parameters in choosing the best method (solution might depend on number of files, filesize, easiness vs. flexibility, ...)?

Answer 1

In a Linux environment, for both security and ease of use, ssh is the best way to go. SSH, SSHFS, SCP, and SFTP as you list are all just different services built on top of the SSH protocol. SCP is very easy to use, it works just like CP but you can provide user and machine names in the path. So, we might do a CP like cp ~/music/ ~/newmusic/, but we could just as easily do scp ~/music/ user@host:~/newmusic to send it to the computer named host. That's it - we don't need to set anything up. You'll be prompted for the account password on the other machine if you don't have certificate or some other authentication set up (scp shares those settings with ssh, of course).

SFTP is a tool that makes it easy to do a lot of operations on a remote file system - it works just like FTP, but it runs through SSH so it's secure and requires only an SSH server. man sftp will tell you all about how to use it. I don't use SFTP just to move a folder between two machines, it's more useful when you have a lot of operations to do, like if you're rearranging files on another computer.

SSHFS just extends SFTP in to a file system: it allows you to mount a virtual host in to your file system, so the network stuff happens totally transparently. SSHFS is for semi-permanent setups, not just a one-time file transfer. It takes some more effort to get set up, which you can read about on the project website.

If you need to work in a mixed-OS environment, Samba becomes your next best bet. Windows and OS X support Samba completely automatically, and Linux does as well although it's sometimes rough to use.

Answer 2

My personal favorite for cases where security doesn't matter is netcat + tar:

To send a directory, cd to inside the directory whose contents you want to send on the computer doing the sending and do:

tar -cz . | nc -q 10 -l -p 45454

On the computer receiving the contents, cd to where you want the contents to appear and do:

nc -w 10 $REMOTE_HOST 45454 | tar -xz

Replace $REMOTE_HOST with ip / hostname of computer doing the sending. You can also use a different port instead of 45454.

What's actually happening here is that the 'receiving' computer is connecting to the sending computer on port 45454 and receiving the tar'd and gzip'd contents of the directory, and is passing that directly to tar (and gzip) to extract it into the current directory.

Quick example (using localhost as a remote host)

Computer 1

caspar@jumpy:~/nctest/a/mydir$ ls
file_a.txt  file_b.log
caspar@jumpy:~/nctest/a/mydir$ tar -cz . | nc -q 10 -l -p 45454

Computer 2

caspar@jumpy:~/nctest/b$ ls
caspar@jumpy:~/nctest/b$ nc -w 10 localhost 45454 | tar -xz
caspar@jumpy:~/nctest/b$ ls
file_a.txt  file_b.log

Answer 3

For one time moves, scp is recommended.

But if you find that this dir may work and you need to move it many times to keep the other position updated then you can use rsync (with ssh).

Since rsync has a lot of arguments I usually put it in a small shell so I get it right (every time). The idea is to only send things that has changed since the last time it ran.

#!/bin/bash

user="nisse"
host="192.168.0.33"

echo "Sync: /home/media/music/"
rsync --archive --delete -v --progress -e "ssh -l $user " /home/media/music/ $host:/home/media/music/

This will move a dir called "/home/media/music/" from the local computer to the the pc called 192.168.0.33, using user "nisse". And delete anything on the target that doesn't exist on the local pc.

Answer 4

I would recommend you trying alternatives instead going straight with SSH for moving files inside your own LAN as the overhead is IMMENSE. I would go with Caspar's solution if this one for whatever reason won't work for you:

At the source:

$ python3 -m http.server {PICK_YOUR_PORT}

On destination:

$ wget -r {ip / hostname}:{port}/{File / Directory}

This won't only be lighter than using SSH but far faster with speeds ranging 45~65MiB on standard CAT6 UTP.
If you really want to squeeze the most out of the connection try replacing wget with lftp and using pget -n20 and mirror -r commands.

Answer 5

The fastest is probably netcat (as caspar described).

I like the combination of tar & ssh, which is secure and still fast:

On the source

tar -cf - . | ( ssh user@target && cd /target/path && tar -xf - )

Doing that as root, it preserves file permissions. Or use -p on both sides. Also -S might be considered if you have sparse files.

It's possible to reduce the encryption overhead of ssh if you use arcfour as cipher which works with openSSH:

tar -cpSf - . | ( ssh -c arcfour user@targethost && cd /target/path && tar -xpSf - )

To update the remote path, rsync is perfect:

rsync -av --sparse --delete -e "ssh -c arcfour" . root@targethost:/target/path

Answer 6

If it absolutely has to be done over the LAN, I'd use rsync, as it will pick up where it left off if it gets interrupted. It also has a few other tricks for minimizing the amount of data that gets transferred, although I doubt that many/any of them would be relevant to the case of copying a music library to a virgin location. If security is a concern, just set RSYNC_RSH=ssh first and the data will be tunneled over ssh.

If I were actually doing it, though, I probably wouldn't use the LAN at all. I'd copy the files onto, then off of, a USB hard drive. In my experience, this can easily be multiple orders of magnitude faster than going over the LAN, despite having to copy the files twice - USB 2.0 is rated for 480Mbps, which is faster than anything short of gigabit ethernet, plus it's less sensitive to conditions which will degrade the performance of a LAN. It's also completely OS-independent, provided you use a file system that all involved machines can handle - I'd recommend VFAT/FAT32, since that's pretty much universal.

Answer 7

I would suggest rsync as it will copy files incrementally. You can set it up to copy either only modified or new files only once you have done the initial update. You can use ssh as a transport layer if you wish.

Answer 8

I use Unison, which is an awesome file synchronizer over many different protocols. You can configure it to use scp, rcp, ftp or even locally on the filesystem between two folders. I use it to synchronize my music library, as it can transfer multiple files at once over the network and is really tuneable in its configuration. I keep my music collection backed up and in sync over 2-3 computers. It will only copy changed files, and does so by keeping an index on both ends of the transfer so as to be able to tell when a client has changed the file or when the server file has changed.

Your mileage may vary, but it's certainly a lot better than scping your entire music collection every time you add a new song :)

Answer 9

I followed ssh process for passwordless login first http://www.tecmint.com/ssh-passwordless-login-using-ssh-keygen-in-5-easy-steps/

For scripts and text files the following works for me just fine

To transfer data from local host to remote host. cat localfile | ssh <user>@<ip> "cat > <path>/<remotefile>"

To transfer data from remote host to local host. ssh <user>@<ip> "cat > <path>/<remotefile>" | cat > localfile

This works for me to transfer files on embedded systems that don't have ssh client or scp built in.

No scp - only ssh.