How can I copy multiple files over scp in one command?

Question

I have to transport a lot of files from one PC to another (both Linux). I would like to use scp for that, but scp only allows for transferring one file at a time.

How can I do this?

I have

• No possibility to use rsync or any other protocol
• No possibility to use passphrase-free certificates (but have a certificate with a passphrase)
• A list of files to transfer and a list with the destination path of the files on the other server
• The files are spread out over a lot of directories, and not all the files in the directories I want to copy

If possible, I would like to gzip and ungzip transparently to save bandwidth!

Is this possible?

Answer 1

Use tar:

tar cvzf - -T list_of_filenames | ssh hostname tar xzf -

Answer 2

I would like to use scp for that, but scp only allows for transferring one file at a time.

I'm pretty sure that isn't true, at least not for the scp command provided by the OpenSSH included with most Linux distributions.

I use scp file1 file2 ... fileN user@host:/destination/directory/ fairly frequently.

For transparent compression, the SSH protocol has this built in and scp can use it if you provide the -C option on the command line. For lots of similar small files you will find the tar+gz option suggested by akira gains better compression as it can make use of similarity between such files where scp compresses each file as a separate entity. I generally prefer to use scp though as it is easier to resume a partial transfer (or, though I know in this situation the questioner doesn't have the option) rsync as that is both even easier to resume and shares the tar+gz option's whole-stream compression advantage.

Answer 3

Doesn't scp -r yourdir otherhost:/otherdir work?

Try this then:

tar cfz - . | ssh otherhost "cd /mydir; tar xvzf -"

the z-flag to tar does compression. Or you can use -C to ssh:

tar cf - . | ssh -C otherhost "cd /mydir; tar xvf -"

Answer 4

On RHEL5, here is the only way possible that I know of to secure copy a list of files that contain special characters (like a space). Make a shell script and include the following:

FILE="/path/filename"
while read line; do
scp "$line" username@servername:/destination/
done < $FILE

Answer 5

It can be done w/o much trouble.

I simply create a list of files with the hostname next to each file and scp seems to parse it fine. ie.

srchost=`hostname`
myfiles="${srchost}:~/.bashrc ${srchost}:~/.bash_profile ${srchost}:~/.vimrc ${srchost}:~/.viminfo ${srchost}:~/.toprc ${srchost}:~/.dir_colors"
scp -v $myfiles $HOME

Scp seems to work fine with this syntax. As I attempted it without the hostname next to each file it failed after the first file. I seem to recall another syntax I used for this before and will post it once I locate it.

Answer 6

I would recommend to parts use a command like this to create an archive from your file list

tar -cvf allfiles.tar -L mylist.txt

Then use scp to download the archive.

scp user@remotehost:~/allfiles.tar

https://stackoverflow.com/questions/8033857/tar-archiving-that-takes-input-from-a-list-of-files