Can I stop a netcat tar pipe and continue later without overriding existing files?

Question

I'm trying to copy a lot of files from my NAS to my server, about 2TB.

I found that using a netcat tar pipe to copy the directories goes pretty fast, about 40mb/s. It seems to work faster than rsync and ssh, so I would prefer to keep using a netcat tar pipe.

Here's what I did:

On the receiving I do this:

nc -l 7000 | pv | tar -xpf -

And on the sending end I do this:

tar -cf - * | pv | nc otherhost 7000 

Now I would like to stop the copying and continue later. I tried with a small directory to restart the process but it will also send the files that are already there.

Is there a way to prevent it from sending the files that already exist on the server?

Answer 1

Not automatically, but when you resume the transfer later, you can exclude the files that have already been copied by using a tar exclude list. See http://www.cyberciti.biz/faq/exclude-certain-files-when-creating-a-tarball-using-tar-command/

How do I exclude zyz and abc file while using a tar command?

The GNU version of the tar archiving utility has --exclude and -X options. So to exclude abc and xyz file you need to type the command as follows:

$ tar -zcvf /tmp/mybackup.tar.gz --exclude='abc' --exclude='xyz' /home/me

If you have more than 2 files use -X option to specify multiple file names. It reads list of exclude file names from a text file. For example create a file called exclude.txt:

$ vi exclude.txt

Append file names:

abc 
xyz
*.bak

Save and close the file. This lists the file patterns that need to be excluded. Now type the command: $ tar -zcvf /tmp/mybackup.tar.gz -X exclude.txt /home/me

If a file was half-transferred, and you don't want to overwrite it again with tar, you can exclude it, and then later use another tool to resume the copy. I'd recommend using rsync or lftp for that.