12

How do I resume a copy of a large file in linux? I have a huge file (serveral gigabyes) partially copied to a network drive, and it took a long time, and it was mostly done before the copy operation stopped due to a network problem that is now fixed. How do I resume the file copy. I don't want an inefficient script, and ecp didn't work (it doesn't seem to work for large files).

5 Answers5

5

The command you want is going to be something like

rsync -v --append /path/to/afile /mnt/server/dest/afile

unless you can access the server over ssh and run rsync that way, in that case the command FlyingFish gave is best.

Justin
  • 3,776
  • 15
  • 20
5

If you know you simply need to append to the local file and do not want to use rsync (which could potentially take a long time calculating checksums), you can use curl. For example, if you have a large file on a slow removable USB stick mounted at /media/CORSAIR/somefile.dat and only half of it is in the current directory, to resume:

curl -C - -O "file:///media/CORSAIR/somefile.dat"

user48985
  • 51
  • 1
  • 1
  • Thanks for the tip, I had to use this tip because I was copying from a Windows SMB server (so no rsync) when the slow connection was lost, and I didn't want to download the 500MB from the beginning. – MiKy Nov 25 '12 at 23:00
4

I would try rsync -a /from/file /dest/file.

Gonzalo
  • 467
  • 5
  • 8
  • 1
    and while we're talking about rsync, note that you can copy directly to any remote machine [supporting SSH] using rsync -az -e ssh /from/file user@dest.ation:/full/dest/path – Mikeage Nov 25 '09 at 04:26
  • 3
    wow. running rsync like that without any options will delete /dest/file and retransmit the entire thing. How did this get voted up? – Justin Feb 23 '10 at 12:22
  • 2
    How does preserving file attributes (-a) help? Should this be --append? – Malvineous Oct 06 '12 at 04:53
2

Yes rsync is the way to go. For me, we've transferred 100+GB data over rsync+ssh. If you're looking for a true backup copy, make sure that you use the -a (archive) option to preserve file attributes (times, owners, perms, etc.)

host1> rsync -aP file user@host2:/path/to/new/dir/

It's also useful for copying large files that may change during the course of a migration. You can pre-load the data to the destination and once ready for final copy, do it again but only for a fraction of the time. You can save on actual downtime by using rsync to it's full potential.

P.S. Using v (verbose) can slow down a transfer of many files.

FlyingFish
  • 41
  • 1
0

rsync is only good if you have rsync on the destination server. In that case, it's indeed the best solution.

But not otherwise. Since rsync's point is to only copy changed parts in big files, it assumes these changed parts can be anywhere in the file. This means it will checksum all blocks which already have been copied. If you don't have rsync on the remote end, your local rsync will thus begin by reading back everything that has already been transferred.

If your source machine has a web or ftp server, you can use wget from the destination server with the "--continue" option. (or curl with the "--continue-at [-|size]" option).

If your destination machine has an FTP server, you can use curl on the source machine with the --append option.

As a last resort, you can use dd with the "bs=" (block size), "skip=" and "seek=" arguments. For example:

Let's assume you will be able to use 2048 byte blocks. If your destination file is currently 2'048'000'000 bytes (2 GB), that is 1'000'000 blocks of 2048 bytes. To append the rest of your source file to the destination, you can

dd if=source_file of=destination_file bs=2048 skip=1000000 seek=1000000

You may be able to use a bigger block size to improve transfer speed. Just remember to specify the block size with bs= and that the value given to skip and seek is in blocks, not in bytes.