rsync takes a very long time to send file list

14

0

I'm trying to use rsync to backup some files from an internal harddisk to a usb-drive. (I'm working on a wd mybook world NAS)

There are two folders that I want to backup. A music folder and a video folder. The music folder(864 files, round about 10gb) worked perfectly in something about 1 hour (for the complete progress).

But the video folder(1025 files, round about 700gb) is still on "sending incremental file list" for now 8 hours. I don't think anything has been transferred. What could cause such a difference? The video folder hasn't that much more files in it. Yeah it is a lot bigger, but that shouldn't make a difference on generating the file list I think.

I'm using this rsync command:

/shares/extern # rsync -avPc --stats /shares/Public/Shared\ Videos /shares/extern/

I now since i am working on a NAS, I haven't that much cpu power, but it's far too long I think.

What do you think, is this behaviour normal? Are there any suggests what i can do to speed the whole thing a bit up?

I'm using rsync 3.0.2

Graslandpinguin

Posted 2011-08-04T10:48:34.350

Reputation: 257

2Please don't put answers in the question body. You can add an answer of your own if no current answers are correct or comment on other people's answers if you have something to add. Thanks! – Caleb – 2011-08-04T11:35:51.673

Answers

26

The -c option forces rsync to calculate a checksum for all files. This means that it is reading the contents of all the video files before it starts deciding which ones to copy. That is obviously slow.

Consider removing that option if possible. You only need it in rare cases. See the description on the manual page for further details:

Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred, but that automatic after-the-transfer verification has nothing to do with this option’s before-the-transfer "Does this file need to be updated?" check.

snap

Posted 2011-08-04T10:48:34.350

Reputation: 611

I had this issue because I passed '-archive' rather than '--archive' to the rsync command. rsync considered '-archive' to be a variant of -c. – Leons – 2014-12-04T19:51:04.770

4Just to expand @Leons comment, -archive is being seen as -a -r -c -h -i -v -e, all valid switches. Tricky little error. – Mark C – 2015-02-17T13:12:56.803

Wait, what is the difference between checking the checksum versus checking the file size/modify date? Does checking the checksum actually take that much longer than checking just the file size/modify date? Does rsync save checksums after the first run or does it calculate them all from scratch every single run? – Ken – 2017-08-07T17:42:55.520

rsync doesn't save anything between runs. – mlissner – 2018-04-26T21:04:20.323

@Ken: The checksum requires reading every byte of a file's content and calculating the checksum, while the size+modify date just requires reading the file metadata: i.e. a stat() call – Bill Cole – 2019-07-04T00:14:44.560

oh thanks for the hint. after reading the manpage, i can say, that i've missunderstood the -c option. Thought it checks after copying with a checksum. Thanks for the help! – Graslandpinguin – 2011-08-04T11:28:39.450

@CaptnLenz, happy to help. Please consider clicking the tick on the left side of my answer to accept my solution. – snap – 2011-08-04T11:36:01.660

Asked: 2011-08-04T10:48:34.350

Viewed: 21 028 times

Active: 2011-12-15T13:35:44.347