I try to mirror a share mounted via SMB to a local dir. The share is hosted on a Windows 2016 server on which I have no access nor can I install any tools. All I have therefore is an SMB share.
The share contains around 350GB of data, mostly small files (4MB). Every night a rsync
jobs runs and transfers all updates to a local dir. This takes ages to complete. Sending the incremental file list takes around 10 to 15 minutes (I need to verify this) but then rsync needs to transfer a couple of MB (roughly 20 files changed per night) a whopping 12 hours.
Maybe I have a misunderstanding on how rsync
works. But I am under the impression that after receiving the incremental file list rsync
is able to figure out what files need transferring. So all that is left is to pick the changed files and transfer them which should take no longer than a couple of minutes to max one hour.
I was able to bring down the transfer time from 12 to 9 hours by mounting the SMB share with actimeo=600
option. But it still takes much longer than expected.
So this is the mount (on a Synology NAS):
mount -t cifs //remote.server/share /volume1/sysadmin/share -o vers=3.0,username=john,password='***',domain=EXAMPLE,ro,soft,uid=1026,forceuid,gid=100,forcegid,addr=remote.server,file_mode=0777,dir_mode=0777,iocharset=utf8,nocase,cache=none,noserverino,actimeo=600
And this is the rsync job:
rsyncOptions=(
--delete
--human-readable
--itemize-changes
"--modify-window=1"
--partial
--protect-args
--recursive
--stats
--times
--verbose
)
rsyncExclusions=(
--exclude=.DS_Store
--exclude=.Trash
--exclude=.TemporaryItems
--exclude=._*
--exclude=.AppleDB
--exclude=.AppleDouble
--exclude=__MACOSX
--exclude=.AppleDesktop
--exclude=.~lock*
--exclude=.fseventsd
--exclude=.lock
--exclude=.afpDeleted*
--exclude=.smbdelete*
--exclude=.TEMP_*
--exclude=@eaDir
--exclude=Thumbs.db
--exclude='* '
--exclude='*.'
)
/usr/bin/rsync "${rsyncOptions[@]}" "${rsyncExclusions[@]}" "/volume1/sysadmin/share/" "/volume1/share/
Here the stats of the last sync:
Number of files: 571170
Number of files transferred: 279
Total file size: 351.95G bytes
Total transferred file size: 387.34M bytes
Literal data: 387.34M bytes
Matched data: 0 bytes
File list size: 25.95M
File list generation time: 0.526 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 413.57M
Total bytes received: 223.89K
sent 413.57M bytes received 223.89K bytes 13.11K bytes/sec
total size is 351.95G speedup is 850.56
How could I speed up the transfer? Maybe other tools like lftp
are more suited to accomplish this... Any pointers appreciated.
UPDATE 1
lftp
withmirror --dry-run --no-perms --exclude-glob="*." --exclude-glob="* " --parallel --only-newer
option also takes around 9 to 12 hours to complete. So this is no alternative.A simple copy in the GUI or via
cp
of a 4.7MB file takes around 2-3 seconds to complete.I started
rsync
with--dry-run
option. I will update this post as soon as I have the results.
UPDATE 2
rsync
with--dry-run
is running now for more than 6 hours. I guess it is save to assume that it will take as well 9 hours plus. The one and only entry in the log issending incremental file list
. Today is a public holiday here (therefore no changes of files/dirs) so I'm not sure how long it took to complete this part of the process.
UPDATE 3
With more logging enabled the result looks like this:
2019/10/04 04:00:03 [20771] building file list
2019/10/04 05:42:16 [20771] .d..t...... dir1/
2019/10/04 05:42:17 [20771] >f+++++++++ dir1/file1
2019/10/04 05:42:44 [20771] .d..t...... dir1/subdir1/
2019/10/04 05:42:44 [20771] >f.st...... dir1/subdir1/file1
2019/10/04 05:43:19 [20771] >f..t...... dir1/subdir1/file2
2019/10/04 05:43:20 [20771] >f.st...... dir1/subdir1/file3
2019/10/04 05:43:20 [20771] .d..t...... dir1/subdir2/
2019/10/04 05:43:21 [20771] >f+++++++++ dir1/subdir2/file1
2019/10/04 10:16:38 [20771] .d..t...... dir2/subdir1/
...
Note the huge time gap between dir1 and dir2, either rsync
needs a lot of time to traverse the directory tree or I something else is going on.