Fastest way to mirror directory over network

Question

I am looking to speed up my deployment process, currently I have a single staging server, and a web-farm in production.

I currently use robocopy to mirror each web-farm server from the staging server, however, this is very slow.

My copy command looks as follows

ROBOCOPY %codeSourceDir% %targetSite% /MIR /XF *.config

Which is to mirror the folders, excluding any *.config files.

This behaves properly, but to copy 200 mb of files to 8 servers, it takes roughly 4 minutes per server (~30 minutes total). If I do a simple delete folder and copy using remote-desktop, it finishes in about 5 seconds! The reason why I think the difference is so large is there are a lot of files (~10 thousand), and most of the time spent by robocopy is the comparison of files.

Is there a way to speed up robocopy, by ignoring the file-compare feature?

What I'm looking to do is to mirror directories (basically, delete the extra files), I do not care if I overwrite files that are the same. Can robocopy do this, or is there a better tool for the job?

score 2 · Answer 1 · answered Aug 12 '14 at 15:57

2

The /is flag will overwrite files. It is 'include same files', and should accomplish your task. I would also use /W:1 to set the retry wait time to 1 second, rather than the default 30. Your delay might be from locking on your target side; have you checked that out?

I don't know of a way to copy without checking, but that should get you where you need to be.

Of course, you could also use another line of robocopy to just delete all the files from the target directory BEFORE running the mirror. That would certainly work as well.

answered Aug 12 '14 at 15:57

jski

911
1
7
20

I'll check out these two flags, and I'll try your other suggestion if it doesn't improve. – Matthew Aug 12 '14 at 16:02
You might also want to increase the threads used by using the /MT:n switch. – Philosophene Aug 13 '14 at 16:05

score 0 · Answer 2 · answered Aug 12 '14 at 16:05

0

I don't think I understand what you mean by "simple delete folder and copy using remote-desktop, it finishes in about 5 seconds". You mean you actually delete a single folder from the staging server, then copy the whole hierarchy over to the web server over RDP and the whole thing takes 5 seconds? Are you telling it to skip all files it already finds in the remote server? That would not update the content of those files, if they have changed.

Back to Robocopy, you can limit the set of files with "/xo" (exclude older files). Also, consider using multi-threading with "/mt:X" where "X" is the number of threads to run in parallel.

You can see all Robocopy's options here.

answered Aug 12 '14 at 16:05

Giovanni Tirloni

5,693
3
24
49

I'm using `robocopy` to copy the entire folder structure, if I delete the destination folder before copying through windows, it ends up copying really quick. – Matthew Aug 12 '14 at 17:29
Got it. One hypothesis is that the amount of I/O operations and the network latency are mostly what is accounting for the delay while comparing files. To test this, try Robocopy from/to a local directory, the time difference will be all the I/O flowing through SMB. If that's so, then looking for ways to optimize SMB operations would be the next step. I'm not an expert in this area but SMB 3.0 is supposed to have enhancements specifically for that (http://technet.microsoft.com/en-us/library/hh831795.aspx) – Giovanni Tirloni Aug 12 '14 at 18:19

score 0 · Answer 3 · answered Aug 12 '14 at 16:15

0

You should have a look into DFS - Distributed File System (Microsoft) that does pretty much what you want, transparently.

And it does even more

answered Aug 12 '14 at 16:15

Phil

228
2
7

Summarize the content on the linked page. Simply linking to the documentation does not make for a good answer as the page may fail to load in the future or may have different content. – 89c3b1b8-b1ae-11e6-b842-48d705 Aug 12 '14 at 17:28

Fastest way to mirror directory over network

3 Answers3