1
2
I have a folder being synchronised (mirrored) remotely, using the synchronize -mirror -delete remote
command, over FTP.
The folder I am synchronising contains 209,068 files, and 19,079 directories, and thus, even just to check that everything is synced, it takes ages.
The way it seems that WinSCP does it, is it takes 1 file a time, and checks if it is up to date. If it is not, it uploads it.
This process takes forever. 210K files, with a ping of 50ms at least, which means around 3 hours of just comparing files! Not even syncing. (not accounting for actual handling time even)
So I was wondering if perhaps there is a way to make it batch compare (ask for meta-data for like a 100 files at a time), or perhaps create a buffer of a 100 requests a time, that would go much faster (around 100 times faster), which would be 1.8 minutes instead of 3 hours.
If not, perhaps there is a different software I could use that performs better in this kind of task?
Edit: Why I think it does 1 file at a time? When running the sync command, this is the output I am seeing:
E:\sync\a.txt -> \Sync\a.txt
Less than a second after:
E:\sync\b.txt -> \Sync\b.txt
And so on, every file shows, in order of the directory, a part of a second after another
Edit2: Correction
- When syncing a folder, it does go file by file.
- When checking if a folder is synced, it goes folder by folder.
My previous information was incorrect, my apologies.
Keep in mind, you are still limited to FTP connection/command semantics when using WinSCP for FTP access. FTP commands are focused on a single file at a time. The only way to avoid that would be to host the files on another kind of server (perhaps SFTP which is not related to FTP) that WinSCP can work with. – Frank Thomas – 2017-02-06T20:59:16.007
@FrankThomas Ok, good to know, but can't it open several connections in parallel? About SFTP - I have no control over the device's firmware, as it is a WDMyCloud box, and thus I can not use SFTP – Amit – 2017-02-06T21:02:52.747
@FrankThomas Correction: I do have a way to use SFTP. Will it help for sure, or was it just a direction to try? – Amit – 2017-02-06T21:09:32.260
"takes 1 file a time" - Why do you think so? WinSCP should get the file data from directory listing. So there should be one request per whole directory, not per file. Show us a sample log, if you think otherwise. – Martin Prikryl – 2017-02-06T22:40:24.360
FTP uses command semantics like GET and PUT which operate on a single file. When WinSCP uses FTP as its target protocol, it strings together many FTP commands to do its job. The only exception is the
MGET
command which works well for files within a given directory that match a pattern, but a synch task would not be able to operate in this fashion. – Frank Thomas – 2017-02-06T23:30:29.897@MartinPrikryl I added an edit explaining my rational. – Amit – 2017-02-07T07:39:19.723
@FrankThomas Thanks for the explanation. So why does it do 1 file at a time? Back to my previous question - will setting up SFTP definitely help? – Amit – 2017-02-07T07:40:40.033
Again, log file please. + SFTP would be no different (of course unless there's some specific problem, what log file would help us identify). – Martin Prikryl – 2017-02-07T07:46:18.247