I have a process that is run every month or so. Such process identifies a set of files and constructs a proper source and destination path from several fields of different tables in DB to then copy all of these files, so I end up with potentially hundreds of thousands or millions of paths. It is possible that these paths point to a file that has been previously been copied, but has been modified.
How can I perform this copy in the fastest way possible?
I tried at first a simple huge cmd script checking if the file, per resulting path, existed in the destination path and then trying to copy if it didn't exist, but the modified files were skipped. So, then I tried to use "xcopy" to avoid overwrite if date is the same, by using the /D switch and /Y switch to silently confirm the prompt, but in practice, a huge script with hundreds of thousands of xcopy commands takes forever and I don't see any speed-up when some or most of these files already exist with the same date. Do you have any ideas that may help me? What can I do with a file containing hundreds of source and destination paths to make a copy of all of these? Is there a faster alternative to xcopy? or a faster date check and then call any copy command?
A more detailed explanation on how I currently get and copy files based on DB information: There are multiple tables in DB:
Table 1: ID, Dir_Name, ...
Table 2: ID, Dir_Name, ...
...
Table N: ID, Dir_Name, ...
Source Folder Paths Table: Dir_Name, Dir_FullPath
Destination Folder Paths Table: Dir_Name, Dir_FullPath
Then I can create the set of copy commands (or just get the source and destination paths)
Select 'xcopy /d /y '|| T2.Dir_FullPath || '\\' || ID || '.ext ' || T3.Dir_FullPath || '\\' || ID || '.ext*'
from Table1 T1
left join SourceFolderPathsTable T2 on T1.Dir_Name=T2.Dir_Name
left join DestFolderPathsTable T3 T1.Dir_Name=T3.Dir_Name where T1.some_criteria=true;
Thank you!