0

I have a process that is run every month or so. Such process identifies a set of files and constructs a proper source and destination path from several fields of different tables in DB to then copy all of these files, so I end up with potentially hundreds of thousands or millions of paths. It is possible that these paths point to a file that has been previously been copied, but has been modified.

How can I perform this copy in the fastest way possible?

I tried at first a simple huge cmd script checking if the file, per resulting path, existed in the destination path and then trying to copy if it didn't exist, but the modified files were skipped. So, then I tried to use "xcopy" to avoid overwrite if date is the same, by using the /D switch and /Y switch to silently confirm the prompt, but in practice, a huge script with hundreds of thousands of xcopy commands takes forever and I don't see any speed-up when some or most of these files already exist with the same date. Do you have any ideas that may help me? What can I do with a file containing hundreds of source and destination paths to make a copy of all of these? Is there a faster alternative to xcopy? or a faster date check and then call any copy command?

A more detailed explanation on how I currently get and copy files based on DB information: There are multiple tables in DB:

Table 1: ID, Dir_Name, ...

Table 2: ID, Dir_Name, ...

...

Table N: ID, Dir_Name, ...

Source Folder Paths Table: Dir_Name, Dir_FullPath

Destination Folder Paths Table: Dir_Name, Dir_FullPath

Then I can create the set of copy commands (or just get the source and destination paths)

Select 'xcopy /d /y '|| T2.Dir_FullPath || '\\' || ID || '.ext ' || T3.Dir_FullPath || '\\' || ID || '.ext*' 
from Table1 T1 
left join SourceFolderPathsTable T2 on T1.Dir_Name=T2.Dir_Name
left join DestFolderPathsTable T3 T1.Dir_Name=T3.Dir_Name where T1.some_criteria=true;

Thank you!

Carlos S. Na
  • 1
  • 1

1 Answers1

0

Take a look at the command line tool "robocopy" that comes with Windows. I believe it may accept a response file, and it already can be told to only update files that have changed dates / times, plus it is about as fast as it can get, being pure command line with no GUI overhead.

tsc_chazz
  • 355
  • 1
  • 7
  • I checked robocopy and found that it doesn't support natively file lists and that it's mostly geared towards copying folder structures. In my case, I can have potentially thousands of folders with possibly thousands of files inside and only need to copy a subset of the files within each folder. So it looks like robocopy is not really the answer I'm looking for. – Carlos S. Na Dec 17 '20 at 20:20
  • That's fair. I'm not sure if the combination of the /JOB and /IF switches between them would do what you want, but it looks like you're doing something more granular than any bulk copying app can do. I would still recommend Robocopy as a straight xcopy replacement, if only because its test for modification seems much faster than xcopy's. – tsc_chazz Dec 17 '20 at 20:47
  • I'd recommend `robocopy`, too. Just use it as xcopy - it's much faster and can do Mutithreading if needed (/MT:n). The only other solution would be to run your process more often. – bjoster Dec 20 '20 at 18:26