Batch file to copy files&folders structure one by one, verify the result and save the report to txt

Question

I have over 1 million (small size) files(<500kB jpg) in a structure like this:

H:\main_folder\folder\sub_folder\sub_sub_folder\sub_sub_sub_folder

Every sub_sub_sub_folder have 10-15 files, sub_sub_folder could be a date/time stamp, sub_folder is a machine_name and folder is another incremented number.

I need to have a copy of main_folder with the same structure but I need to verify that my copy is 100% the same as the source. Windows Explorer will block after ~1000 copies, doesn't even show the properties (size, size on disk, contains).

I need a batch that will go to H:\main_folder\folder\sub_folder\sub_sub_folder\sub_sub_sub_folderA read&copy all the 14 files attributes and paste them in Z:\main_folder\folder\sub_folder\sub_sub_folder\sub_sub_sub_folderA, open a file log.txt, write:

"file1.jpg 490kB copied from H:\main_folder\folder\sub_folder\sub_sub_folder\sub_sub_sub_folderA to Z:\main_folder\folder\sub_folder\sub_sub_folder\sub_sub_sub_folderA successfully ..."

and

"file15.jpg 470kB copied from H:\main_folder\folder\sub_folder\sub_sub_folder\sub_sub_sub_folderA to Z:\main_folder\folder\sub_folder\sub_sub_folder\sub_sub_sub_folderA successfully"

e.t.c.

mark the end of the log(one line of ====== or something) and only then proceed to the next operation(copy content and folder structure of sub_sub_sub_folderB). If copy operation not successfully terminated stop, create a new log error_sub_sub_sub_folderB.txt (the next log will be error_sub_sub_sub_folderZ.txt) and only after the log is created move to the next sub_sub_sub_folder

In a few words: a step by step logging copy batch that doesn't kill the system or the machine hardware resources.

Answer 1

Both xcopy and robocopy will do the job, but robocopy's output is closer to what you described.

xcopy src dst /i /e /f /v /c > log.txt

/F will output the full source and destination paths.

/V is claimed to verify the file sizes, but doesn't print them in the output.

Apart from not having the file sizes logged, this mostly achieves what you described. There are other switches that can control whether ACLs and attributes are copied. Being a native exe, it will be faster than a script loop. But you won't get a separate error log, and you can't abort a subfolder and continue with next subfolder on error (/C continues all files on error).

Sample output:

C:\Users\Celery\test>xcopy src dst /i /e /f /v /c
C:\Users\Celery\test\src\a\blah.txt -> C:\Users\Celery\test\dst\a\blah.txt
C:\Users\Celery\test\src\a\blah2.txt -> C:\Users\Celery\test\dst\a\blah2.txt
C:\Users\Celery\test\src\b\blah.txt -> C:\Users\Celery\test\dst\b\blah.txt
C:\Users\Celery\test\src\b\blah2.txt -> C:\Users\Celery\test\dst\b\blah2.txt
4 File(s)

Robocopy produces a fairly detailed log, including file sizes. Try:

robocopy src dst /e /np /log:log.txt [/tee]

Sample output:

C:\Users\Celery\test>robocopy src dst /e

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows

-------------------------------------------------------------------------------

  Started : Thursday, 22 January 2015 10:58:55 PM
   Source : C:\Users\Celery\test\src\
     Dest : C:\Users\Celery\test\dst\

    Files : *.*

  Options : *.* /S /E /DCOPY:DA /COPY:DAT /R:1000000 /W:30

------------------------------------------------------------------------------

          New Dir          0    C:\Users\Celery\test\src\
          New Dir          2    C:\Users\Celery\test\src\a\
            New File                  17        blah.txt
            New File                  17        blah2.txt
          New Dir          2    C:\Users\Celery\test\src\b\
            New File                  17        blah.txt
            New File                  17        blah2.txt

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         3         3         0         0         0         0
   Files :         4         4         0         0         0         0
   Bytes :        68        68         0         0         0         0
   Times :   0:00:00   0:00:00                       0:00:00   0:00:00


   Speed :                2193 Bytes/sec.
   Speed :               0.125 MegaBytes/min.
   Ended : Thursday, 22 January 2015 10:58:55 PM

Otherwise, to script the iteration yourself, you would need four nested loops, like:

for /d %%a in (main_folder\*) do (
  for /d %%b in (%%a\*) do (
    for /d %%c in (%%b\*) do (
      for /d %%d in (%%c\*) do (
        :: code to copy files and write logs goes here
      )
      echo ================= end of %%d =============== >> log.txt
    )
  )
)

That will only copy files in the sub_sub_sub_folders: any files in higher level folders will be ignored, because for /d only iterates over directories.