5
2
My question in short: is there a tool that copies a file from directory A to B while, simultaneously, generating the checksum of the file it has copied/read, without doing an extra read/pass just to generate the said checksum?
I will be copying a few TBs of files from one HDD to another and instead of:
- Copy files from HDD1 -> HDD2 (X hours)
- Generate checksums of files on HDD1 (Y hours)
- Verify checksums of files on HDD2 (~Y hours)
I was thinking of a more streamlined process:
i. Copy files from HDD1 -> HDD2 and generate checksums of the files copied as well (Z hours)
ii. Verify checksums of files on HDD2 (~Y hours)
My assumption is that Z ~= X because the program that can so this will have read the complete file (as it's copying it from one HDD to another) and hence does not need to read the file again just to generate its checksum.
Now I know this idea of mine might not work, if for example, the OS uses DMA to copy the file, and I am not sure what technique Windows 7 uses to copy files from one HDD to another.
Any suggestions to this effect will be appreciated - specially speeding up the copying process and making sure the transfer is 1:1 without corruption or missing files.
I am writing something which does just this. The file, typically, is never in memory, only parts of it... Sticking 1gb into memory is bad as we don't know what system resources are per PC, so we copy a few thousand bytes at a time. The checksum can be done during, before or after for sourcefile thanks to multi threading. For destination, only after copy, but also at same time next file is copied...any way, most modern/decent backup programs do this as standard. any way, what is your question? – Dave – 2012-12-26T14:25:08.410
@Dave: We don't need to read the whole file into memory - blocks of it at a time will do. I have multi GB files that might not fit into memory if multiple files were loaded. I just don't want to read a file twice if it can be done once. Is your tool ready to try today? – PoorLuzer – 2012-12-26T14:28:17.853
no, but things like acronis offer a way to verify files (although at a cost). – Dave – 2012-12-26T14:29:48.363
@DaveRook: What language is your tool in? I would like to work on it if it's in C/C++/Java/PERL,Python – PoorLuzer – 2012-12-26T17:36:15.460
No you wouldn't, I would not give it out :D C# – Dave – 2012-12-26T19:07:00.383