-1

I've run the following test I've created a folder containing 15'000 files of 400 bytes using this batch :

@ECHO off
SET times=15000

FOR /L %%i IN (1,1,%times%) DO (
    fsutil file createnew filename%%i.txt 400
)

then I copy past it on my Windows Computer using this command : 

robocopy LargeNumberOfFiles\ LargeNumberOfFiles2\

After it has completed I can see that the transfer rate was 915810 Bytes/sec this is less than 1 MB/s. It took me several seconds to copy 7 MBytes Please note that this is very slow.

I've tried the same with a folder with a single file of 50 Mbytes and the transfer rate is 1219512195 Bytes/sec. (yeah GB/s) instantaneous.

Why copying large number of files take so much time - ressources on a windows filesystem ?

Please note that I've tried to do the same on a linux system which runs on the same computer in a virtual machine (vmware player) with ext3 filesystem.

I use the cp command and the copy is instantaneous !

Please also note the following :

  • no antivirus
  • I've tested that behaviour on multiple windows computers (always ntfs) i always get comparable results (transfer rate under 1MB/s avg 7-8 seconds to copy 7 MBytes)
  • I've tested on multiple linux ext3 system the copy is always instantaneous for that amount (15000 files of 400 bytes)
  • The question is about understanding what makes windows filesystem so slow to copy large number of files compared to a linux one for instance.
Arno2501
  • 117
  • 1
  • 6
  • 1
    How exactly are you performing the copies? – Zoredache Dec 11 '12 at 16:33
  • I've tried everything from copy past on the desktop to robocopy. Like I said I really think it has something to do with the way windows handle the files. It's not a problem on my computer all windows box are the same when it comes to deal with a great number of small files. – Arno2501 Dec 12 '12 at 08:10
  • Another possible cause is a virus scanner. Do you use a virus scanner on Windows and no scanner on Linux? Even scanning 15.000 small files may easily take 30 seconds. – Jeff Dec 11 '12 at 18:44
  • I've turned my virus scanner completely off and it doesn't change the speed. I really think it has something to do with the way windows handle the files. Thought someone would know ... – Arno2501 Dec 12 '12 at 08:08
  • It's not just Windows; it's all systems when dealing with a "great number of files" (has nothing to do with the size of the files either; other than the overhead being more apparent when file size is small). – Chris S Dec 12 '12 at 15:08

2 Answers2

4

Your 15,000 files take at least 60,000 HD transactions to complete:

  1. Read the Source file
  2. Allocate the New Destination file descriptor
  3. Write the Destination file
  4. Update the Destination file descriptor

You're probably using a SATA drive too (just guessing) which doesn't support Command Queuing, meaning the drive will have to thrash through all 60,000 operations individually. High end SCSI drives and arrays would increase performance by queuing up several operations in similar locations before executing them.

Your single file copy took at least 4 (though probably a few more, it depends) operations; the same as before. It might be more than 4 if there are file fragments on either the read or write operations.

Throughput is measured not only in bytes per second, but also IOps (essentially operations per second). You are being limited by the latter of these two.

Chris S
  • 77,337
  • 11
  • 120
  • 212
  • It's a great explanation for the slowliness thanks ! But then how do you explain that on the same computer running a linux virtual machine the same operation is instantaneous ??? What makes ext3 more efficient in handling this kind of operation ? – Arno2501 Dec 13 '12 at 07:12
  • My first thought is that the VM has both the VMs internal caching and the host OS's caching (as well as any hardware caching, though the host would take advantage of that as well). You'd have to have a native install or you'll be comparing apples to oranges. – Chris S Dec 13 '12 at 14:12
1

The most likely cause is that your Windows systems are creating thumbnail previews when you copy and paste. Use a CLI tool, like robocopy, instead.

HopelessN00b
  • 53,385
  • 32
  • 133
  • 208