2

I have a mystery regarding pauses after file copies on windows server 2008 (and other releases)

When copying large files, like vhds, to locally attached USB disks I often see a long pause after the copy has completed 100%.

As an example: robocopying vhd files. The bytes read/written count matches the vhd file size and robocopy shows 100% but it pauses for several minutes. If I do nothing it will continue, but I will have to wait for quite some time - about the same amount of time as it took to get to 100%. The bytes read/bytes written counters for robocopy do not change.

My first thought was that the AV had to scan it, but I'm looking at a machine right now which doesn't have an AV installed and this is occurring, so impossible. No other processes are showing read/write byte counts as going up.

The behavior is the same if I use the copy command or xcopy.

I've seen this on other systems but have never worked out what the cause is. Anyone got any suggestions as to what might be going on?

Ian Murphy
  • 1,329
  • 4
  • 19
  • 29

1 Answers1

1

You are experiencing the process of emptying a local buffer to the actual drive. When you copy a file to slow media, your operating system can tell you the operation is done before it actually is finished for performance reasons. It actually keeps track of the operation by itself (in memory), unless the program you are using to track progress has some sort of "wait for the file to be actually written down" call at the end of the process. This is what is causing the wait.

For general Windows file copies, you can change this behaviour to trade off performance for safety by changing the USB drive's settings as described here: http://www.pcworld.com/article/254868/safely_remove_usb_drives_just_by_unplugging_them.html

If you do that, then "remove safely" will become absolutely necessary to avoid filesystem corruption on the external drive. This will shift the (unavoidable) wait to the moment you actually remove the drive, with the added benefit that if you later make further changes to the copied files before removing it, you will save a bunch of expensive I/O since the changes will be performed directly on the local buffer.

GomoX
  • 776
  • 3
  • 8
  • 21
  • It can't be related to buffer size as I'm talking about files of 50-100Gb. The 'pause' after copying one can be several minutes long. Its not possible to have a 50Gb file in buffer as buffer is memory space. The behavior of the OS informing that a copy has completed when it has not is a behavior of windows explorer and doesn't affect command line utilities. The write buffer does come into play, but its tiny in comparison to the file size. – Ian Murphy Nov 07 '13 at 12:21
  • I don't mean the buffer in the USB hard drive. I mean the buffer of pending I/O operations on the computer. There are no limits to its size - the OS might decide to tell you it's done copying before even starting to actually do it. If you then issue a "sync"-like call the operation may block until all I/O is done. It depends on the semantics of the file copy syscall the program is using, not whether it's a GUI tool or not. If you change the drive's settings as indicated in the link above you will see the behavior changes. – GomoX Nov 07 '13 at 18:36
  • I wasn't talking about the disk buffer. The OS buffer is only a few 10's of Mb in size at most. I don't know why you think there is no limit to its size. Buffers are allocated out of the OS memory pool. Its not possible to have 50Gb of I/O ops waiting to be written out. That would exceed the total physical memory on the Server. This has nothing to do with buffers. If it did it would occur with all media and it doesn't. Internal SAS drives do not see this behaviour and they too are buffered. – Ian Murphy Nov 11 '13 at 09:52
  • Did you even try changing the setting I mentioned? Internal SAS drives have a different driver altogether. BTW, read up on transaction logs and DMA, you don't need to store 50 GB of data in memory to buffer an I/O operation of that size. – GomoX Nov 11 '13 at 20:34