Preface; I have done hours of trouble shooting on this issue and simply cannot find out what the sudden "bottleneck" is.
I have (2) 4 SSD Disk Raid 0 Arrays (3.8TB Each).
They are run off a single Intel RS2BL040 which is on a 16x PCI Slot Ported to the Intel RES2SV240 Raid Expander.
From the Raid Expander; each RAID is setup on it's own port using a 4 SATA breakout cable.
OTHER SYSTEM INFORMATION
Windows 7 Ultimate 64 Bit
Dual E5-2697 v2 Xeon CPUs
128 GB PC3-12800 Server RAM
GPU Geforce Titan
OS Drive: 1TB Crucial SSD
PROBLEM
When transfering 2TB of data (a large number of files/folders) from RAID A to RAID B; the copy (I have also tried XCOPY and ROBOCOPY using the /j unbuffered command); - the transfer starts off as expected: About 800MB/s - 1000MB/s. And it holds this speed until about 32 GB has transfered.
At this point; the transfer speed (monitoring with Resource Monitor) drops to a very very static 60 MB/s. There also seems to be a high level of latency on the files it is copying.
I have benchmarked the disks with ATTO; and the disks perform fine, with up to 3GB/s when above the 128kb testmark.
While I understand expecting 1 GB/s sustained is unrealistic for a large number of files (of various sizes - the fact that it starts off fast (even with trying different files - for about 32 GB) then drops to a 60 MB/s Read and 60 MB/s Write; and holds that number is odd.
Could this have something to do with Virtual Memory; pagefile.sys?
I get a sustained transfer speed when transfering from RAID A to a SINGLE SSD running off the onboard SATA controller of 400 MB/s. There is no drop off.
I am trying to transfer ALL of RAID B to RAID A - to re-initialize/format it. Could the fact that RAID A has a stripe size of 16 KB vs RAID B has 128 KB have something to do with it?
Could my settings in the RAID BIOS CONSOLE be incorrect regarding Always Write Back; Read Ahead; Disabling BGI; etc?
I don't understand why I get a solid sustained transfer speed for about 40 seconds at the 800-1000 MB/s range; then it just drops; while copying the same files to a single SSD running off the MOBO can hold 400 MB/s sustained.
Any thoughts/ideas - much appreciated at this point!! Thanks!
