I've experienced this problem on a number of Dell 11G (R610) and 12G (R620/R720) systems. While I can't be sure what the problem is, it just seems to boil down to a hardware compatibility problem.
In my case, the system will usually boot fine the first time when powering up, but a warm reboot without cycling the power will lead to the error. For a while I was just ensuring the servers were power cycled rather than just restarted, but ultimately it will require some kind of replacement to fix.
One of the errors I received was caused by installing a USB3.0 PCIe card. Moving it to a different slot did not help, but replacing it with a USB3.0 card that used a different chipset solved the problem. In another case, installing an Intel X520 10GbE card from a Dell R610 into an R620 caused the problem, but replacing the Intel card with one from Silicon Image (that used a different Intel chipset) fixed the problem. This is quite surprising as the Dell firmware supports the Intel X520 (and the R620 lifecycle manager even did a firmware update of it for me) yet it still resulted in the PCIe training error.
Ultimately it just seems to come down to this - if you get the PCIe training error, replace the card in question with a different model.