I am attempting to transfer about a million files of various sizes from one server to another, and due to various size, network and time constraints, I'm attempting to use BITS in PowerShell to do the file transfer.
I am not able to compress the directory before sending due to time constraints and limited access to the receiving server.
It appears that there's an (undocumented?) limitation of 60 asynchronous BITS transfers going at once in PowerShell.
After 60 jobs start, I get:
Start-BitsTransfer : Object reference not set to an instance of an object.
At line:1 char:14
+ $jobs = $s | Start-BitsTransfer -TransferType Upload -Asynchronous
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Start-BitsTransfer], NullReferenceException
+ FullyQualifiedErrorId : System.NullReferenceException,Microsoft.BackgroundIntelligentTransfer.Management.NewBits
TransferCommand
The PowerShell script is:
$s = Get-ChildItem -Recurse | Select-Object -Property @{Name = "Source"; Expression = {$_.Fullname}}, @{Name = "Destination"; Expression = {$_.Fullname.Replace("C:\folder\", "\\server\c$\folder\")}}
$jobs = $s | Start-BitsTransfer -TransferType Upload -Asynchronous
I am able to run this synchronously, and I am able to transfer individual files asynchronously without issue.
Is there anyway to do transfer about a million files asychronously with PowerShell as efficiently as possible?
