I have some Seagate Nytro SSDs (model xs800le70004), SuperMicro SAS3 JBODs (2C models with redundancy), LSI 9300 SAS3 Cards, and SAS HD Cables (SFF-8644).
When the SSDs are attached with both primary and redundant links, the performance drops to 72% of the performance when only attached via the primary. Even when talking directly to the drive on the primary link, if the redundant link is attached, the performance suffers. This performance degradation is across basically all types of IO, read,write,r&w, 4KiB-512KiB IO Sizes, QD1-QD16, and different thread counts ("jobs" in fio).
If for instance, a server is connected with a single primary link, but another server is connected with the redundant link, the performance suffers to the server with a single primary link.
My expectation was that the drive would need to be accessed via multipathing to achieve the drive's full performance, but not that the performance for the drive would degrade for any use case less than completely saturating the drive via both links.
So, my question is, is this normal? Should I expect that this performance penalty is a typical tradeoff for the redundancy?