10

For a certain read/write workload, I get 30 MiB/s from a 4-disk array using esata port multiplication. When I move the exact same array to four individual sata connections, I get 90 MiB/s for the same workload. Is this the expected performance impact of the multiplier? If not, how do I narrow the problem down between the software, controller, or enclosure?

I've gotten the exact same results even after trying a large number of variables over the years:

  • Many Linux versions (RHEL5, Fedora 9, 11, 13, 16) and all the kernels that entails.
  • Two types of enclosures (both low-end $100 4-disk models), four enclosures in total.
  • Two types SATA controller chipsets (Marvell 88SE91xx and Silicon Image 3132)
  • Four SATA controllers (Around $30 - $60 pci-x1 interface cards and one PCI, all FIS-based switching).
  • Three SATA speeds (1.5-, 3.0-, and 6.0-Gbps)
  • Four types of hdds (Samsung Spinpoint F1 1TB, WD Caviar Black 2TB, Seagate Barracuda XT 3TB).
  • Three computers (Athlon 3ghz+1GB RAM when I made the first e-sata array, then Core 2 Duo+4GB, now a Core i5 750+8GB).

When I did my first eSATA enclosure all those years ago, I assumed the slow speed was related to using a PCI (33mhz) card, immature SATA drivers, or perhaps the SATA-1.5Gbps 5400 RPM drives. But now with an x1 6Gbps controller, 3.0 kernel, and 6Gbps 7200rpm HDDS, still nothing has changed. I still get the same 30 MiB/s for that particular workload until I move it to individual SATA connections.

Is this a fundamental limitation of all port multiplication implementations?

Daniel Browning
  • 113
  • 1
  • 6

2 Answers2

10

Yes. Poor performance is to be expected.

The ATA protocol was never intended to carry data to more than one drive (it was actually based on the ISA Bus, extending the communication channel to a storage drive). The IDE interface cheated this limitation by carrying data for "Master" and "Slave" devices. SATA has no equivalent mechanism however, as it was intended for only a single device per cable connection.

eSATA Port Multipliers function as an ATA target that can select multiple physical media (so only one physical device can be addressed at a time, and there is a slight latency in switching to a different device). You could compare this to old Optical Libraries.

This is radically different from SAS which was built with Expanders and initiator/target routing built-in (SAS 2.0+ functions somewhat like a switched Ethernet network). SAS is also able to bond multiple lines into trunks allowing simultaneous access to multiple devices across the multiple trunk lines.

Update:
Apparently there are newer controllers that support FIS, which allows multiple drives to be "active" at the same time. This should allow performance much higher than described above. Apparently at this time only one manufacturer makes these chips. I would still highly recommend starting away from SATA in server environments. If you must use SATA drives consider still using SAS controllers and Expanders.

Chris S
  • 77,337
  • 11
  • 120
  • 212
3

Are you sure you tried controller and port multiplier supporting FIS (Frame Information Structure) port multiplying?

AFAIK only Silicon Image controllers and multipliers (SiI3726) support this, but the performance is similar to SAS of same speed (80-90% link saturation).

Hubert Kario
  • 6,351
  • 6
  • 33
  • 65
  • I didn't realize that the port multiplier itself would have to support FIS too, but in retrospect that makes perfect sense. Perhaps the port multiplier embedded in all my dirt-cheap enclosures ($100/each, e.g. Mediasonic HF2-SU3S2) only support command-based switching. Is there any way to tell programatically (e.g. dmesg output)? It's not mentioned in their manual or website except for this excerpt, "When connecting via eSATA interface, customer's computer hardware needs to have Port Multiplier w/ FIS-based switching in order to access multiple HDDs simultaneously." – Daniel Browning Jan 05 '12 at 21:06
  • 1
    @DanielBrowning: there was a flag that showed FIS support for the controller, but I don't think there was one for multiplier. There could be interoperability problems too (SiI refusing to work with JMicron). Getting the chipset used and reading the chipset specs would be the only way to be sure, I think... – Hubert Kario Jan 05 '12 at 21:11
  • I pulled the fan off of the HF2-SU3S2 enclosure and posted the pictures here: http://thebrownings.name/images/2012/2012-01-esata/ -- It looks like a JMicron JMB321 port multiplier, and I don't see anything about FIS on their website: http://www.jmicron.com/Product_JMB321.htm -- what do you think? – Daniel Browning Jan 05 '12 at 22:03
  • This could be the issue. The SiI one costs just as much as your whole enclosure: http://www.amazon.com/5-Port-Port-SATA-II-Multiplier-SiI3726/dp/B004G9SKEE/ I'd definitely try this before going ATAoE or full SCSI (either SAS or iSCSI) – Hubert Kario Jan 05 '12 at 22:13