17

I've been asking me this for a couple of days and after a bunch of searching I wasn't able to come up with a comprehensible answer, not even a theoretical one that makes sense in my head.

I'm playing around with solutions for Mac hosting and I was wondering if I could add thunderbolt ethernet cards to the Macs and bond them in VLANs and therefore semi solve bandwidth bottlenecks to the machines in order to increase access speeds to a DB or external storage.

For example: Plug two ethernet cards into a Mac Mini, bond them and have a VLAN with 2 Gb/s bandwidth.

Braiam
  • 622
  • 4
  • 23
Constantin Jacob
  • 281
  • 2
  • 6

3 Answers3

39

Simply put, no, they are different:

  • with a 10 GbE interface, you get a bandwidth of 10 Gb/s even for a single connection
  • with 10x 1GbE interfaces (and using 802.ad protocol), a single connection/session is limited to 1 Gb/s only. On the other hand, you can serve 10 concurrent session each with a bandwidth of 1 Gb/s

In other words, bonding generally does not increase the speed of a single connection. The only exception is Linux bonding type 0 (balance-rr), which sends packets in a round robin fashion, but it has significant drawbacks and limited scaling. For a practical example, give a look here

Dave M
  • 4,494
  • 21
  • 30
  • 30
shodanshok
  • 44,038
  • 6
  • 98
  • 162
  • But what about link bonding? Wouldn't the 10 physical interfaces be one virtual one to the OS. The overall load/connection throughput would be spread across all physical interfaces, wouldn't it? – Constantin Jacob May 20 '17 at 22:34
  • 16
    No, that's not how bonding works. – EEAA May 20 '17 at 22:45
  • 2
    Great and correct answer, but let's be helpful. While it is theoretically different and OP may not be semantically accurate, _link aggregation_ is still a very suitable solution for the original problem, bandwidth bottlenecks. – Esa Jokinen May 21 '17 at 06:41
  • 4
    @ConstantinJacob Bonding will generally hash some of the fields in the header and use that to choose a link in the bundle. If there is sufficient variation in the fields being hashed, the traffic will be evenly distributed. If the fields being hashed are all the same, only one link in the bundle will be utilized. The fields included in the hash depends on implementation but would usually include fields such as source/destination MAC/IP/port and protocol number. – kasperd May 21 '17 at 08:45
  • 5
    @ConstantinJacob Worded slightly differently. Each TCP connection you run across your bundle will be assigned to a random link in the bundle. If you have 10 TCP connections you may be lucky enough to have them get one link each. But much more likely there will be one or more idle links and some of the TCP connections will be sharing links. This is a case of [the coupon collector's problem](https://en.wikipedia.org/wiki/Coupon_collector%27s_problem) and you should expect to reach about 30 TCP connections before you even start utilizing all the links, and even more before traffic will be even. – kasperd May 21 '17 at 08:51
  • @Constantin Jacob as stated in the answer, bonding generally does not increase the speed of a single connection. The only exception is Linux bonding type 0, which sends packets in a round robing fashion, but it has significant drawbacks and limited scaling. I'll update the answer to document this possibility. – shodanshok May 21 '17 at 09:20
  • 8
    From a business standpoint, there's another major difference. First, one 48-port 10GbE switch still costs less than ten 48-port 1 Gb switches. Connecting and managing 48 cables is dramatically easier than 480 cables. Servers have limited numbers of card slots, so one 10 GbE card can easily carry 20 Gbps, maybe 40 Gbps, but a single 1 Gb card generally can only carry 4 Gbps. If we call all of that something like "bandwidth density", then the increased bandwidth density of 10 GbE has real engineering and cost benefits. – Todd Wilcox May 21 '17 at 13:55
20

10gb/s via x10 1gb/s ports

I am answering only for completeness sake and to save you some headaches. I have over 20k servers doing something similar to this and I can tell you it is a bad idea. This method adds a lot of complexity that will cause operational problems later on. We did this with 4 1gb nics per server. At the time it actually made more sense than going 10gig. At the time, 10gig everywhere would have been many times the cost for very little gain. Our recent itterations of our datacenters no longer do this.

An LACP bond (mode 4) with a single LAG partner will give you 10gb/s, nearly the same as a single 10gb/s port. This can actually be done using more than 1 switch, but they have to support MLAG, otherwise you have to connect only to one switch. If they don't support MLAG, then you only get 1 switch of bandwidth. The other interfaces will be in standby. (so 5gb/s if you have 2 switches).

A single connection will only utilize one link, but you can split up traffic where required at layer-7 if you need to, or you could look into MPTCP, but support for that is new in recent kernels and I am not sure it is ready for prime time. You can split up data sync's using LFTP+SFTP and the mirror sub-system of LFTP. It can even split up one file into multiple streams. There is also bittorrent.

You will not be able to do DHCP on these ports from a client perspective to PXE boot a OS installer, unless you force up eth0 on the server side which technically breaks the LACP monitoring. It can be done, but should not be and it will make troubleshooting problems more difficult if you force an interface up.

In your bonding config, you will have to generate a unique MAC address that is different than all of your physical interfaces, or you will have race conditions due to the way PXE/DHCP work, assuming there is DHCP/PXE in your setup. There are many examples online of how to generate the unique bond0 MAC on the fly.

This also requires configuration on the switch side that aligns with each of your servers bond configuration. You will want to have the LLDP daemon installed to make troubleshooting this less painful, and LLDP enabled on your switches.

If you do this, your cabeling and labeling needs to be flawless. Your switch automation needs to be solid. One cable offset that mixes 2 servers will cause very fun problems.

Kudos to Jay at IBM for making the bonding code as good as he did and for helping us figure out how to get DHCP working in this configuration.

Aaron
  • 2,809
  • 2
  • 11
  • 29
  • 1
    I am somewhat curious as to where you work to have 20k nodes with 4x1 bonded links... that's some nontrivial infrastructure you're describing lol. – Kaithar May 22 '17 at 06:01
  • "You will not be able to do DHCP on these ports" - Is that DHCP _client_ or _server_, or both? – MSalters May 22 '17 at 07:22
  • On the client. If you kickstart your servers, then you would have to force eth0 up and force eth0's switch port up, or you have to use another method to load the OS. – Aaron May 22 '17 at 12:50
  • 1
    @Kaithar I second the non-trivial aspect. I would suggest it was an academic exercise that went too far to accomodate two classes of servers. Uniformity dominated the overall deisgn. Now that everything is going dual 10gb/s we are still doing LACP with one MLAG partner, so 20gb/s total and no more forcing up interfaces. – Aaron May 22 '17 at 14:05
0

It depends. If the primary traffic protocol has inbuilt multipath load balancing, e.g iSCSI, this works very well. Bonding has a host of problems that the other answers already describe.

Secespitus
  • 111
  • 5
camelccc
  • 255
  • 1
  • 15