9

We are trying to design an architecture that will be able to handle more than 64k websockets.

We first tried with Amazon ELB, but its design doesn't allow unexpected spike of traffic nor websocket. (TCP mode timeout the websockets unexpectedly)

With HAProxy, those limits do not apply, but we'll be limited to ~64k websockets maintained between HA and the back-end servers.

Multiple solutions that came to mind :

  • Multiple HAProxy instances, load balance with DNS (Route53 have a weighted option)
  • Two HAProxy instances with Keepalived, multiple internal IP addresses (not sure if it's doable)

Is there a better way to do this ?

masegaloeh
  • 17,978
  • 9
  • 56
  • 104
Bastien974
  • 1,824
  • 12
  • 43
  • 61
  • 1
    Why 64k limit? Is it a source port thing? If that is the case you can just add more 'servers' to the backend that are bound to different ports... – Kyle Brandt Nov 05 '14 at 17:15
  • @Bastien974, the most easy way, is use differents source ip for backends, to scale to 130K connections, I used two ips and tw_reuse sysctl option – c4f4t0r Jan 20 '17 at 14:07

2 Answers2

7

If your 64k limit is due to source ports, you can do something like the following (a little hacky, but it was we currently do at SE for websockets (we have something like .5 million concurrent usually with HAProxy):

server ny-web01-1 10.0.0.1:8081 check
server ny-web01-2 10.0.0.1:8082 check
server ny-web01-3 10.0.0.1:8083 check

Also multiple instances is doable with keepalived. Just do something like round robin DNS over multiple IPs. Just ensure that the IPs always get picked up by active load balancers since DNS itself won't give you the load balancing (there are more options here as well, this one is just simple).

Kyle Brandt
  • 82,107
  • 71
  • 302
  • 444
  • 1
    If I understand correctly, since a TCP connection is defined by srcIP:srcPORT/destIP:destPORT, if I am able to listen in the back-end servers on multiple ports, that would mean between HAProxy and back-end servers I would be able to have multiple connection from the same 127.0.0.1:12345 -> 10.0.0.1:8081, 127.0.0.1:12345 -> 10.0.0.1:8082, etc ? Does this really works ? – Bastien974 Nov 05 '14 at 18:34
  • @Bastien974: You understand correctly - it works. – Kyle Brandt Nov 05 '14 at 18:47
  • @Bastien974: You can use `source 0.0.0.0 usesrc client` in haproxy's backend config for tproxy source transparency. This way srcIP:srcPORT are going to be the actual client IP/ports (not haproxy machine's internal IPs) -- neat for logging too. – wqw Feb 25 '15 at 14:27
0

You could setup multiple HAproxy systems which share same IPs using Anycast and BGP or some other border routing protocol. This way all HAproxy systems are active; if any of those goes down you stop advertising BGP route on that system and it will in ~30 sec stop receiving traffic; which will be re-distributed to other available systems that advertise same range.

For instance check this url on how to set up such layout

Hrvoje Špoljar
  • 5,162
  • 25
  • 42
  • I'm not quite sure this would work inside an AWS VPC infrastructure as I need to use Elastic IP associated to each instance. Your solution would be very close to the DNS one, since Amazon Route53 offers the option to add a health check. My concern is that even with a low TTL, we cannot afford to wait the propagation to other countries (we have client worldwide) to stop sending traffic to a "dead" HA instance. – Bastien974 Nov 05 '14 at 16:37