0

I am running 2 haproxy servers that keep state of each other with keepalived to be fault tolerant. These haproxies have many frontends and backends, in order to avoid port exhaustion when forwarding to backends I added the "source" param so it will perform the health check and forwarding to backend with the noted IP address:

server neo0010 10.70.0.100:8443 check check-ssl verify none source 10.200.22.105:1025-65000

Here 10.200.22.105 is one of the dynamic IPs that got assigned to the primary server in keepalived. These check do work, and traffic gets forwarded using that IP address. However, the issue that I have is that the backup keepalived haproxy sees the backend hosts in down state since it doesn't have the IP address locally assigned and it cannot perform the health check to the backends. Is there any way of performing the health checks from one IP and forwarding request via another one? Did anyone have a similar issue and manage to solve it in a different way?

Thank you! Marcelo

MarceloPrego
  • 46
  • 1
  • 3
  • Your HAProxy servers don't have a fixed IP address in your network? – Gerard H. Pille Jul 13 '20 at 14:51
  • They do have fixed IP addresses, but I have more than 40 backends, and in case of using the 1 local ip address to forward traffix it could exhaust my euphemeral ports. – MarceloPrego Jul 13 '20 at 16:22
  • You have at least 28.000 ephemeral ports at your disposition (check /proc/sys/net/ipv4/ip_local_port_range). They are released after each check. What made you worry about these? – Gerard H. Pille Jul 13 '20 at 16:30
  • Im doing TCP load balancing to more than 40 backends, 28,000 is not that many and can easily get exhausted. And I dont believe they get released after each check, this is the address that is used to connect to the backends to forward data, and also many ports stay in TIME_WAIT state for about one min after connecting to the backends. – MarceloPrego Jul 14 '20 at 08:37
  • Well, I disagree on the "easily". You could either reuse the existing connections, or reduce the time wait. – Gerard H. Pille Jul 14 '20 at 08:45
  • I would we thankful if you explain why do you disagree on this? - wont reduce time wait, it is not recommendend – MarceloPrego Jul 14 '20 at 08:47
  • I disagree because it would be a known problem - there are quite a number of HAProxy users. – Gerard H. Pille Jul 14 '20 at 08:48
  • - reduce time wait.. it is not recommendend - cannot use tcp_reuse or recycle, im doing nat, that is also not recommended. and it is a know inssue, please check: https://www.haproxy.com/de/blog/haproxy-high-mysql-request-rate-and-tcp-source-port-exhaustion/ – MarceloPrego Jul 14 '20 at 08:55
  • An article from 2012. What is your HAProxy version? http_reuse parameter? The article ends with "This issue can’t happen on HAProxy in HTTP mode, since it let the server closes the connection before sending a TCP RST". – Gerard H. Pille Jul 14 '20 at 09:00
  • Do you have "NAT" between the HAProxy servers and the backends? – Gerard H. Pille Jul 14 '20 at 09:01
  • it is operating TCP mode. I have NAT to the haproxy servers. – MarceloPrego Jul 14 '20 at 09:08
  • In stead of having the source hardcoded in your configuration, couldn't you manipulate the routing table? – Gerard H. Pille Jul 14 '20 at 09:18
  • all is in the same subnet, you cannot modify that with routing.. – MarceloPrego Jul 14 '20 at 11:00
  • You can't have a specific route for a specific host? – Gerard H. Pille Jul 14 '20 at 11:02
  • You can, but again, these ip addresses are not assigned when keepalived is in standby mode. These interfaces will come up only when it becomes the master. – MarceloPrego Jul 14 '20 at 11:09
  • When that address is assigned, you add it to the routing table to be used for that specific host. – Gerard H. Pille Jul 14 '20 at 11:18
  • when source parameter is added it is actually working as expected. the only issue I have is that it is working as long as it is the master, when it is the backup the ip addresses are not assigned, therefore no health checks can take place. – MarceloPrego Jul 14 '20 at 11:55
  • Do you risk running out of ephemeral ports on a backup haproxy server? Why not use an existing IP address for the health checks, if these are necessary on a backup server. Concerning the port exhausting, a more recent article (https://making.pusher.com/ephemeral-port-exhaustion-and-how-to-avoid-it/) says the number of backencs can alleviate the problem. If you are exhausting the ports to a specific backend, I would check that backend. – Gerard H. Pille Jul 14 '20 at 12:16

0 Answers0