1

I know some people are using keepalived and heartbeat for active/standby but what action is taken if the haproxy process were to die?

What would be nice is if the virtual IP would switch servers if the haproxy process were to die and/or a networking issue were to occur. We are currently investigating heartbeat and corosync with pacemaker. Can anyone explain their solution to this issue in depth?

UPDATE: Thanks Kyle, see answer and links therein.

CarpeNoctem
  • 2,397
  • 4
  • 23
  • 32

2 Answers2

4

I would say that heartbeat is what you're looking for.

If the monitored service (haproxy in your case) has an lsb-compliant init-script - heartbeat will run initscript status. If it says that the service is down, it will attempt to start it. If it fail at starting it a couple of times - it will perform a fail over to the other node. As long as the nodes have a way to communicate with each other, this will be performed in a very controlled manner - the addresses are removed on one node, and brought up on the other.

If two heartbeat-machines lose communication with each other, they may both attempt to perform failover. One way to solve this, is by configuring a STONITH-plugin (Shoot The Other Node In The Head). This will use a management interface and attempt to turn off the other service before starting its services. Some mechanism like this is crucial if you involve shared storage in fail over.

Personally, I have never experienced haproxy die - I consider it a very stable service. I use heartbeat for failover of IP-addresses only if running on the haproxy-nodes.

Kvisle
  • 4,113
  • 23
  • 25
  • 1
    HAProxy serves all of Stack Exchange's sites, never crashed. We also only use heartbeat for the IP failover. Monitoring the process would introduce more complexity. which isn't a bad thing, but you have to follow through with that and make sure you test the all of the additional scenarios. Re-arping isn't always smooth, so I opt to lean on the side of avoiding the IP failover situation. – Kyle Brandt Oct 21 '11 at 21:35
  • @KyleBrandt: what was your reason for choosing heartbeat over keepalived in this case? – CarpeNoctem Oct 21 '11 at 21:54
  • Simply because it was already set up in another instance of haproxy when I started working here. It works well for me, Willy T (haproxy other) has mentioned keepalived is more appropriate though IIRC ... let me see if I can find that... Here we go: http://www.formilux.org/archives/haproxy/1003/3259.html (When in doubt, do what Willy says ;-) ) – Kyle Brandt Oct 21 '11 at 21:57
  • The only reason I've needed to monitor the haproxy service at this point is human error. If I stop haproxy for some reason and don't stop the heartbeat first this can be messy. Maybe the solution to that is to setup the haproxy script to stop/start heartbeat along with it. – flickerfly May 21 '15 at 22:27
0

Thank you Kvisle and Kyle! We ended up going with keepalived since, as Kyle mentioned, it is the haproxy creators recommended solution. Keepalived includes the ability to run a user-space script of your choice and determine resource availability based on exit status. So you can very easily accomplish failover with process monitoring. Another benefit is the simplicity compared to heartbeat with pacemaker / corosync, just one application / process and one configuration file.

In our tests we have seen failover happen so quick as to not drop a single ping. Thanks again!

http://www.keepalived.org/

Great article on keepalived and VRRP

Also see: related serverfault answer

Community
  • 1
CarpeNoctem
  • 2,397
  • 4
  • 23
  • 32
  • Backup on the VRRP article which is no longer available: https://web.archive.org/web/20160421053420/http://linuxdevcenter.com/pub/a/linux/2005/09/01/keepalived.html?page=1 – duct_tape_coder Aug 08 '19 at 17:38