Heartbeat meatware STONITH on kernel panic

Question

I have a two node cluster with heartbeat and DRBD managing a mysql resource. The failover works great if I halt the primary, reboot it, or disconnect the network connection.

However, if the primary suffers from a kernel panic (simulated by running echo c > /proc/sysrq-trigger), the secondary does not takeover the resources.

This is what the heartbeat log on the secondary looks like:

Jul 11 21:33:32 rad11 heartbeat: [7519]: WARN: node rad10: is dead
Jul 11 21:33:32 rad11 heartbeat: [7519]: info: Link rad10:eth0 dead.
Jul 11 21:33:32 rad11 heartbeat: [8442]: info: Resetting node rad10 with [Meatware STONITH device]
Jul 11 21:33:32 rad11 heartbeat: [8442]: ERROR: glib: OPERATOR INTERVENTION REQUIRED to reset rad10.
Jul 11 21:33:32 rad11 heartbeat: [8442]: ERROR: glib: Run "meatclient -c rad10" AFTER power-cycling the machine.

Does anybody have any idea why the secondary fails to take-over in this situation? Normally failover works great, but I'm trying to simulate a kernel panic on the primary node.

EDIT: Here is my heartbeat config, ha.cf

# /etc/ha.d/ha.cf

logfile /var/log/ha-log

keepalive 1

deadtime 10

udpport 695

ucast eth0 rad11
auto_failback on
stonith_host rad10 meatware rad11
stonith_host rad11 meatware rad10
node rad10 rad11

Answer 1

When cluster nodes lose contact with each other, to avoid a split-brain scenario, where both nodes think that they are primary and tries to simultaniously run the shared resource with potential disaster as a result (this is especially a big problem in two node clusters, because who has quorum if both nodes have one vote each?), so to mitigate this, some clusters implement various forms of fencing.

On linux-ha wiki page:

Fencing is the process of locking resources away from a node whose status is uncertain.

There are a variety of fencing techniques available.

One can either fence nodes - using Node Fencing, or fence resources using Resource Fencing. Some types of resources are Self Fencing Resources, and some aren't damaged by simultaneous use, and don't require fencing at all.

When a node preforms a clean shutdown, it will nicely leave the cluster, and thereby the others will know what's up and therefore will just take over any services the node might have been running and then carry on. When the node instead of leaves the cluster nicely gets a kernel panic, the other cluster members won't know the status of the other node. It will be "uncertain" from their point of view, so instead they will perform the configured "fencing" actions, which in the case of STONITH means trying to remove the fauly node by force from the cluster (by power-cycling etc).

Looking in your logs, it seems like the meatware STONITH mechanism is chosen for your cluster configuration. Like the name suggests, it implies manually power cycling the other node and then running said command. From doc:

meatware

Strange name and a simple concept. meatware requires help from a human to operate. Whenever invoked, meatware logs a CRIT severity message which should show up on the node’s console. The operator should then make sure that the node is down and issue a meatclient(8) command to tell meatware that it’s OK to tell the cluster that it may consider the node dead. See README.meatware for more information.

There are other ways to configure fencing. When making a cluster, I usually get two APC switches for the PSU:s and configure "APC fencing" (stonith -t apcmaster -h). That way, when one node fails, the other will preform a hard reboot by power-cycling the faulty member through logging into the APC interface and sending shutdown/reboot command on connected PSU slots (I get two to avoid a single point of failure).