8

The DRBD documentation (in section Integrating DRBD with Pacemaker clusters) recommends that DRBD should be disabled in a Pacemaker cluster:

If you are employing the DRBD OCF resource agent, it is recommended that you defer DRBD startup, shutdown, promotion, and demotion exclusively to the OCF resource agent. That means that you should disable the DRBD init script: chkconfig drbd off.

Under systemd this would amount to systemctl disable drbd.service.

Is there any harm in enabling DRBD despite of this recommendation? The idea would be to enable DRBD, but to disable Corosync and Pacemaker, so that after a cluster node fails and reboots it will continue receiving DRBD-synced data but will otherwise remain "passive". This should allow for the failed node to be analyzed before it can reenter the cluster, but live data still being saved on both cluster nodes in the meantime. What is the rationale behind the recommendation that would outweigh this rationale?

rookie099
  • 345
  • 2
  • 14
  • 1
    _The idea would be to enable DRBD, but to disable Corosync and Pacemaker,_ What controls your cluster, then? What you are describing is usually handled via resource stickiness etc. – Lenniey Jan 15 '19 at 12:32
  • @Lenny Pacemaker controls the cluster. It is active in normal operation. But when a node fails it does not restart automatically. At least that's the configuration I've inherited :) – rookie099 Jan 15 '19 at 13:30
  • 2
    Well the intention of disabling the DRBD service on the OS level is that everything is controlled by pacemaker. If two services (PCMK and your OS, for example) are trying to start / stop / promote / demote etc., you are risking split-brains. For a controlled cluster-environment, everything should be handled by your cluster resource manager, in this case pacemaker, to avoid confusion between the cluster nodes. In a case of split-brain or similar, you CRM will either STONITH or fence or use the configured quorum on the other nodes to resolve it. – Lenniey Jan 15 '19 at 13:34

3 Answers3

9

Well the intention of disabling the DRBD service on the OS level is that everything is controlled by pacemaker. If two services (PCMK and your OS, for example) are trying to start / stop / promote / demote etc., you are risking split-brains. For a controlled cluster-environment, everything should be handled by your cluster resource manager, in this case pacemaker, to avoid confusion between the cluster nodes. In a case of split-brain or similar, you CRM will either STONITH or fence or use the configured quorum on the other nodes to resolve it.

Lenniey
  • 5,090
  • 2
  • 17
  • 28
3

When using a cluster resource manager, any resource should be controlled by, well, the resource manager. Any resource enabled/disabled from the outside of the cluster resource manager is a potential source of confusion, both for the administrator and the resource manager itself.

shodanshok
  • 44,038
  • 6
  • 98
  • 162
3

There are two answers already that detail clearly that this is a bad idea and why, but maybe some details as to how it could go wrong for you and how you can use Pacemaker to address these problems would help to convince you and/or others to not do things this way.

First, Pacemaker logs and accounts for resource failures. The default failure count for a resource before it gets "banned" from a node is three within the resource-failure-timeout window, which by default never times out. So if your DRBD resource (or any other resource for that matter) fails three times in a row, it is banned from its currently active node by using a strong (infinite) "negative location constraint", meaning that the resource can run anywhere BUT its currently active node. Once that rule is in place, the resource either moves elsewhere if it can, or it stops until its failures are addressed.

So you can see, Pacemaker can be made to handle these failures gracefully on its own.

You need to understand what Pacemaker is and how it behaves to grok why managing resources it enforces the state of outside of Pacemaker is bad. Pacemaker is a finite state system. It depends on being in complete control of the resources that it manages so that it can gracefully recover from failures and ensure that resources are either stopped or started where they should be.

Consider a simple resource that should only be run on one node at a time, lest it become "split-brain" and create a divergent dataset - just about the worst thing that could happen, as this will almost certainly cause either data loss or require large amounts of operator attention to prevent data loss.

Pacemaker controls this resource, and starts an instance of the software on node "Able". A well-meaning administrator finds that the service is started on Able, but that its systemd unit file is "disabled". That admin enables the unit file so that the service will "come back" on reboot, unaware that Pacemaker is handling this already. The systemd unit file is configured to restart the resource on failure, as many are.

Once Pacemaker tries to migrate this resource away from Able to the second node in the cluster "Baker", the resource encounters a stop failure, as the service was killed but somehow it's still alive and we're in the middle of a zombie apocalypse. Since the resource cannot be stopped, it cannot be started on Baker without causing a split-brain condition. The resource flaps between stopped and started as systemd and Pacemaker battle for control. Eventually, Pacemaker "gives up" on the resource and puts it in "unmanaged mode", meaning that no start or stop operations will be performed on that resource.

So in that scenario, Systemd won because it was "stupider and more insistent" than Pacemaker. This is extremely difficult for an admin who's not familiar with the behavior of both Pacemaker and Systemd to understand, as it will simply look like Pacemaker is failing all over the place -- when in reality it's doing exactly what it's supposed to do given the conditions at hand.

Also consider that the above scenario had the best possible ending for that condition. Given the slightest infrastructure failure, the cluster would have become split-brain with that resource active on both nodes.

As an aside, fencing via STONTIH would prevent the cluster from becoming split-brain in that scenario, but STONITH is a last resort for cluster stability, while the above condition would put it as almost a first resort. And as always, you NEED STONITH to make a cluster production-ready.

Spooler
  • 7,016
  • 16
  • 29
  • This is very helpful, esp. the scenario where Systemd may restart a failed service (when it is enabled). BTW, STONITH is in place. If I understand correctly the motivation was that a failed node would reboot after STONITH and that it should then not rejoin the cluster immediately/automatically (hence `corosync` and `pacemaker` disabled) but should restart to DRBD-sync data immediately (hence `drbd` enabled), the second as a precaution in case the remaining node also failed "somehow" in the meantime. – rookie099 Jan 16 '19 at 17:37
  • `systemctl cat drbd.service | grep ^Restart=` says `Restart=no`. – rookie099 Jan 16 '19 at 21:11
  • Yeah, that one shouldn't be set to restart. All the same, if you have the systemd unit enabled, it will be started on boot (which is not acceptable). – Spooler Jan 16 '19 at 21:14
  • So I guess your advice against "not acceptable" is then to `systemctl disable drbd.service`, `systemctl enable corosync.service`, and `systemctl enable pacemaker.service` on both cluster nodes and use `stonith-action=reboot`, right? – rookie099 Jan 16 '19 at 21:33
  • 1
    The default STONITH action is usually the most appropriate for your agent, unless you're solving some kind of issue by defining that action explicitly. As for the rest, yes. – Spooler Jan 16 '19 at 21:35
  • 1
    @rookie099 it really depends on your use-case. For example: Most of my PCMK clusters are set to STONITH = poweroff via IPMI, because then I have the time to investigate in a controlled manner before the node rejoins the cluster or starts. On one other cluster it is set to restart, but this one controls far less resources than the others, so there should not be anything breaking because of STONITH. – Lenniey Jan 17 '19 at 08:03
  • How would you handle this use case then: 1) node 2 encounters problems (e.g. transient network problem) and is STONITHed; 2) DRBD disk (or its RAID controller) fails on node 1 while you are investigating; 3) you cannot afford to loose any (new) DRBD data. – rookie099 Jan 17 '19 at 10:54
  • That would be... unfortunate. Essentially that's a 100% failure of your storage stack which would stop the cluster entirely. So at least you wouldn't have to worry about any new data :) Either bringing your fenced node back online or allowing it to reboot on fence would at least get the cluster back online, but probably with some point-in-time data loss. And you'd likely want to put that node with a dead RAID controller into "standby", which will prevent anything from running on it. – Spooler Jan 17 '19 at 18:01
  • And if you're concerned with losing a single node and only having one left, then use three nodes. DRBD-9 can do more than two nodes (up to 1024 nodes with up to 32 data replicas among those nodes for a single "resource"). – Spooler Jan 17 '19 at 18:14
  • 2-node HA is very finicky. Even with all available tools it's rather failure-prone. You should AT LEAST have another quorum / witness member. Depending on the use-case (again) for filesystem HA you should go another way with 3-nodes (and DRBD) or a different filesystem (a clustering FS of course), or CEPH or whatever. – Lenniey Jan 23 '19 at 11:18
  • If you're using DRBD, you should avoid dual-primary and clustering filesystems such as GFS2 on a dual-primary setup. Regarding two node clusters, three nodes is always better than two - however, two node clusters can be made to be extremely reliable. You *must* use fencing, DRBD should be configured to integrate with that fencing, and you *should* be using redundant corosync rings. The guarantees that redundant rings and fencing provide reach near-total parity with similarly configured three node clusters at a 33% cost reduction. DRBD+XFS will perform better than Ceph. – Spooler Jan 23 '19 at 15:24