For obvious reasons, best practice in building clusters that require distributed consensus is to use 3, 5, or another node count that has a tiebreaker in place.
Sometimes, however, one has two physical devices and wants to cluster them. One way to get better availability than having no clustering at all, in these cases, is two give one of these systems two votes. One way to do this with Zookeeper is to configure a hierarchical quorum, and give one of the nodes two votes:
# 2-node zookeeper configuration that allows node-2 to fail without losing quorum
# ...node-1 cannot go down without losing quorum, *unless* these weights are first
# ...reconfigured; such configuration is allowed at runtime, with no service disruption!
group.1=1
group.2=2
weight.1=2
weight.2=1
In the original case, with each node having one vote, if either node failed then no quorum could be achieved. In this case, the node having only one vote is able to fail without bringing the other one down with it. Thus, we've successfully avoided having a second node make us less robust than we were when not clustered at all -- and for planned maintenance, when both nodes are present in the cluster we can update configuration with a new set of weights to allow group 1 but not group 2 to go offline without losing quorum.
...so, the question: How can this be achieved using etcd rather than zookeeper? Do I need to actually run two copies of etcd on node 1 (and, if we want to transfer this extra vote to node-2 to allow node-1 to be removed from the cluster, spin up a second copy there instead and decommission the second copy on node-1)? If so, can this be done without the overhead of having data synchronized to this instance (since its only purpose is to act as an observing voter)?
