Active/Passive clusters are still heavilly used in many places, and running in production.
Please find below our production setup, it is working fine, and you can either let it run in manual mode (orchestrate=start), or enable automatic failover (orchestrate=ha). We use zfs to benefit from zfs send/receive, and zfs snapshots, but it is also possible to use drbd if you prefer synchronous replication.
Prerequisites :
- 2 nodes (in my setup 2 physical nodes 400 kilometers distance)
- internal disks
- 1 zfs pool on each node
- stretched vlan (in my setup we use "vrack" at OVH hosting provider)
Steps :
- install opensvc agent on both nodes (https://repo.opensvc.com)
- form opensvc cluster (3 commands needed, described in the screencast at https://www.opensvc.com)
- create a root ssh trust between both nodes
- create 1 opensvc service per kvm guest [service config file below]
root@node1:~$ svcmgr -s win1 print config
[DEFAULT]
env = PRD
nodes = node1.acme.com node2.acme.com
id = 7a10881d-e5d5-4817-a8fe-e7a2004c5520
orchestrate = start
[fs#1]
mnt_opt = rw,xattr,acl
mnt = /srv/{svcname}
dev = data/{svcname}
type = zfs
[container#0]
type = kvm
name = {svcname}
guestos = windows
shared = true
[sync#1]
src = data/{svcname}
dst = data/{svcname}
type = zfs
target = nodes
recursive = true
schedule = @12h
A few explanations :
- service is named "win1" and each
{svcname} in the service config file is a reference pointing to actual service name (win1)
- service start do the following :
- mount zfs dataset
data/win1 on mountpoint /srv/win1
- start kvm container
win1
- ressource
sync#1 is used to declare an asynchronous zfs dataset replication to the slave node (data/win1 on node1 is sent to data/win1 on node2), once per 12 hours in the example (zfs send/receive is managed by the opensvc agent)
- opensvc agent is also dealing with kvm qemu config replication, and defining it when the service is relocated to the slave node
Some management commands :
svcmgr -s win1 start start the servicesvcmgr -s win1 stop stop the servicesvcmgr -s win1 stop --rid container#0 stop the container referenced container#0 in the config filesvcmgr -s win1 switch relocate the service to the other nodesvcmgr -s win1 sync update trigger an incremental zfs dataset copysvcmgr -s win1 sync full trigger a full zfs dataset copy
Some services I manage also need zfs snapshots on a regular basis (daily/weekly/monthly), with retention, in this case I add the following config snippet to the service configuration file, and the opensvc agent does the job.
[sync#1sd]
type = zfssnap
dataset = data/{svcname}
schedule = 23:00-23:59@61
keep = 7
name = daily
recursive = true
sync_max_delay = 1d
[sync#1sw]
type = zfssnap
dataset = data/{svcname}
schedule = 23:00-23:59@61 sun
keep = 4
name = weekly
recursive = true
sync_max_delay = 7d
[sync#1sm]
type = zfssnap
dataset = data/{svcname}
schedule = 23:00-23:59@61 * *:first
keep = 6
name = monthly
recursive = true
sync_max_delay = 31d
As requested, I also add one lvm/drbd/kvm config :
drbd resource config /etc/drbd.d/kvmdrbd.res :
resource kvmdrbd {
device /dev/drbd10;
disk /dev/drbdvg/drbdlv;
on node1 {
address 1.2.3.4:12345;
meta-disk internal;
}
on node2 {
address 4.3.2.1:12345;
meta-disk internal;
}
}
opensvc service config file /etc/opensvc/kvmdrbd.conf :
root@node1# svcmgr -s kvmdrbd print config
[DEFAULT]
env = PRD
nodes = node1.acme.com node2.acme.com
id = 7a10881d-f4d3-1234-a2cd-e7a2018c4321
orchestrate = start
[disk#1]
type = lvm
vgname = {env.drbdvgname}
standby = true
[disk#2]
type = drbd
standby = true
shared = true
res = {svcname}
[fs#0]
mnt = {env.basedir}/{svcname}
type = ext4
dev = /dev/{env.drbddev}
shared = true
[container#0]
type = kvm
name = {svcname}
shared = true
[sync#i0]
schedule = @1440
[env]
basedir = /srv
drbddev = drbd10
drbdvgname = drbdvg
Some explanations :
- in my setup, I replicate lvm lv with drbd. I create a filesystem on the drbd block device. In this filesystem, I create 1 flat file per disk I want to present to the kvm guest.
disk#1 : is the lvm vg hosting the big logical volume. should be at least 5GB.disk#2 : is the drbd disk pointed by the drbd resource name. If opensvc service is named "foo", you should have /etc/drbd.d/foo.res. Or change disk#2.res parameter in the service config file.fs#0 : the main filesystem hosting all disk files for kvm guestcontainer#0 : the kvm guest, same name as the opensvc service in the example. agent must be able to dns resolve the kvm guest, to do a ping check before accepting to start the service (if ping answer, the kvm guest is already running somewhere, and it is not a good idea to start it. double start protection ensured by opensvc agent)standby = true : mean that this resource must remain up when the service is running on the other node. In our example, it is needed to keep drbd running fineshared = true : https://docs.opensvc.com/latest/agent.service.provisioning.html#shared-resources
