Windl,
In order to eliminate the gap between virtualization and HA product documents, I set up a highly available virtualization environment, the detailed steps are as below,
SLE HA hardware environment details:
|
|
Cluster node size |
DELL Precision Tower 5810 |
Shared Disk |
Iscsi (50G) |
SLE HA version |
SLE15 HA SP2 |
# of the cluster nodes |
2 |
SBD partition |
Iscsi (50MB) |
Filesystem partition |
Iscsi (20GB) |
VM image partition |
Iscsi (30GB) |
Installation and setup process:
1. Attach iscsi disk to two cluster node, divide the iscsi disk into 3 partitions (e.g. 50MB for sbd, 20GB for ocfs2, the remain for VM image).
Add network bridge br0 on each cluster node (will be used when install/run virtual machine).
Set up password-free ssh login for the root user between cluster nodes.
2. Install HA and virtualization related packages on each cluster node.
# zypper in -t pattern ha_sles
# zypper in -t pattern kvm_server kvm_tools
3. Setup HA cluster and add SBD device.
refer to HA guide at http://docserv.suse.de/documents/en-us/sle-ha/15-SP2/single-html/SLE-HA-guide/#book-sleha-guide
4. Setup DLM and OCFS2 resources in crm.
e.g.
primitive dlm ocf:pacemaker:controld \
op monitor interval=60 timeout=60
primitive ocfs2-2 Filesystem \
params device="/dev/disk/by-id/scsi-149455400000000004100c3befec3dc9a81f9ce28f7a8b8de-part1" directory="/mnt/shared" fstype=ocfs2 \
op monitor interval=20 timeout=40
group base-group dlm ocfs2-2
clone base-clone base-group \
meta interleave=true
5. Setup virtlockd and libvirtd service on each cluster node.
edit /etc/libvirt/qemu.conf, set lock_manager = "lockd"
edit /etc/libvirt/qemu-lockd.conf, set file_lockspace_dir = "/mnt/shared/lockd" (note: /mnt/shared is ocfs2 file system mount point)
restart/enable libvirtd service (note: virtlockd service will be started by libvirtd service according to the configuration)
6. Install a virtual machine (e.g. sle15-nd) on the shared partition from one cluster node, dump the domain configuration to a XML file.
Move the virtual machine configuration file to the ocfs2 file system (e.g. /mnt/shared).
note: please make sure the XML configuration file does not include any references to unshared local paths.
7. Setup VirtualDomain resource and order in crm.
e.g.
primitive vm_nd1 VirtualDomain \
params config="/mnt/shared/sle15-nd.xml" remoteuri="qemu+ssh://%n/system" \
meta allow-migrate=true \
op monitor timeout=30s interval=10s \
utilization cpu=2 hv_memory=1024
order ord_fs_virt Mandatory: base-clone vm_nd1
8. Check all your changes with the show command in crm, then commit:
e.g.
crm(live/sle15sp2-test1)configure# show
node 172167755: sle15sp2-test2
node 172168091: sle15sp2-test1
primitive dlm ocf:pacemaker:controld \
op monitor interval=60 timeout=60
primitive ocfs2-2 Filesystem \
params device="/dev/disk/by-id/scsi-149455400000000004100c3befec3dc9a81f9ce28f7a8b8de-part1" directory="/mnt/shared" fstype=ocfs2 \
op monitor interval=20 timeout=40
primitive stonith-sbd stonith:external/sbd \
params pcmk_delay_max=30s
primitive vm_nd1 VirtualDomain \
params config="/mnt/shared/sle15-nd.xml" remoteuri="qemu+ssh://%n/system" \
meta allow-migrate=true \
op monitor timeout=30s interval=10s \
utilization cpu=2 hv_memory=1024
group base-group dlm ocfs2-2
clone base-clone base-group \
meta interleave=true
order ord_fs_virt Mandatory: base-clone vm_nd1
property cib-bootstrap-options: \
have-watchdog=true \
dc-version="2.0.3+20200511.2b248d828-1.10-2.0.3+20200511.2b248d828" \
cluster-infrastructure=corosync \
cluster-name=hacluster \
stonith-enabled=true
rsc_defaults rsc-options: \
resource-stickiness=1 \
migration-threshold=3
op_defaults op-options: \
timeout=600 \
record-pending=true
Verify if VM resource can work on HA cluster:
1. Verify VM resource is protected across cluster nodes.
Test result: cannot start the VM manually via virsh command when this VM is running on another cluster node.
2. Verify VM resource can be taken by another cluster node when the current cluster node crashes
Test result: after a few seconds (cluster fence time), the VM is started on another cluster node.
3. Verify VM resource can be taken by another cluster node when the current cluster node reboots
Test result: the VM is migrated to another cluster node.
4. Check if we can migrate VM resource between cluster nodes
Test result: Yes, the remote SSH connection to the VM is not broken during the whole migration.
Remarks:
1. In the actual production environment, cluster communication/management should use a separate network.
2. In the actual production environment, cluster sbd device should use a separate shared disk to avoid IO starvation.
3. Do not start VM instance manually until ocfs2 file system is mounted, since the file lockspace directory is under ocfs2 file system. In other words, you should let cluster(pacemaker) manage the start and stop of all virtual machines.