I'm trying to deploy a HA NFS Cluster Architecture with PCS on Rocky Linux 8.5 . Current kernel and nfs related package versions, pcs configurations are shown with detail below.
I'm not able to declare specific IP for NFSD instances (rpc.statd, rpc.mountd etc) to bind. Whatever i do, services keeps binding 0.0.0.0:$default-ports.
I would like to initiate different "ocf:Hearthbeat:nfsserver" with specific VirtualIP resource per each of my NFS(block) resource group. When i declare second NFS share resource group on the same node of cluster (where i plan to have more NFS than cluster size), "ocf:Hearthbeat:nfsserver" resources blocks each other, and one evantually wins and the other resource is gets in to the "blocked" state.
[root@node1 ~]# uname -a
Linux node1.local 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Wed Jan 19 17:53:40 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@node1 ~]# rpm -qa nfs* rpc*
nfs-utils-2.3.3-46.el8.x86_64
rpcbind-1.2.5-8.el8.x86_64
[root@node1 ~]#
PCS Cluster Status
[root@node1 ~]# pcs status
Cluster name: cluster01
Cluster Summary:
* Stack: corosync
* Current DC: node5 (version 2.1.0-8.el8-7c3f660707) - partition with quorum
* Last updated: Thu Mar 24 13:10:09 2022
* Last change: Thu Mar 24 13:03:48 2022 by root via crm_resource on node3
* 5 nodes configured
* 5 resource instances configured
Node List:
* Online: [ node1 node2 node3 node4 node5 ]
Full List of Resources:
* Resource Group: Group_SHARE:
* ROOT-FS_SHARE (ocf::heartbeat:Filesystem): Started node2
* NFSD_SHARE (ocf::heartbeat:nfsserver): Started node2
* NFS_SHARE (ocf::heartbeat:exportfs): Started node2
* NFS-IP_SHARE (ocf::heartbeat:IPaddr2): Started node2
* NFS-NOTIFY_SHARE (ocf::heartbeat:nfsnotify): Started node2
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@node1 ~]#
PCS Resource Config output
[root@node2 ~]# pcs resource config
Group: Group_SHARE
Resource: ROOT-FS_SHARE (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=/dev/disk/by-id/wwn-0x6001405ce6b7033688d497a91aa23547 directory=/srv/block/SHARE fstype=xfs
Operations: monitor interval=20s timeout=40s (ROOT-FS_SHARE-monitor-interval-20s)
start interval=0s timeout=60s (ROOT-FS_SHARE-start-interval-0s)
stop interval=0s timeout=60s (ROOT-FS_SHARE-stop-interval-0s)
Resource: NFSD_SHARE (class=ocf provider=heartbeat type=nfsserver)
Attributes: nfs_ip=10.1.31.100 nfs_no_notify=true nfs_shared_infodir=/srv/block/SHARE/nfsinfo/
Operations: monitor interval=10s timeout=20s (NFSD_SHARE-monitor-interval-10s)
start interval=0s timeout=40s (NFSD_SHARE-start-interval-0s)
stop interval=0s timeout=20s (NFSD_SHARE-stop-interval-0s)
Resource: NFS_SHARE (class=ocf provider=heartbeat type=exportfs)
Attributes: clientspec=10.1.31.0/255.255.255.0 directory=/srv/block/SHARE/SHARE fsid=0 options=rw,sync,no_root_squash
Operations: monitor interval=10s timeout=20s (NFS_SHARE-monitor-interval-10s)
start interval=0s timeout=40s (NFS_SHARE-start-interval-0s)
stop interval=0s timeout=120s (NFS_SHARE-stop-interval-0s)
Resource: NFS-IP_SHARE (class=ocf provider=heartbeat type=IPaddr2)
Attributes: cidr_netmask=24 ip=10.1.31.100 nic=team31
Operations: monitor interval=30s (NFS-IP_SHARE-monitor-interval-30s)
start interval=0s timeout=20s (NFS-IP_SHARE-start-interval-0s)
stop interval=0s timeout=20s (NFS-IP_SHARE-stop-interval-0s)
Resource: NFS-NOTIFY_SHARE (class=ocf provider=heartbeat type=nfsnotify)
Attributes: source_host=SHARE.local
Operations: monitor interval=30s timeout=90s (NFS-NOTIFY_SHARE-monitor-interval-30s)
reload interval=0s timeout=90s (NFS-NOTIFY_SHARE-reload-interval-0s)
start interval=0s timeout=90s (NFS-NOTIFY_SHARE-start-interval-0s)
stop interval=0s timeout=90s (NFS-NOTIFY_SHARE-stop-interval-0s)
[root@node2 ~]#
Virtual IP binded successfully on node2
[root@node2 ~]# ip -4 addr show team31
6: team31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
inet 10.1.31.2/24 brd 10.1.31.255 scope global noprefixroute team31
valid_lft forever preferred_lft forever
inet 10.1.31.100/24 brd 10.1.31.255 scope global secondary team31
valid_lft forever preferred_lft forever
[root@node2 ~]#
TCP LISTEN Binds
[root@node2 ~]# netstat -punta | grep LISTEN
tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:44321 0.0.0.0:* LISTEN 1803/pmcd
tcp 0 0 0.0.0.0:34661 0.0.0.0:* LISTEN 630273/rpc.statd
tcp 0 0 127.0.0.1:199 0.0.0.0:* LISTEN 1257/snmpd
tcp 0 0 127.0.0.1:4330 0.0.0.0:* LISTEN 2834/pmlogger
tcp 0 0 10.20.101.136:2379 0.0.0.0:* LISTEN 3285/etcd
tcp 0 0 10.20.101.136:2380 0.0.0.0:* LISTEN 3285/etcd
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd
tcp 0 0 0.0.0.0:20048 0.0.0.0:* LISTEN 630282/rpc.mountd
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 317707/nginx: maste
tcp 0 0 0.0.0.0:2224 0.0.0.0:* LISTEN 3725/platform-pytho
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1170/sshd
tcp 0 0 0.0.0.0:41017 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 317707/nginx: maste
tcp 0 0 0.0.0.0:35261 0.0.0.0:* LISTEN -
tcp6 0 0 :::2049 :::* LISTEN -
tcp6 0 0 ::1:44321 :::* LISTEN 1803/pmcd
tcp6 0 0 ::1:4330 :::* LISTEN 2834/pmlogger
tcp6 0 0 :::111 :::* LISTEN 1/systemd
tcp6 0 0 :::20048 :::* LISTEN 630282/rpc.mountd
tcp6 0 0 :::2224 :::* LISTEN 3725/platform-pytho
tcp6 0 0 :::37329 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN 1170/sshd
tcp6 0 0 :::41179 :::* LISTEN 630273/rpc.statd
tcp6 0 0 :::43487 :::* LISTEN -
[root@node2 ~]#
PCS Cluster Resources (cib.xml format,in case you need deep dive)
<resources>
<group id="Group_SHARE">
<primitive class="ocf" id="ROOT-FS_SHARE" provider="heartbeat" type="Filesystem">
<instance_attributes id="ROOT-FS_SHARE-instance_attributes">
<nvpair id="ROOT-FS_SHARE-instance_attributes-device" name="device" value="/dev/disk/by-id/wwn-0x6001405ce6b7033688d497a91aa23547"/>
<nvpair id="ROOT-FS_SHARE-instance_attributes-directory" name="directory" value="/srv/block/SHARE"/>
<nvpair id="ROOT-FS_SHARE-instance_attributes-fstype" name="fstype" value="xfs"/>
</instance_attributes>
<operations>
<op id="ROOT-FS_SHARE-monitor-interval-20s" interval="20s" name="monitor" timeout="40s"/>
<op id="ROOT-FS_SHARE-start-interval-0s" interval="0s" name="start" timeout="60s"/>
<op id="ROOT-FS_SHARE-stop-interval-0s" interval="0s" name="stop" timeout="60s"/>
</operations>
</primitive>
<primitive class="ocf" id="NFSD_SHARE" provider="heartbeat" type="nfsserver">
<instance_attributes id="NFSD_SHARE-instance_attributes">
<nvpair id="NFSD_SHARE-instance_attributes-nfs_ip" name="nfs_ip" value="10.1.31.100"/>
<nvpair id="NFSD_SHARE-instance_attributes-nfs_no_notify" name="nfs_no_notify" value="true"/>
<nvpair id="NFSD_SHARE-instance_attributes-nfs_shared_infodir" name="nfs_shared_infodir" value="/srv/block/SHARE/nfsinfo/"/>
</instance_attributes>
<operations>
<op id="NFSD_SHARE-monitor-interval-10s" interval="10s" name="monitor" timeout="20s"/>
<op id="NFSD_SHARE-start-interval-0s" interval="0s" name="start" timeout="40s"/>
<op id="NFSD_SHARE-stop-interval-0s" interval="0s" name="stop" timeout="20s"/>
</operations>
</primitive>
<primitive class="ocf" id="NFS_SHARE" provider="heartbeat" type="exportfs">
<instance_attributes id="NFS_SHARE-instance_attributes">
<nvpair id="NFS_SHARE-instance_attributes-clientspec" name="clientspec" value="10.1.31.0/255.255.255.0"/>
<nvpair id="NFS_SHARE-instance_attributes-directory" name="directory" value="/srv/block/SHARE/SHARE"/>
<nvpair id="NFS_SHARE-instance_attributes-fsid" name="fsid" value="0"/>
<nvpair id="NFS_SHARE-instance_attributes-options" name="options" value="rw,sync,no_root_squash"/>
</instance_attributes>
<operations>
<op id="NFS_SHARE-monitor-interval-10s" interval="10s" name="monitor" timeout="20s"/>
<op id="NFS_SHARE-start-interval-0s" interval="0s" name="start" timeout="40s"/>
<op id="NFS_SHARE-stop-interval-0s" interval="0s" name="stop" timeout="120s"/>
</operations>
</primitive>
<primitive class="ocf" id="NFS-IP_SHARE" provider="heartbeat" type="IPaddr2">
<instance_attributes id="NFS-IP_SHARE-instance_attributes">
<nvpair id="NFS-IP_SHARE-instance_attributes-cidr_netmask" name="cidr_netmask" value="24"/>
<nvpair id="NFS-IP_SHARE-instance_attributes-ip" name="ip" value="10.1.31.100"/>
<nvpair id="NFS-IP_SHARE-instance_attributes-nic" name="nic" value="team31"/>
</instance_attributes>
<operations>
<op id="NFS-IP_SHARE-monitor-interval-30s" interval="30s" name="monitor"/>
<op id="NFS-IP_SHARE-start-interval-0s" interval="0s" name="start" timeout="20s"/>
<op id="NFS-IP_SHARE-stop-interval-0s" interval="0s" name="stop" timeout="20s"/>
</operations>
</primitive>
<primitive class="ocf" id="NFS-NOTIFY_SHARE" provider="heartbeat" type="nfsnotify">
<instance_attributes id="NFS-NOTIFY_SHARE-instance_attributes">
<nvpair id="NFS-NOTIFY_SHARE-instance_attributes-source_host" name="source_host" value="SHARE.local"/>
</instance_attributes>
<operations>
<op id="NFS-NOTIFY_SHARE-monitor-interval-30s" interval="30s" name="monitor" timeout="90s"/>
<op id="NFS-NOTIFY_SHARE-reload-interval-0s" interval="0s" name="reload" timeout="90s"/>
<op id="NFS-NOTIFY_SHARE-start-interval-0s" interval="0s" name="start" timeout="90s"/>
<op id="NFS-NOTIFY_SHARE-stop-interval-0s" interval="0s" name="stop" timeout="90s"/>
</operations>
</primitive>
</group>
</resources>
EDIT-1
It seems OpenClusterFramework - nfsserver resource doesn't use nfs_ip field for "rpc.nfsd -H $nfs_ip" at all. On RockyLinux 8.5 this resource also doesn't allow us to overwrite the default behaviour of supporting each NFS versions. RockyLinux 8.5 uses following packages resource-agents-4.1.1-98.el8_5.2.x86_64.
I will try to solve my issue by specifying custom pcs systemd resources for nfs-server@i.service
