Can not switch drbd to secondary

Question

I'm running drbd83 with ocfs2 in centos 5 and planning to use packemaker with them. Afer some time, I'm facing drbd split brain problem.

version: 8.3.13 (api:88/proto:86-96)
GIT-hash: 83ca112086600faacab2f157bc5a9324f7bd7f77 build by mockbuild@builder10.centos.org, 2012-05-07 11:56:36

 1: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown   r-----
    ns:0 nr:0 dw:112281991 dr:797551 al:99 bm:6401 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:60

I can't switch my drbd to secondary.

drbdadm secondary r0
1: State change failed: (-12) Device is held open by someone
Command 'drbdsetup 1 secondary' terminated with exit code 11

My drbd resource config:

resource r0 {
        syncer { 
                rate 1000M; 
                verify-alg sha1;
        }

        disk {
                on-io-error detach;
        }

        handlers {
           pri-lost-after-sb "/usr/lib/drbd/notify-split-brain.sh root";
        }

        net {
                allow-two-primaries;
                after-sb-0pri discard-younger-primary;
                after-sb-1pri call-pri-lost-after-sb;
                after-sb-2pri call-pri-lost-after-sb;
        }
        startup { become-primary-on both; }

        on serving_4130{
                device          /dev/drbd1;
                disk            /dev/sdb1;
                address         192.168.4.130:7789;
                meta-disk       internal;
        }
        on MT305-3182 {
                device          /dev/drbd1;
                disk            /dev/xvdb1;
                address         192.168.3.182:7789;
                meta-disk       internal;
        }
}

Status of ocfs2 status:

service ocfs2 status
Configured OCFS2 mountpoints:  /data

lsof show that, there is one process relative with drbd.

lsof | grep drbd
COMMAND     PID      USER   FD      TYPE             DEVICE     SIZE       NODE NAME
drbd1_wor  7782      root  cwd       DIR              253,0     4096          2 /
drbd1_wor  7782      root  rtd       DIR              253,0     4096          2 /
drbd1_wor  7782      root  txt   unknown                                        /proc/7782/exe

And it's a dead symlink:

# ls -l /proc/7782/exe
ls: cannot read symbolic link /proc/7782/exe: No such file or directory
lrwxrwxrwx 1 root root 0 May  4 09:56 /proc/7782/exe

# ps -ef | awk '$2 == "7782" { print $0 }'
root      7782     1  0 Apr22 ?        00:00:20 [drbd1_worker]

Notice that this process is wrapped in square brackets:

man ps:

args       COMMAND  command with all its arguments as a string. Modifications to the arguments may be shown. The
                    output in this column may contain spaces. A process marked <defunct> is partly dead, waiting to
                    be fully destroyed by its parent. Sometimes the process args will be unavailable; when this
                    happens, ps will instead print the executable name in brackets.

So, the final question is: how can we manually recover DRBD in this case without rebooting?

Reply to @andreask:

My partition table:

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       35G  6.9G   27G  21% /
/dev/xvda1             99M   20M   74M  22% /boot
tmpfs                 1.0G     0  1.0G   0% /dev/shm
/dev/drbd1            100G  902M  100G   1% /data

The device names:

# dmsetup ls --tree -o inverted
 (202:2)
 ├─VolGroup00-LogVol01 (253:1)
 └─VolGroup00-LogVol00 (253:0)

Pay attention to the block device (253:0), it is the same as from the output of lsof:

# lvdisplay 
  --- Logical volume ---
  LV Name                /dev/VolGroup00/LogVol00
  VG Name                VolGroup00
  LV UUID                vCd152-amVZ-GaPo-H9Zs-TIS0-KI6j-ej8kYi
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                35.97 GB
  Current LE             1151
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

Reply to @Doug:

# vgdisplay 
  --- Volume group ---
  VG Name               VolGroup00
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               39.88 GB
  PE Size               32.00 MB
  Total PE              1276
  Alloc PE / Size       1276 / 39.88 GB
  Free  PE / Size       0 / 0   
  VG UUID               OTwzII-AP5H-nIbH-k2UA-H9nw-juBv-wcvmBq

UPDATE Fri May 17 16:08:16 ICT 2013

Here are some ideas from Lars Ellenberg:

if the filesystem is still mounted ... oh well. unmount it. not lazy, but really.

I'm sure, OCFS2 was already unmounted.

If nfs was involved, try

killall -9 nfsd
killall -9 lockd
echo 0 > /proc/fs/nfsd/threads

No, NFS was not involved.

if lvm/dmsetup/kpartx/multipath/udev is involved, try
dmsetup ls --tree -o inverted
and check if there are dependencies from drbd.

As you can see from my above output, LVM doesn't related to DRBD:

pvdisplay -m

  --- Physical volume ---
  PV Name               /dev/xvda2
  VG Name               VolGroup00
  PV Size               39.90 GB / not usable 20.79 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              1276
  Free PE               0
  Allocated PE          1276
  PV UUID               1t4hkB-p43c-ABex-stfQ-XaRt-9H4i-51gSTD

  --- Physical Segments ---
  Physical extent 0 to 1148:
    Logical volume      /dev/VolGroup00/LogVol00
    Logical extents     0 to 1148
  Physical extent 1149 to 1275:
    Logical volume      /dev/VolGroup00/LogVol01
    Logical extents     0 to 126

fdisk -l

Disk /dev/xvda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot      Start         End      Blocks   Id  System
/dev/xvda1   *           1          13      104391   83  Linux
/dev/xvda2              14        5221    41833260   8e  Linux LVM

Disk /dev/xvdb: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdb1               1       13054   104856223+  83  Linux

if loop/cryptoloop/etc is involved, check if one of those is still accessing them.

if some virtualization tecknique is in use, shut down/destroy all containers/VMs that may have been accessing that drbd during their life time.

No, it doesn't.

Sometimes it is just udev or equivalent doing a race.

I have disabled the multipath rule and even stop the udevd, and nothing change.

Sometimes it is a unix domain socket or similar still held open (won't necessary show up in lsof/fuser).

If so, how can we find out this unix socket?

UPDATE Wed May 22 22:10:41 ICT 2013

Here is the stacktrace of DRBD worker process when dumping via magic SysRq key:

kernel: drbd1_worker  S ffff81007ae21820     0  7782      1          7795  7038 (L-TLB)
kernel:  ffff810055d89e00 0000000000000046 000573a8befba2d6 ffffffff8008e82f 
kernel:  00078d18577c6114 0000000000000009 ffff81007ae21820 ffff81007fcae040 
kernel:  00078d18577ca893 00000000000002b1 ffff81007ae21a08 000000017a590180 
kernel: Call Trace: 
kernel:  [<ffffffff8008e82f>] enqueue_task+0x41/0x56 
kernel:  [<ffffffff80063002>] thread_return+0x62/0xfe 
kernel:  [<ffffffff80064905>] __down_interruptible+0xbf/0x112 
kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe 
kernel:  [<ffffffff80064713>] __down_failed_interruptible+0x35/0x3a 
kernel:  [<ffffffff885d461a>] :drbd:.text.lock.drbd_worker+0x2d/0x43 
kernel:  [<ffffffff885eca37>] :drbd:drbd_thread_setup+0x127/0x1e1 
kernel:  [<ffffffff800bab82>] audit_syscall_exit+0x329/0x344 
kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11 
kernel:  [<ffffffff885ec910>] :drbd:drbd_thread_setup+0x0/0x1e1 
kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11

I'm not sure if this OCFS2 heartbeat region is preventing DRBD from switching to secondary:

kernel: o2hb-C3E41CA2 S ffff810002536420     0  9251     31                3690 (L-TLB)
kernel:  ffff810004af7d20 0000000000000046 ffff810004af7d30 ffffffff80063002
kernel:  1400000004000000 000000000000000a ffff81007ec307a0 ffffffff80319b60
kernel:  000935c260ad6764 0000000000000fcd ffff81007ec30988 0000000000027e86
kernel: Call Trace:
kernel:  [<ffffffff80063002>] thread_return+0x62/0xfe
kernel:  [<ffffffff8006389f>] schedule_timeout+0x8a/0xad
kernel:  [<ffffffff8009a41d>] process_timeout+0x0/0x5
kernel:  [<ffffffff8009a97c>] msleep_interruptible+0x21/0x42
kernel:  [<ffffffff884b3b0b>] :ocfs2_nodemanager:o2hb_thread+0xd2c/0x10d6
kernel:  [<ffffffff80063002>] thread_return+0x62/0xfe
kernel:  [<ffffffff800a329f>] keventd_create_kthread+0x0/0xc4
kernel:  [<ffffffff884b2ddf>] :ocfs2_nodemanager:o2hb_thread+0x0/0x10d6
kernel:  [<ffffffff800a329f>] keventd_create_kthread+0x0/0xc4
kernel:  [<ffffffff80032632>] kthread+0xfe/0x132
kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
kernel:  [<ffffffff800a329f>] keventd_create_kthread+0x0/0xc4
kernel:  [<ffffffff80032534>] kthread+0x0/0x132
kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11

This is a programming forum, read the [FAQ](http://stackoverflow.com/faq); therefore this question would be a better fit for [Superuser](http://superuser.com). Voting to close. — t0mm13b, Jul 18 '12 at 21:24
Did you `umount` `ocfs` before trying to demote it to secondary? — Nils, Aug 03 '12 at 21:03
@Nils: Yes, OCFS2 was already unmounted before trying to demote. — quanta, May 20 '13 at 02:53

score 6 · Answer 1 · edited Jul 28 '12 at 15:38

6

A common reason for DRBD being unable to demote a resource is an active device-mapper device ... like a volume group. You can check it e.g. with:

dmsetup ls --tree -o inverted

edited Jul 28 '12 at 15:38

quanta

50,327
19
152
213

answered Jul 25 '12 at 21:38

andreask

61
1

1

`dmsetup ls --tree -o inverted` (8:2) ├─VolGroup00-LogVol01 (253:1) └─VolGroup00-LogVol00 (253:0) So how can I deal with this? – favadi Jul 28 '12 at 04:03
Either just remove it from device mapper or deactivate the volume group (vgchange -an volume_group). – Doug Jul 30 '12 at 10:27
DRBD doesn't use Logical Volume as a backing device. – quanta Aug 04 '12 at 03:49
and if it's dangling device, you can remove the device with `dmsetup remove`. – Igor Galić May 01 '13 at 09:30

score 3 · Accepted Answer · edited Apr 13 '17 at 12:14

I'm not sure if this OCFS2 heartbeat region is preventing DRBD from switching to secondary:

Maybe. Have you tried to kill that region follow this guide?

# /etc/init.d/o2cb offline serving
Stopping O2CB cluster serving: Failed
Unable to stop cluster as heartbeat region still active

OK, firstly you should list the OCFS2 volumes along with their labels and uuids:

# mounted.ocfs2 -d
Device                FS     Stack  UUID                              Label
/dev/sdb1             ocfs2  o2cb   C3E41CA2BDE8477CA7FF2C796098633C  data_ocfs2
/dev/drbd1            ocfs2  o2cb   C3E41CA2BDE8477CA7FF2C796098633C  data_ocfs2

Secondly, check to see if you have any reference to this device:

# ocfs2_hb_ctl -I -d /dev/sdb1
C3E41CA2BDE8477CA7FF2C796098633C: 1 refs

Try to kill it:

# ocfs2_hb_ctl -K -d /dev/sdb1 ocfs2

then stop the cluster stack:

# /etc/init.d/o2cb stop
Stopping O2CB cluster serving: OK
Unmounting ocfs2_dlmfs filesystem: OK
Unloading module "ocfs2_dlmfs": OK
Unmounting configfs filesystem: OK
Unloading module "configfs": OK

and bring the device back into secondary role:

# drbdadm secondary r0
# drbd-overview 
  1:r0  StandAlone Secondary/Unknown UpToDate/DUnknown r-----

Now you can recover split brain as usual:

# drbdadm -- --discard-my-data connect r0
# drbd-overview 
  1:r0  WFConnection Secondary/Unknown UpToDate/DUnknown C r-----

On the other node (the split brain survivor):

# drbdadm connect r0
# drbd-overview                                                                                                
  1:r0  SyncSource Primary/Secondary UpToDate/Inconsistent C r---- /data ocfs2 100G 1.9G 99G 2% 
        [>....................] sync'ed:  3.2% (753892/775004)K delay_probe: 28

On the split brain victim:

# /etc/init.d/o2cb start
Loading filesystem "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading filesystem "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster serving: OK

# /etc/init.d/ocfs2 start
Starting Oracle Cluster File System (OCFS2)                [  OK  ]

Verify that this mount point is up and running:

# df -h /data/
Filesystem            Size  Used Avail Use% Mounted on
/dev/drbd1            100G  1.9G   99G   2% /data

Great work! You save me from frustration with DRBD. The problem is solved after nearly one year long. Thank you! — favadi, May 22 '13 at 16:29

Eric L · Answer 3 · 2022-01-04T09:18:35.020

For me the cause was multipathd.

I followed Pacemaker's official documentation up to the end of section 8.4 without installing OCFS2 or GFS2 (on Ubuntu 20.04), and wasn't able to demote a DRBD host from primary down to secondary:

eric@host1:~$ sudo drbdadm status
r0 role:Primary
  disk:UpToDate
  peer role:Secondary
    replication:Established peer-disk:UpToDate

eric@host1:~$ sudo drbdadm secondary r0
1: State change failed: (-12) Device is held open by someone
Command 'drbdsetup-84 secondary 1' terminated with exit code 11

Checking who's holding it open:

eric@host1:~$ sudo fuser -m /dev/drbd1
/dev/drbd1:   558

eric@host1:~$ ps aux | grep 558
root  558  0.0  0.8 345856 18124 ?  SLsl 07:04  0:00 /sbin/multipathd -d -s

Stop multipathd:

eric@host1:~$ sudo systemctl stop multipathd
Warning: Stopping multipathd.service, but it can still be activated by: multipathd.socket

eric@host1:~$ sudo systemctl stop multipathd.socket
eric@host1:~$ sudo fuser -m /dev/drbd1

Now try demoting to secondary again:

eric@host1:~$ sudo drbdadm secondary r0
eric@host1:~$ sudo drbdadm status
r0 role:Secondary
  disk:UpToDate
  peer role:Secondary
    replication:Established peer-disk:UpToDate

Success!

Before you restart multipathd, add your DRBD device (/dev/drbd1 in my case) to multipath's blacklist:

Add this section to the bottom of /etc/multipath.conf:

blacklist {
    devnode "^drbd[0-9]"
}

Restart multipathd.socket, then multipathd

eric@host1:~$ sudo systemctl start multipathd.socket
eric@host1:~$ sudo systemctl start multipathd

Can not switch drbd to secondary

3 Answers3