Can't run drbdadm up with DRBD 8.4 on CentOS 7.3

Question

Nodes:

node1
node2

Use this way installed DRBD on both nodes:

rpm -ivh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
yum -y install drbd84-utils kmod-drbd84

Added new hard disk and do partition on both nodes:

fdisk /dev/sdb

Then format partition:

mkfs.ext4 /dev/sdb

(Failed on node2, so removed it and added a new disk become /dev/sdc. Formated it.)

Follow official guide to config the basic resource on both nodes:

http://docs.linbit.com/docs/users-guide-8.4/p-build-install-configure/#ch-configure

Created /etc/drbd.d/r0.res on both nodes:

resource r0 {
    protocol C;
    on node1 {
            device /dev/drbd0;
            disk /dev/sdb;
            address 192.168.0.1:7789;
            meta-disk internal;
    }
    on node2 {
            device /dev/drbd0;
            disk /dev/sdc;
            address 192.168.0.2:7789;
            meta-disk internal;
    }
}

Edit iptables on both nodes:

iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 7788 -j ACCEPT
iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 7799 -j ACCEPT
service iptables save

Initialize data on both nodes:

drbdadm create-md r0

All OK.

Put the resource up on both nodes: drbdadm up r0

node1:

Device '0' is configured!
Command 'drbdmeta 0 v08 /dev/sdb internal apply-al' terminated with exit code 20

node2:

??: Failure: (162) Invalid configuration request
additional info from kernel:
minor exists in different resource
Command 'drbdsetup-84 new-minor r0 0 0' terminated with exit code 10

Both of them happened issues.

Run cat /proc/drbd on both nodes:

node1:

version: 8.4.9-1 (api:1/proto:86-101)
GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by akemi@Build64R7, 2016-12-04 01:08:48
 0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r----s
    ns:0 nr:0 dw:0 dr:912 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:2097052

node2:

version: 8.4.9-1 (api:1/proto:86-101)
GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by akemi@Build64R7, 2016-12-04 01:08:48
 0: cs:WFConnection ro:Secondary/Unknown ds:Diskless/DUnknown C r----s
    ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

Primary and Secondary unknown.

Run drbdadm connect all on node1:

r0: Failure: (125) Device has a net-config (use disconnect first)
Command 'drbdsetup-84 connect r0 ipv4:192.168.0.1:7789 ipv4:192.168.0.2:7789 --protocol=C' terminated with exit code 10

Run drbdadm -- --discard-my-data connect all on node2:

r0: Failure: (102) Local address(port) already in use.
Command 'drbdsetup-84 connect r0 ipv4:192.168.0.1:7789 ipv4:192.168.0.2:7789 --protocol=C --discard-my-data' terminated with exit code 10

What's wrong of using it?

Edit 1

After ran dd if=/dev/zero of=/dev/sdb(sdc) bs=1M oflag=direct status=progress on both nodes, they also can't ran drbdadm up r0 successfully:

node1

No valid meta data found
Command 'drbdmeta 0 v08 /dev/sdb internal apply-al' terminated with exit code 255

node2

??: Failure: (162) Invalid configuration request
additional info from kernel:
minor exists in different resource
Command 'drbdsetup-84 new-minor r0 0 0' terminated with exit code 10

Edit 2

At this time, ran systemctl start brdb on both nodes, all of them got pending. Wait for a long time returned no result.

And ran lsblk -a now:

NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0   10G  0 disk
├─sda1        8:1    0    1G  0 part /boot
└─sda2        8:2    0    9G  0 part
  ├─cl-root 253:0    0    8G  0 lvm  /
  └─cl-swap 253:1    0    1G  0 lvm  [SWAP]
sdb           8:32   0   16G  0 disk
sr0          11:0    1 1024M  0 rom
drbd0       147:0    0    2G  0 disk

score 5 · Accepted Answer · answered Jul 20 '17 at 17:39

You don't want to create the filesystem on your backing disks for DRBD (sdb and sdc); you want to create the DRBD first, and then format the resulting DRBD device with your filesystem. It can be done the other way, but then you'd need to grow the backing partition, or reduce the size of the filesystem to make room for DRBD's metadata (which lives at the end of the block device when using internal metadata).

The process should be something like this:

-Install all your software like you have done above.

-Zero out your partition on both nodes (optional, but do it):

# dd if=/dev/zero of=/dev/sdb bs=1M oflag=direct status=progress
# dd if=/dev/zero of=/dev/sdc bs=1M oflag=direct status=progress

-Create the configuration file for your DRBD resources like you have done above.

-Add the firewall rules like you've done above (except you don't really need the port 7799, as that's not used in your configuration).

-Create DRBD's metadata on the backing block devices, and bring up r0 on both nodes:

# drbdadm create-md r0
# drbdadm up r0
# cat /proc/drbd

-Check the output of cat /proc/drbd, you should see your device is Connected, Secondary/Secondary, and Inconsistent/Inconsistent. If any of those things are NOT true, STOP, something isn't right.

-Then, pick either node (NOT BOTH), force it to be primary (DRBD will not let you go primary on a node with Inconsistent data) and create the FS:

# drbdadm primary r0 --force
# mkfs.ext4 /dev/drbd0

Then you'd use /dev/drbd0 just like you would have used /dev/sda or /dev/sdc; don't touch /dev/sdb or /dev/sdc ever again unless you're confident in what you're doing. Touching DRBD's backing disks can introduce inconsistencies that DRBD will be unaware of (until you run a verify or otherwise overwrite the block).

Keep reading LINBIT's documentation. DRBD is obviously a great tool, and is easy to use once you understand the basics, but you're dealing with storage so there is a lot of room to mess things up in a permanent way.

Thank you very much for your very good answer! I have did what you suggest: `Zero out your partition on both nodes` on both nodes. I was using basic 10GB disk on each node. Added 16GB new hard disk on each node. After run that zero out command, both of showed: `dd: '/dev/sdc' write error: not enough space exists`. So how much space is required in this case? — cloud_cloud, Jul 21 '17 at 02:07
After I ran your commands, the result was the same. I added the information to the bellow of the question. `Edit 1`. — cloud_cloud, Jul 21 '17 at 05:55
I know the reason. After I added the new hard disk on both nodes, I did partition and format on them! The only necessary thing is do the partition only. Then use DRBD configuration to control `/dev/sdb`. After initial data and up, then format `/dev/drbd0`. It's the right order. Ref: https://doc.zarafa.com/trunk/Zarafa_HA_Manual/en-US/html/_drbd_device_initialization.html — cloud_cloud, Jul 21 '17 at 09:10
@cloud_cloud incomplete. That last link also uses `sdb1` for DRBD instead of `sdb`. It's not an issue of the "right order", it's that you're writing stuff to a device which does or will underly DRBD (other than to clear it in advance as suggested in this answer). Don't do that, you don't know what you're doing :). — sourcejedi, Jul 21 '17 at 11:56

Can't run drbdadm up with DRBD 8.4 on CentOS 7.3

Edit 1

node1

node2

Edit 2

1 Answers1

Linked