How to re-add a drive when RAID is in degraded mode?

Question

I am running an unmanaged Ubuntu server and due to recent performance issue and sent a request to the provider only to receive the following response:

"Dear Client,

as requested we have checked your drives. Both drives pass a test without any error and do not show any critical value here. We noticed the RAID is degraded state so you will need to re-add the drive to array here:

-----------------%<-----------------
Personalities : [raid1]
md3 : active raid1 sda4[0]
      1839089920 blocks super 1.2 [2/1] [U_]

md2 : active raid1 sda3[0]
      1073610560 blocks super 1.2 [2/1] [U_]

md1 : active raid1 sda2[0]
      523968 blocks super 1.2 [2/1] [U_]

md0 : active raid1 sda1[0]
      16768896 blocks super 1.2 [2/1] [U_]

unused devices: <none>
-----------------%<-----------------"

Google says that U_ indicates a failed drive, which appears to be in contradiction to what the support said. How do I proceed from here?

Something like

mdadm /dev/mdX -r /dev/sdaY  
mdadm /dev/mdX -a /dev/sdaY

And if so, what is X and Y for my case? If more info is needed about my configuration, please let me know.

EDIT, more info below:

root@Ubuntu-1204-precise-64-minimal /home # df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/md2       1008G   26G  931G   3% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev             16G  4.0K   16G   1% /dev
tmpfs           3.2G  504K  3.2G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none             16G     0   16G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/md1        488M   66M  397M  15% /boot
/dev/md3        1.7T   31G  1.6T   2% /home

root@Ubuntu-1204-precise-64-minimal /home # mdadm -D /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Wed Nov  6 08:02:41 2013
     Raid Level : raid1
     Array Size : 16768896 (15.99 GiB 17.17 GB)
  Used Dev Size : 16768896 (15.99 GiB 17.17 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Wed Dec 21 03:50:13 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:0
           UUID : 872ad258:c42ccb36:e9e19c96:98b55ee9
         Events : 816844

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       0        0        1      removed

root@Ubuntu-1204-precise-64-minimal /home # mdadm -D /dev/md1
/dev/md1:
        Version : 1.2
  Creation Time : Wed Nov  6 08:02:41 2013
     Raid Level : raid1
     Array Size : 523968 (511.77 MiB 536.54 MB)
  Used Dev Size : 523968 (511.77 MiB 536.54 MB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Wed Dec 21 07:30:55 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:1
           UUID : 18cb39fc:9eaea61c:0074a6c2:661b5862
         Events : 387

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       0        0        1      removed

root@Ubuntu-1204-precise-64-minimal /home # mdadm -D /dev/md2
/dev/md2:
        Version : 1.2
  Creation Time : Wed Nov  6 08:02:42 2013
     Raid Level : raid1
     Array Size : 1073610560 (1023.87 GiB 1099.38 GB)
  Used Dev Size : 1073610560 (1023.87 GiB 1099.38 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Wed Dec 21 07:58:52 2016
          State : active, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:2
           UUID : eb9be750:7ff778b4:31fd7ce9:9d86d191
         Events : 38799724

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       0        0        1      removed

root@Ubuntu-1204-precise-64-minimal /home # mdadm -D /dev/md3
/dev/md3:
        Version : 1.2
  Creation Time : Wed Nov  6 08:02:42 2013
     Raid Level : raid1
     Array Size : 1839089920 (1753.89 GiB 1883.23 GB)
  Used Dev Size : 1839089920 (1753.89 GiB 1883.23 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Wed Dec 21 07:57:41 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:3
           UUID : c9b748ef:332d3bf9:5fa8fef1:5b433b0a
         Events : 7429895

    Number   Major   Minor   RaidDevice State
       0       8        4        0      active sync   /dev/sda4
       1       0        0        1      removed

Are you seeking a way to fix that drive problem, or how to deal with the support answer ? — Tolsadus, Dec 21 '16 at 07:00
Please note that I have reverted your spell correction (readd is really readd, or re-add, not read). — captain lizard, Dec 21 '16 at 07:02
Maybe https://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array ? — Tolsadus, Dec 21 '16 at 07:06
If the controller ejected a drive from the array, even if no hard error is shown I never thrust back a disk. I already seen a drive without error repluged and it corrupted the array data. — yagmoth555, Dec 21 '16 at 12:19

Tolsadus · Answer 1 · 2020-05-15T15:31:53.043

Adding Back a RAID 1 Member

To add a RAID member back into the array, use:

mdadm <RAID Device> -a <Device to add into the array>

For example:

$ mdadm /dev/md0 -a /dev/sdc1
mdadm: re-added /dev/sdc1

If you check again using mdadm --detail , you'll see that /dev/sdc1 is shown as "rebuilding":

Number   Major   Minor   RaidDevice State
   0       8       17        0      active sync   /dev/sdb1
   2       8       33        1      spare rebuilding   /dev/sdc1

This means that the newly added member is being synchronised with the data on the other member; if you now check proc/mdstat, you can see the synchronisation progress - as a percentage and a bar chart:

$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sdc1[2] sdb1[0]
      976759936 blocks [2/1] [U_]
      [>....................]  recovery =  1.2% (12409024/976759936) finish=195.2min speed=82312K/sec

unused devices: <none>

Verify that you can still see your files on the RAID drive following the addition of the new RAID member:

$ ls /mnt/raid1
Music  lost+found  Images Films Misc

Thanks, but this tutorial assumes only one disc has failed, but in my configuration all discs have U_ and none has UU. I edited the post with more info. — captain lizard, Dec 21 '16 at 07:14
http://aplawrence.com/Linux/rebuildraid.html feels closer to your scenario ? — Tolsadus, Dec 21 '16 at 07:18
If you're running zfs, I really recommend reading http://docs.oracle.com/cd/E19253-01/819-5461/gamno/index.html — Tolsadus, Oct 07 '17 at 19:01

score 2 · Answer 2 · edited Dec 22 '16 at 13:07

I solved my own issue. I did not understand that mdX are partitions and sda and sdb are actual physical discs. So sdb was missing, so I needed to add it back. I executed the following commands:

sgdisk -R /dev/sdb /dev/sda
sgdisk -G /dev/sdb
mdadm /dev/md0 -a /dev/sdb1
mdadm /dev/md1 -a /dev/sdb2
mdadm /dev/md2 -a /dev/sdb3
mdadm /dev/md3 -a /dev/sdb4

After a day of resync (can be observed with)

cat /proc/mdstat

What I see today looks good again:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md3 : active raid1 sdb4[2] sda4[0]
      1839089920 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdb2[2] sda2[0]
      523968 blocks super 1.2 [2/2] [UU]

md2 : active raid1 sdb3[2] sda3[0]
      1073610560 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sdb1[2] sda1[0]
      16768896 blocks super 1.2 [2/2] [UU]

Vikelidis Kostas · Answer 3 · 2016-12-21T10:57:58.403

1

Before adding the drive as suggested by @Tolsadus its a good idea to check your drive health using smartmontools by executing short and long tests:

Initially you have to ensure that the hard disk supports SMART and is enabled:

smartctl -i /dev/sdb

short test:

smartctl -t short /dev/sdb

long test:

smartctl -l selftest /dev/sdb

edited Dec 21 '16 at 10:57

answered Dec 21 '16 at 10:49

Vikelidis Kostas

927
1
6
15

I've seen this only now. Can I possibly run these tests AFTER I added sdb? Everything seems fine right now. – captain lizard Dec 22 '16 at 13:06
Yes you can this after adding the disk – Vikelidis Kostas Dec 22 '16 at 13:12
I assume this is what their support team did to check the disks in the first place, but I agree that it is a good idea to run the tests again yourself. – Kevin Aug 05 '20 at 13:23

How to re-add a drive when RAID is in degraded mode?

3 Answers3