Last week we had a series of power failures, which unfortunately lead to HDD4 dying in our ReadyNAS drive. The hard disk was swapped out, and the ReadyNAS automatically started rebuilding the array, stating it would take 2 hours to complete.
However, after approximately 47 minutes, HDD3 appears to have failed. The ReadyNAS then reported that the array was in a degraded state due to the failure of 3, and I received errors stating the volumes were no longer available.
I was able to SSH into the server, and using mdadm --force it is now up and running, however, is still showing that only 12&4 are used in the array. 3 is still showing healthy.
There appear to be 3 partitions, which are then added to a RAID5 array. /dev/sdc is included in two of them, but not the third:
mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Feb 20 14:19:18 2014
Raid Level : raid1
Array Size : 4193268 (4.00 GiB 4.29 GB)
Used Dev Size : 4193268 (4.00 GiB 4.29 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Tue Nov 1 16:11:41 2016
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Name : 30469A7FFABC:0
UUID : 386349a8:baa621d2:4d938e5e:3aba33db
Events : 7417
Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
2 8 33 2 active sync /dev/sdc1
4 8 49 3 active sync /dev/sdd1
mdadm --detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Thu Feb 20 14:19:18 2014
Raid Level : raid6
Array Size : 1048448 (1024.05 MiB 1073.61 MB)
Used Dev Size : 524224 (512.02 MiB 536.81 MB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Sun Oct 30 21:47:42 2016
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : 30469A7FFABC:1
UUID : 8ec1d475:bc4293be:63b1e929:a1669431
Events : 77
Number Major Minor RaidDevice State
0 8 2 0 active sync /dev/sda2
1 8 18 1 active sync /dev/sdb2
2 8 34 2 active sync /dev/sdc2
4 8 50 3 active sync /dev/sdd2
mdadm --detail /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Thu Feb 20 14:19:18 2014
Raid Level : raid5
Array Size : 2916122112 (2781.03 GiB 2986.11 GB)
Used Dev Size : 972040704 (927.01 GiB 995.37 GB)
Raid Devices : 4
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Tue Nov 1 16:09:21 2016
State : active, degraded
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : 30469A7FFABC:2
UUID : 59f98978:f215b5ec:2e2d93b5:19f17e1f
Events : 9559
Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 8 19 1 active sync /dev/sdb3
2 0 0 2 removed
4 8 51 3 active sync /dev/sdd3
As you can see, /dev/sdc3 is missing from the main data array.
I have tried to add it, but it will not do so - it tries to add it as a 5th disk.
I've tried to remove it from the array:
mdadm --remove /dev/md2 /dev/sdc3
mdadm: hot remove failed for /dev/sdc3: No such device or address
Any suggestions on how I can re-add /dev/sdc3?
