Why is resync 10 times slower than anticipated?

Question

Two days ago, one of the Barracudas in my Synology (3 disks as RAID-5) triggered some "bad sector" warnings. No data lost. They're not many (62 over 24 hours, then none more) and according to SMART info, the drive is "just fine". But still, it is enough for me to replace the disk. Your mileage may vary, but for me, anything non-zero in terms of bad sector is NG.

So... thanks to a large online bookstore which also sells harddrives, I got same-size replacement disks (Ironwolf) literally over night.

First plugged one in the 4th slot, and changed the array type to RAID-6 while the old disk is still alive and kicking, to add some extra redundancy. Better be on the safe side, just in case. Once that's done, next step will be replacing the old disks one by one.

So... it's currently resyncing.

I've changed settings from "less impact" to "resync faster", which apparently meddles with IO priorities. The impact is very noticeable, trying to access a share is very, very slow now (but of course still works). That's fine, after all we want the resync to finish soon before something more drastic may happen. Still, disk usage is only 60-56% on each disk in resource monitor. Well, that's not so bad, I guess.

The new disk is able to sustain sequential 150 MB/s writes and the old disks shouldn't have any trouble delivering that while reading sequentialy (even moreso as there's three of them, cutting down necessary bandwidth). 60% of that is something around 90 MB/s. They're 4TB disks.

Let's be pessimistic and assume we only get 50 MB/s throughput total. So, that is 4*(1024*1024)/50 seconds to perform the resync, or just about a little more than 23 hours.

I've left the thing alone doing its work over night, and it is meanwhile running for 26 hours. Looking at the status window, it shows 11% complete.

Not like there's anything I could do about it anyway, but seriously... what's wrong? 11% after 26 hours means it's going to take almost two weeks. What the?

This is beyond my comprehension. Is there any technical reason why it would take that long?

Answer 1

You started with this:

and will eventually end up with this:

When you converted to RAID6 you've started a operation, to add a parity bit to every single disk in the array including the 'dying' disk. If this disk is dying it'll likely be very slow to write out any data, which may of been previously hidden by the other disks being fast to write data and the slow disk playing constant catch up.

Adding to the pain is that the fact this parity data is likely re-created for each disk as well, instead of just copying the existing parity data, meaning all the disks must be read - calculated - and then written back.

Then there's the fact data from each of the existing disks is now moved to the new disk and likely re-arranging of existing data on each of the existing disks, to speed up access.

All in all you're looking at upto 3x the reads and writes (for 4 drives) to move the data around for RAID6 layout, as opposed to copying and writing out one disk to another (1x read, 1xwrite).

Also, as well as all this work moving data about, you don't have a infinitely large room to load the data into while your working on it. The ram constraints will also slow down any processing, and as you've seen file accesses during this process slow it down as well. Finally, a lot of prebuilt NAS boxes have rather underpowered CPU's, which is generally fine for normal operation, such things as resyncs and architecture changes end up taking a long time.

This is why it takes so long.