I have:
Openfiler SAN ML370 G5 Smart Array 6400 slot 1 Array B Logical Drive 2 is a RAID5 array of 6 x 148GB 10k hot plug drives making 680GB no spare
Last Friday, the power went out, this machine was just plugged in to the wall and it went down hard. When it came back up, drive 1 and 4 of 0-5 drive changed to red flashing Fault light. The chart from the array guide shows that as "predictive failure has been received for this drive, replace as soon as possible". At the commandline hpacucli utility reports the same message; Predictive Failure. The activity lights flash normally. The fault chart says the drive hasn't 'failed' until the fault LED is on solid.
During all of this and now a week later the system stays up and no users reported any problems so far - all ESX hosts/VMs are using this SAN and are still working fine and I manually made a backup of everything on the array and new drives showed up today. So I can try a few things without too much effort, but I sure would like to just replace the drives and have rebuilding work if I'm careful.
Normally I would kind of assume that with just a predictive failure that I could get away with replacing them one at a time, letting them rebuild one at a time and be fine BUT when I run hpacucli I get the following output on the LD
Array: B Interface Type: Parallel SCSI Unused Space: 0 MB Status: OK
Logical Drive: 2
Size: 683.6 GB
Fault Tolerance: RAID 5
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Stripe Size: 64 KB
Status: OK
Array Accelerator: Enabled
Parity Initialization Status: Initialization Failed
Unique Identifier: 600508B100104B39535153303250000F
Disk Name: /dev/cciss/c0d1
Mount Points: None
Logical Drive Label: A01E9878P57820K9SQS02PBE24
So the Status is OK but the Parity Initialization is what has me spooked there. Any guidance on a procedure to have a successful rebuild appreciated - or advice along the lines of "all data is suspect now anyways, just replace the bad drives, make a new array out of it and restore since you have backup" are fine also. I get that it's a risk no matter what. Should I restart before I attempt to replace anything?
Full hpacucli output at the bottom.
Seems like if that Predictive Failure is just SMART errors piling up, it would still have parity and rebuild, just maybe slowly?
Many thanks for any guidance, Peace!
---full hpacucli---
Array: B Interface Type: Parallel SCSI Unused Space: 0 MB Status: OK
Logical Drive: 2
Size: 683.6 GB
Fault Tolerance: RAID 5
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Stripe Size: 64 KB
Status: OK
Array Accelerator: Enabled
Parity Initialization Status: Initialization Failed
Unique Identifier: 600508B100104B39535153303250000F
Disk Name: /dev/cciss/c0d1
Mount Points: None
Logical Drive Label: A01E9878P57820K9SQS02PBE24
physicaldrive 1:0
SCSI Bus: 1
SCSI ID: 0
Status: OK
Drive Type: Data Drive
Interface Type: Parallel SCSI
Transfer Mode: Ultra 3 Wide
Size: 146.8 GB
Transfer Speed: 160 MB/Sec
Rotational Speed: 10000
Firmware Revision: HPB8
Serial Number: 3HY83F3Y00007442557Q
Model: COMPAQ BD14685A26
physicaldrive 1:1
SCSI Bus: 1
SCSI ID: 1
Status: Predictive Failure
Drive Type: Data Drive
Interface Type: Parallel SCSI
Transfer Mode: Ultra 3 Wide
Size: 146.8 GB
Transfer Speed: 160 MB/Sec
Rotational Speed: 10000
Firmware Revision: HPB8
Serial Number: 3HY8393700007345XU2M
Model: COMPAQ BD14685A26
physicaldrive 1:2
SCSI Bus: 1
SCSI ID: 2
Status: OK
Drive Type: Data Drive
Interface Type: Parallel SCSI
Transfer Mode: Ultra 3 Wide
Size: 146.8 GB
Transfer Speed: 160 MB/Sec
Rotational Speed: 10000
Firmware Revision: HPB8
Serial Number: 3HY9NWGY00007524BFV1
Model: COMPAQ BD14685A26
physicaldrive 1:3
SCSI Bus: 1
SCSI ID: 3
Status: OK
Drive Type: Data Drive
Interface Type: Parallel SCSI
Transfer Mode: Ultra 3 Wide
Size: 146.8 GB
Transfer Speed: 160 MB/Sec
Rotational Speed: 10000
Firmware Revision: HPB8
Serial Number: 3HY9PA1N00007523W3DP
Model: COMPAQ BD14685A26
physicaldrive 1:4
SCSI Bus: 1
SCSI ID: 4
Status: Predictive Failure
Drive Type: Data Drive
Interface Type: Parallel SCSI
Transfer Mode: Ultra 3 Wide
Size: 146.8 GB
Transfer Speed: 160 MB/Sec
Rotational Speed: 10000
Firmware Revision: HPB8
Serial Number: 3HY72WR9000075216UNS
Model: COMPAQ BD14685A26
physicaldrive 1:5
SCSI Bus: 1
SCSI ID: 5
Status: OK
Drive Type: Data Drive
Interface Type: Parallel SCSI
Transfer Mode: Ultra 3 Wide
Size: 146.8 GB
Transfer Speed: 160 MB/Sec
Rotational Speed: 10000
Firmware Revision: HPB8
Serial Number: 3HY9NT3F000075231R9V
Model: COMPAQ BD14685A26