Confirm disk is broken when it passes all diagnostics

Question

I have a system with a potentially broken disk, but the disk passes all manner of diagnostics. I have been unable to confirm that the disk is broken. What are my options?

I could just replace the disk, but because this situation is very similar to another more severe situation I have (long story), I'd like to actually make a proper diagnosis as opposed to randomly binning hardware.

The issue and history is this:

I had a Debian Linux PC (500 MHz P3) acting as router, nagios and munin.
It crashed every couple of weeks. No logs or dmesg could be obtained (because it's an old Compaq that only boots when you configure it as keyboardless, making connecting a keyboard later, once it's booted, impossible).
At the time, I just replaced the computer with another Compaq (P4 2.4 GHz) because I thought the hardware was faulty. However, it still crashed every couple of weeks.
the difference is that on this computer, I can still SSH into it. It gives all kinds of errors on hda.

I'd like to confirm that the disk is broken, but nothing I do confirms this:

SMART error logs shows no errors. Normally when a disk starts acting up, SMART my pass, but it still records a read-error in the error log.
SMART self-test (smartctl -t long /dev/sda) completes without errors.
re-allocated sector count (a tell-tale parameter) has been 31 all its life, even when the disk was still in use in my desktop PC years ago, and it still is. The figure never changed.
dd if=/dev/sda of=/dev/null bs=4096 passes with flying colors.

What else can I do to assess the health of the drive?

Again, this is not about making this router fully functional again, this is a disk forensic question, because it just so happens that I have another server that potentially has the same problem, and knowing the answer to this will possibly help me greatly.

For the record, below are logs and such.

This is the smartctl -a output:

smartctl 5.40 2010-07-12 r3124 [i686-pc-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.7 and 7200.7 Plus family
Device Model:     ST3120026A
Serial Number:    5JT1CLQM
Firmware Version: 3.06
User Capacity:    120,034,123,776 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   6
ATA Standard is:  ATA/ATAPI-6 T13 1410D revision 2
Local Time is:    Mon Jul  1 21:18:33 2013 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (  24) The self-test routine was aborted by
                    the host.
Total time to complete Offline 
data collection:         ( 430) seconds.
Offline data collection
capabilities:            (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    No General Purpose Logging support.
Short self-test routine 
recommended polling time:    (   1) minutes.
Extended self-test routine
recommended polling time:    (  85) minutes.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   050   046   006    Pre-fail  Always       -       47766662
  3 Spin_Up_Time            0x0003   097   096   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       10
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       31
  7 Seek_Error_Rate         0x000f   084   060   030    Pre-fail  Always       -       820305
  9 Power_On_Hours          0x0032   048   048   000    Old_age   Always       -       46373
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       605
194 Temperature_Celsius     0x0022   036   065   000    Old_age   Always       -       36
195 Hardware_ECC_Recovered  0x001a   050   046   000    Old_age   Always       -       47766662
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   196   000    Old_age   Always       -       6
200 Multi_Zone_Error_Rate   0x0000   100   253   000    Old_age   Offline      -       0
202 Data_Address_Mark_Errs  0x0032   100   253   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Aborted by host               80%     46361         -
# 2  Extended offline    Completed without error       00%     46358         -
# 3  Short offline       Completed without error       00%     12046         -
# 4  Extended offline    Completed without error       00%     10472         -
# 5  Short offline       Completed without error       00%     10471         -
# 6  Short offline       Completed without error       00%     10471         -
# 7  Short offline       Completed without error       00%      6770         -
# 8  Extended offline    Aborted by host               90%      5958         -
# 9  Extended offline    Aborted by host               90%      5951         -
#10  Short offline       Completed without error       00%      5024         -
#11  Extended offline    Aborted by host               80%      5024         -
#12  Short offline       Completed without error       00%      3697         -
#13  Short offline       Completed without error       00%       237         -
#14  Short offline       Completed without error       00%       145         -
#15  Short offline       Completed without error       00%        69         -
#16  Extended offline    Completed without error       00%        68         -
#17  Short offline       Completed without error       00%        66         -
#18  Short offline       Completed without error       00%        49         -
#19  Short offline       Completed without error       00%        29         -
#20  Short offline       Completed without error       00%        29         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

And this is the dmesg error when it has crashed (which repeats for a bunch of different sectors):

[1755091.211136] sd 0:0:0:0: [sda] Unhandled error code
[1755091.211144] sd 0:0:0:0: [sda] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[1755091.211151] sd 0:0:0:0: [sda] CDB: Read(10): 28 00 08 fe ad 38 00 00 08 00
[1755091.211166] end_request: I/O error, dev sda, sector 150908216

If this is at a business that depends on access to the internet for any operations that make money or are otherwise important, I wouldn't spend any time at all on it. I would replace the bloody thing, and probably with a dedicated network device on a support contract, because my time is valuable too (to my employer and/or clients). If there is a similar problem with another server that is an important machine, post *that* question. — mfinni, Jul 01 '13 at 19:40
I [posted](http://serverfault.com/questions/517533/3ware-raid6-array-sometimes-hanging-undetected-broken-disk) that question, no answer. And this router is not that crucial, but it is a test case. — Halfgaar, Jul 01 '13 at 19:52

Roman · Accepted Answer · 2013-07-01T21:53:17.053

You can't reliably.

Or rather, you have already done it with the options at your disposal.

As a study at google found out, failing disks do not necessarily show abnormal SMART values (The other way round however is more reliable: when they do, they will fail).

Keeping this aside for a moment, bear in mind that even though alot is standardized in computing, in reality there are bugs in both hard- and software, error margins which can accumulate, etc. The real world isn't perfect, and it's not unseen of hard disks not playing nice with particular controllers - and the other way round. Sometimes it's a question of a faulty firmware, sometimes some completely different system components not behaving, for example a sub-par PSU which barfs at particular load spikes. Or even temperature changes, age...the list could be expanded almost at will.

So, standard procedure here is to put the disk into a significantly different system configuration and re-run tests - but since you already have done so with the complete change of your system, you have correctly concluded that the disk must be at fault. (Unless you did not change everything else as you've told us - Cable/HBA comes to mind, in which case the assumption would not hold true).

Edit: I just realized that there is one option left; you can search if there are newer firmware revisions available for this disk drive than what's currently on your particular drive. If so, you may have a look at the change log pointing out possible problems in your case.

In conclusion, to establish with complete confidence (in this particular situation!) that the drive is misbehaving, you'll need to send it back to the manufacturer.

I didn't bring anything over to the new system, no cables, no HBA, nothing. I guess you're right. Replacing it is easy, it's just that I wish I could be more definitive. — Halfgaar, Jul 01 '13 at 20:03
I updated my answer a bit as to reflect two more options. However, don't spend too much time on it...disks really are commodity nowadays. Makes your life a whole bunch easier :) — Roman, Jul 01 '13 at 20:08

score 1 · Answer 2 · answered Jul 01 '13 at 19:46

Im thinking this is a bad controller. You can do a few more things to check out the disk as well as the controller...

Run 'badblocks' on the drive. This is similar to the 'dd' that you ran. Take another drive that has good SMART status and place it into the computer. If this disk gives you similar behavior then you know that it is hardware other than the disk that is giving you problems. In that case I would think that it is the controller. You did mention that you changed systems and that it was still giving you problems so, after all is said and done, I would still think that there had to be one common component that was causing the system instability. You can also look at:

bad cable (was the cable swapped to the second machine with the drive?)
bad configuration on the systems (are you setting up the system the same with different hw?)

I doubt it's the controller, because it's a very different machine. And yes, the machines are setup the same (but just default Debian 6). I just plugged the disk into the second computer. As for replacing the disk just to know what happens; the thing is, it can run fine for months, so it's very hard to confirm it works. It's easier to confirm it doesn't :) — Halfgaar, Jul 01 '13 at 19:56
At that point I would say that the disk is bad. The only thing I could suggest would be to replace the disk with one that you know works and run with it. If you get errors then you know it was not the disk. — Atari911, Jul 01 '13 at 20:13

score 0 · Answer 3 · answered Nov 13 '18 at 10:01

SF brought up this question because it was 'notable', so I didn't want to leave out new insights, namely diskscan. It reads the disk and graphs the latency of sectors, assuming broken sectors need retries.

Here is a the result of a Seagate disk I had lying around that passed all traditional tests:

You can see there are a few regions of very high latency, which obviously are a problem.

And another disk, a western digital:

Interestingly, after a wipe with dd, the SMART status was OK again: no pending or reallocated sectors. This was the subsequent rerun of diskscan:

So this WDC disk was good again, supposedly. I ran a smartctl -t long on it, and then it showed a read failure in the disk again.

Conclusion: diskscan helps, but of course, nothing is 100% reliable.

Confirm disk is broken when it passes all diagnostics

3 Answers3