Is my SSD drive failing?

3

1

I'm using a fairly new SSD (bought 5 months ago), and I recently experienced a read failure. Here is what the kernel reported:

Jul  1 01:02:44 tryphon kernel: [1554891.463590] ata2.00: exception Emask 0x0 SAct 0x1000000 SErr 0x0 action 0x0
Jul  1 01:02:44 tryphon kernel: [1554891.478096] ata2.00: irq_stat 0x40000008
Jul  1 01:02:44 tryphon kernel: [1554891.486388] ata2.00: failed command: READ FPDMA QUEUED
Jul  1 01:02:44 tryphon kernel: [1554891.497141] ata2.00: cmd 60/80:c0:00:18:2c/05:00:02:00:00/40 tag 24 ncq dma 720896 in
Jul  1 01:02:44 tryphon kernel: [1554891.497141]          res 41/40:80:00:18:2c/00:05:02:00:00/40 Emask 0x409 (media error) <F>
Jul  1 01:02:44 tryphon kernel: [1554891.530266] ata2.00: status: { DRDY ERR }
Jul  1 01:02:44 tryphon kernel: [1554891.538714] ata2.00: error: { UNC }
Jul  1 01:02:44 tryphon kernel: [1554891.586520] ata2.00: configured for UDMA/133
Jul  1 01:02:44 tryphon kernel: [1554891.586576] sd 1:0:0:0: [sdb] tag#24 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul  1 01:02:44 tryphon kernel: [1554891.586581] sd 1:0:0:0: [sdb] tag#24 Sense Key : Medium Error [current]
Jul  1 01:02:44 tryphon kernel: [1554891.586585] sd 1:0:0:0: [sdb] tag#24 Add. Sense: Unrecovered read error - auto reallocate failed
Jul  1 01:02:44 tryphon kernel: [1554891.586590] sd 1:0:0:0: [sdb] tag#24 CDB: Read(10) 28 00 02 2c 18 00 00 05 80 00
Jul  1 01:02:44 tryphon kernel: [1554891.586593] blk_update_request: I/O error, dev sdb, sector 36444160

A few minutes later, the smartd daemon sent an email to root:

Subject: SMART error (OfflineUncorrectableSector) detected on host: tryphon

This message was generated by the smartd daemon running on:

   host name:  tryphon
   DNS domain: home

The following warning/error was logged by the smartd daemon:

Device: /dev/sdb [SAT], 1 Offline uncorrectable sectors

Device info:
LDLC, S/N:07072217A1171, FW:Q0601B, 120 GB

For details see host's SYSLOG.

You can also use the smartctl utility for further investigation.
Another message will be sent in 24 hours if the problem persists.

Since it is used in a RAID 1 configuration with an HDD, this is not dramatically bad, but it still sounds bad.

Anyway, I wanted to check the readability of my SSD with dd if=/dev/sdb of=/dev/null and experienced more read failures. Here are the kernel logs for the next read errors:

Jul  3 19:28:24 tryphon kernel: [ 4398.319485] ata2.00: exception Emask 0x0 SAct 0x180 SErr 0x0 action 0x0
Jul  3 19:28:24 tryphon kernel: [ 4398.332971] ata2.00: irq_stat 0x40000008
Jul  3 19:28:24 tryphon kernel: [ 4398.340961] ata2.00: failed command: READ FPDMA QUEUED
Jul  3 19:28:24 tryphon kernel: [ 4398.351416] ata2.00: cmd 60/00:38:08:23:80/01:00:00:00:00/40 tag 7 ncq dma 131072 in
Jul  3 19:28:24 tryphon kernel: [ 4398.351416]          res 41/40:00:08:23:80/00:01:00:00:00/40 Emask 0x409 (media error) <F>
Jul  3 19:28:24 tryphon kernel: [ 4398.383720] ata2.00: status: { DRDY ERR }
Jul  3 19:28:24 tryphon kernel: [ 4398.391853] ata2.00: error: { UNC }
Jul  3 19:28:24 tryphon kernel: [ 4398.439236] ata2.00: configured for UDMA/133
Jul  3 19:28:24 tryphon kernel: [ 4398.439258] sd 1:0:0:0: [sdb] tag#7 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul  3 19:28:24 tryphon kernel: [ 4398.439261] sd 1:0:0:0: [sdb] tag#7 Sense Key : Medium Error [current] 
Jul  3 19:28:24 tryphon kernel: [ 4398.439264] sd 1:0:0:0: [sdb] tag#7 Add. Sense: Unrecovered read error - auto reallocate failed
Jul  3 19:28:24 tryphon kernel: [ 4398.439268] sd 1:0:0:0: [sdb] tag#7 CDB: Read(10) 28 00 00 80 23 08 00 01 00 00
Jul  3 19:28:24 tryphon kernel: [ 4398.439270] blk_update_request: I/O error, dev sdb, sector 8397576
Jul  3 19:28:24 tryphon kernel: [ 4398.451841] ata2: EH complete
Jul  3 19:28:24 tryphon kernel: [ 4398.831494] ata2.00: exception Emask 0x0 SAct 0x1000 SErr 0x0 action 0x0
Jul  3 19:28:24 tryphon kernel: [ 4398.845137] ata2.00: irq_stat 0x40000008
Jul  3 19:28:24 tryphon kernel: [ 4398.853069] ata2.00: failed command: READ FPDMA QUEUED
Jul  3 19:28:24 tryphon kernel: [ 4398.863461] ata2.00: cmd 60/08:60:78:23:80/00:00:00:00:00/40 tag 12 ncq dma 4096 in
Jul  3 19:28:24 tryphon kernel: [ 4398.863461]          res 41/40:08:78:23:80/00:00:00:00:00/40 Emask 0x409 (media error) <F>
Jul  3 19:28:24 tryphon kernel: [ 4398.895577] ata2.00: status: { DRDY ERR }
Jul  3 19:28:24 tryphon kernel: [ 4398.903692] ata2.00: error: { UNC }
Jul  3 19:28:24 tryphon kernel: [ 4398.951089] ata2.00: configured for UDMA/133
Jul  3 19:28:24 tryphon kernel: [ 4398.951111] sd 1:0:0:0: [sdb] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul  3 19:28:24 tryphon kernel: [ 4398.951116] sd 1:0:0:0: [sdb] tag#12 Sense Key : Medium Error [current] 
Jul  3 19:28:24 tryphon kernel: [ 4398.951119] sd 1:0:0:0: [sdb] tag#12 Add. Sense: Unrecovered read error - auto reallocate failed
Jul  3 19:28:24 tryphon kernel: [ 4398.951125] sd 1:0:0:0: [sdb] tag#12 CDB: Read(10) 28 00 00 80 23 78 00 00 08 00
Jul  3 19:28:24 tryphon kernel: [ 4398.951128] blk_update_request: I/O error, dev sdb, sector 8397688
Jul  3 19:28:24 tryphon kernel: [ 4398.963674] Buffer I/O error on dev sdb, logical block 1049711, async page read
Jul  3 19:28:24 tryphon kernel: [ 4398.978559] ata2: EH complete
Jul  3 19:28:24 tryphon kernel: [ 4399.051447] ata2.00: exception Emask 0x0 SAct 0x80000 SErr 0x0 action 0x0
Jul  3 19:28:24 tryphon kernel: [ 4399.065213] ata2.00: irq_stat 0x40000008
Jul  3 19:28:24 tryphon kernel: [ 4399.073159] ata2.00: failed command: READ FPDMA QUEUED
Jul  3 19:28:24 tryphon kernel: [ 4399.083555] ata2.00: cmd 60/08:98:78:23:80/00:00:00:00:00/40 tag 19 ncq dma 4096 in
Jul  3 19:28:24 tryphon kernel: [ 4399.083555]          res 41/40:08:78:23:80/00:00:00:00:00/40 Emask 0x409 (media error) <F>
Jul  3 19:28:24 tryphon kernel: [ 4399.115655] ata2.00: status: { DRDY ERR }
Jul  3 19:28:24 tryphon kernel: [ 4399.123792] ata2.00: error: { UNC }
Jul  3 19:28:24 tryphon kernel: [ 4399.171197] ata2.00: configured for UDMA/133
Jul  3 19:28:24 tryphon kernel: [ 4399.171223] sd 1:0:0:0: [sdb] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul  3 19:28:24 tryphon kernel: [ 4399.171228] sd 1:0:0:0: [sdb] tag#19 Sense Key : Medium Error [current] 
Jul  3 19:28:24 tryphon kernel: [ 4399.171232] sd 1:0:0:0: [sdb] tag#19 Add. Sense: Unrecovered read error - auto reallocate failed
Jul  3 19:28:24 tryphon kernel: [ 4399.171237] sd 1:0:0:0: [sdb] tag#19 CDB: Read(10) 28 00 00 80 23 78 00 00 08 00
Jul  3 19:28:24 tryphon kernel: [ 4399.171240] blk_update_request: I/O error, dev sdb, sector 8397688
Jul  3 19:28:24 tryphon kernel: [ 4399.183787] Buffer I/O error on dev sdb, logical block 1049711, async page read
Jul  3 19:28:24 tryphon kernel: [ 4399.198659] ata2: EH complete

and:

Jul  5 12:28:06 tryphon kernel: [    5.513650] ata2.00: exception Emask 0x0 SAct 0x4000 SErr 0x0 action 0x0
Jul  5 12:28:06 tryphon kernel: [    5.527221] ata2.00: irq_stat 0x40000008
Jul  5 12:28:06 tryphon kernel: [    5.535114] ata2.00: failed command: READ FPDMA QUEUED
Jul  5 12:28:06 tryphon kernel: [    5.545448] ata2.00: cmd 60/08:70:98:af:32/00:00:00:00:00/40 tag 14 ncq dma 4096 in
Jul  5 12:28:06 tryphon kernel: [    5.545448]          res 41/40:08:98:af:32/00:00:00:00:00/40 Emask 0x409 (media error) <F>
Jul  5 12:28:06 tryphon kernel: [    5.577373] ata2.00: status: { DRDY ERR }
Jul  5 12:28:06 tryphon kernel: [    5.585423] ata2.00: error: { UNC }
Jul  5 12:28:06 tryphon kernel: [    5.632752] ata2.00: configured for UDMA/133
Jul  5 12:28:06 tryphon kernel: [    5.632778] sd 1:0:0:0: [sdb] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul  5 12:28:06 tryphon kernel: [    5.632784] sd 1:0:0:0: [sdb] tag#14 Sense Key : Medium Error [current] 
Jul  5 12:28:06 tryphon kernel: [    5.632790] sd 1:0:0:0: [sdb] tag#14 Add. Sense: Unrecovered read error - auto reallocate failed
Jul  5 12:28:06 tryphon kernel: [    5.632797] sd 1:0:0:0: [sdb] tag#14 CDB: Read(10) 28 00 00 32 af 98 00 00 08 00
Jul  5 12:28:06 tryphon kernel: [    5.632801] blk_update_request: I/O error, dev sdb, sector 3321752
Jul  5 12:28:06 tryphon kernel: [    5.645296] md/raid1:md1: sdb3: rescheduling sector 173976
Jul  5 12:28:06 tryphon kernel: [    5.656333] ata2: EH complete
Jul  5 12:28:06 tryphon kernel: [    5.695829] md/raid1:md1: redirecting sector 173976 to other mirror: sda3

Then, I had a look at what the drive reports with the S.M.A.R.T. system. This is the result:

# smartctl -a /dev/sdb
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.9.0-0.bpo.6-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     LDLC
Serial Number:    07072217A1171
Firmware Version: Q0601B
User Capacity:    120,034,123,776 bytes [120 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Jul 11 00:43:45 2018 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)     Offline data collection activity
                                    was never started.
                                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)     The previous self-test routine completed
                                    without error or no self-test has ever 
                                    been run.
Total time to complete Offline 
data collection:            (  120) seconds.
Offline data collection
capabilities:                        (0x11) SMART execute Offline immediate.
                                    No Auto Offline data collection support.
                                    Suspend Offline collection upon new
                                    command.
                                    No Offline surface scan supported.
                                    Self-test supported.
                                    No Conveyance Self-test supported.
                                    No Selective Self-test supported.
SMART capabilities:            (0x0002)     Does not save SMART data before
                                    entering power-saving mode.
                                    Supports SMART auto save timer.
Error logging capability:        (0x01)     Error logging supported.
                                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   2) minutes.
Extended self-test routine
recommended polling time:    (  10) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x0032   100   100   050    Old_age   Always       -       0
  5 Reallocated_Sector_Ct   0x0032   100   100   050    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   050    Old_age   Always       -       3828
 12 Power_Cycle_Count       0x0032   100   100   050    Old_age   Always       -       18
160 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       6
161 Unknown_Attribute       0x0033   100   100   050    Pre-fail  Always       -       99
163 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       60
164 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       3268
165 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       2
166 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       1
167 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       1
168 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       1500
169 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       100
175 Program_Fail_Count_Chip 0x0032   100   100   050    Old_age   Always       -       0
176 Erase_Fail_Count_Chip   0x0032   100   100   050    Old_age   Always       -       0
177 Wear_Leveling_Count     0x0032   100   100   050    Old_age   Always       -       18
178 Used_Rsvd_Blk_Cnt_Chip  0x0032   100   100   050    Old_age   Always       -       1
181 Program_Fail_Cnt_Total  0x0032   100   100   050    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   050    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   050    Old_age   Always       -       7
194 Temperature_Celsius     0x0022   100   100   050    Old_age   Always       -       40
195 Hardware_ECC_Recovered  0x0032   100   100   050    Old_age   Always       -       11868576
196 Reallocated_Event_Count 0x0032   100   100   050    Old_age   Always       -       6
197 Current_Pending_Sector  0x0032   100   100   050    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0032   100   100   050    Old_age   Always       -       6
199 UDMA_CRC_Error_Count    0x0032   100   100   050    Old_age   Always       -       0
232 Available_Reservd_Space 0x0032   100   100   050    Old_age   Always       -       99
241 Total_LBAs_Written      0x0030   100   100   050    Old_age   Offline      -       10975
242 Total_LBAs_Read         0x0030   100   100   050    Old_age   Offline      -       29075
245 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       2904

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

Selective Self-tests/Logging not supported

As I understand it, the Offline_Uncorrectable attribute with value of 6 looks very bad since it seems to me that some data was lost 6 times. Am I missing something?

Also, the raw value of the Hardware_ECC_Recovered attribute looks huge, especially when compared to the Total_LBAs_Read attribute.

Should I worry about my SSD drive? Should I replace it?

For what it's worth, this is a TLC drive. I know they are less durable than the MLC (or the even better SLC) drives, but I did not expect data loss after only a few months.

user2233709

Posted 2018-07-09T00:50:53.147

Reputation: 163

Why do you mean you experienced a “read failure?” What do you mean it is in a RAID 1 configuration with an HDD? You can’t put two completely dissimilar drives in RAID 1 together and expect it to function properly. – Appleoddity – 2018-07-09T04:06:27.863

2Hardware_ECC_Recovered is likely several bytes crammed in a single word, and has to be decoded. Proper decoding depends on brand/model (no standard there). – xenoid – 2018-07-09T06:33:24.390

1

@Appleoddity I mean I experienced a read failure because my (linux) kernel reported it (short: blk_update_request: I/O error, dev sdb, sector 36444160). Yes, I do have a (software) RAID 1 array built on same-size partitions of the SSD and HDD. Why would you expect this not to function properly? Other use that, see this superuser question.

– user2233709 – 2018-07-09T09:14:46.973

At least judging from the SMART values, everything looks good (100). Offline_uncorrectable means errors found through an offline test run, probably the initial test run before the drive was shipped. Finding bad sectors in this is expected. OTOH, I've heard of SSDs which give trouble even though the SMART values are still good. Do you have details about the I/O error? Possibly a transport error? – dirkt – 2018-07-09T13:00:51.877

@dirkt I just added the kernel log about the first read failure. I am pretty confident that the Offline_Uncorrectable attribute was 0 before that read failure since: 1) smartd immediately complained 2) that attribute then was 1 3) it then raised after some new read failures. – user2233709 – 2018-07-09T22:55:29.647

Those attributes look mighty suspicious. There are some that simply do not apply to SSDs. Please provide the entire smartctl -a output. // Also, by RAIDing HDD + SSD, you’re throwing away much of the SSDs performance. – Daniel B – 2018-07-10T08:09:08.000

@DanielB I just replaced the output of smartctl -A with that of smartctl -a. As for “throwing away much of the SSDs performance”, that’s true for the write performance, but that’s wrong for the read performance if the HDD id added to the RAID array with the “write-mostly” option. – user2233709 – 2018-07-10T22:48:37.663

Hm, okay. That’s not terribly helpful either. What make and model is that even supposed to be? // How much data do you estimate you wrote to that SSD until now? – Daniel B – 2018-07-11T05:14:52.747

@DanielB The SSD is a “LDLC SSD F7 PLUS 120 GB”. LDLC is a french computer hardware reseller. I don’t think they build their own SSD, I’m pretty sure they only rebrand drives manufactured by some other company, but they won’t tell what other company.

– user2233709 – 2018-07-12T23:54:47.400

@DanielB As for the amount of data written, it’s hard to tell. Reading 40GB of data gives an increase of 1280 of the “Total_LBAs_Read” S.M.A.R.T. attribute. So I guess the “Total_LBAs_Written” data might decode to 10975×32MB ≈ 343GB. Doesn’t look unrealistic to me. – user2233709 – 2018-07-13T00:06:16.803

Answers

2

Partial answer:

You have 6 Offline_uncorrectable (which should occur during offline-tests) and 6 Reallocated_Event_Count, but zero Reallocated_Sector_Ct. I read this as "tried to reallocate a block during an offline test six times, but failed each time".

The error message in the kernel log also says "auto reallocate failed", though this is an online event, not an offline one.

So:

1) Use smartctl to look at the result of the offline-tests, in particular at what time they were run, and if errors occured.

2) Use smartctl to start another offline test (overnight etc.). Look at the results.

3) You mentioned "some new read failures" in the comment. What are the block numbers for them? Is it always the same block?

If it's a single block that for some reason can't be auto-reallocated, I'd consider marking that block as bad in the file system (most file systems have that feature). That's usually not done today because all harddrives auto-reallocate, both mechanical and SSD, but the feature exists for exactly that purpose.

Though I'd be more worried that the auto-reallocate failed; that may mean that the (invisible) block that contains the reallocation-meta-information is bad.

I'm not sure what to do about that; if the vendor of the SDD has a special tool for download where you can adjust the low-level stuff (some vendors do), this may help.

dirkt

Posted 2018-07-09T00:50:53.147

Reputation: 11 627

Thanks for that partial answer. I replaced the output of smartctl -A with that of smartctl -a. I think the No self-tests have been logged. part is the current answer to your first question. I also added the kernel logs for the next read errors. The sector 8397688 shows up twice; the other ones only once. I will try to start an offline test with smartctl -t offline /dev/sdb. As for the vendor, I don’t know the “real” vendor as my SSD is rebranded by a reseller. I asked the reseller if there is a dedicated diagnostic tool, and am waiting for their answer. – user2233709 – 2018-07-10T22:58:14.150

It looks like this SSD doesn't really have an offline scan ("no offline surface scan supported"), so Offline_uncorrectable may refer to some other way of collecting data. Repeated uncorrectable errors for different blocks are definitely a sign something is wrong. If the data on this disk is important: Replace it. – dirkt – 2018-07-11T05:58:43.937

The reseller replied there is no dedicated diagnostic tool for this drive, and they won’t tell me who manufactured it. Replacing it looks like the only viable option. Thanks for your help. – user2233709 – 2018-07-13T00:08:22.910

Asked: 2018-07-09T00:50:53.147

Viewed: 725 times

Active: 2018-07-10T22:45:28.887