I think our HDD is failing. Before I dump all the relevant logs: don't worry, everything is backed up daily, HDD can be replaced. I just need to confirm my suspicions. HDD is in raid so there is another disk and (as you can see) this server is online 24/7 for past 2+ years.
Smartctl reported Error 348 reported at 19677h which match error in syslog. Same thing for 349-351.
This drive is dying, right?
Logs:
Nov 9 02:50:04 XXXXX kernel: [21716010.818934] ata1.00: exception Emask 0x0 SAct 0x3ffff SErr 0x0 action 0x0
Nov 9 02:50:04 XXXXX kernel: [21716010.862259] ata1.00: irq_stat 0x40000008
Nov 9 02:50:04 XXXXX kernel: [21716010.904504] ata1.00: failed command: READ FPDMA QUEUED
Nov 9 02:50:04 XXXXX kernel: [21716010.946338] ata1.00: cmd 60/80:00:00:84:49/00:00:0b:00:00/40 tag 0 ncq 65536 in
Nov 9 02:50:04 XXXXX kernel: [21716010.946338] res 41/40:80:78:84:49/00:00:0b:00:00/00 Emask 0x409 (media error) <F>
Nov 9 02:50:04 XXXXX kernel: [21716011.110653] ata1.00: status: { DRDY ERR }
Nov 9 02:50:04 XXXXX kernel: [21716011.151079] ata1.00: error: { UNC }
Nov 9 02:50:04 XXXXX kernel: [21716011.324910] ata1.00: configured for UDMA/133
Nov 9 02:50:04 XXXXX kernel: [21716011.324938] sd 0:0:0:0: [sda] Unhandled sense code
Nov 9 02:50:04 XXXXX kernel: [21716011.324941] sd 0:0:0:0: [sda]
Nov 9 02:50:04 XXXXX kernel: [21716011.324943] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Nov 9 02:50:04 XXXXX kernel: [21716011.324944] sd 0:0:0:0: [sda]
Nov 9 02:50:04 XXXXX kernel: [21716011.324946] Sense Key : Medium Error [current] [descriptor]
Nov 9 02:50:04 XXXXX kernel: [21716011.324949] Descriptor sense data with sense descriptors (in hex):
Nov 9 02:50:04 XXXXX kernel: [21716011.324951] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Nov 9 02:50:04 XXXXX kernel: [21716011.324957] 0b 49 84 78
Nov 9 02:50:04 XXXXX kernel: [21716011.324960] sd 0:0:0:0: [sda]
Nov 9 02:50:04 XXXXX kernel: [21716011.324963] Add. Sense: Unrecovered read error - auto reallocate failed
Nov 9 02:50:04 XXXXX kernel: [21716011.324965] sd 0:0:0:0: [sda] CDB:
Nov 9 02:50:04 XXXXX kernel: [21716011.324967] Read(10): 28 00 0b 49 84 00 00 00 80 00
Nov 9 02:50:04 XXXXX kernel: [21716011.324973] end_request: I/O error, dev sda, sector 189367416
Nov 9 02:50:04 XXXXX kernel: [21716011.364405] ata1: EH complete
Nov 9 02:50:24 XXXXX kernel: [21716031.325428] ata1.00: exception Emask 0x0 SAct 0x1e000 SErr 0x0 action 0x0
Nov 9 02:50:24 XXXXX kernel: [21716031.367450] ata1.00: irq_stat 0x40000008
Nov 9 02:50:24 XXXXX kernel: [21716031.408335] ata1.00: failed command: READ FPDMA QUEUED
Nov 9 02:50:24 XXXXX kernel: [21716031.448762] ata1.00: cmd 60/80:68:00:86:49/00:00:0b:00:00/40 tag 13 ncq 65536 in
Nov 9 02:50:24 XXXXX kernel: [21716031.448762] res 41/40:80:18:86:49/00:00:0b:00:00/00 Emask 0x409 (media error) <F>
Nov 9 02:50:24 XXXXX kernel: [21716031.607084] ata1.00: status: { DRDY ERR }
Nov 9 02:50:24 XXXXX kernel: [21716031.645924] ata1.00: error: { UNC }
Nov 9 02:50:25 XXXXX kernel: [21716031.967600] ata1.00: configured for UDMA/133
Nov 9 02:50:25 XXXXX kernel: [21716031.967616] sd 0:0:0:0: [sda] Unhandled sense code
Nov 9 02:50:25 XXXXX kernel: [21716031.967619] sd 0:0:0:0: [sda]
Nov 9 02:50:25 XXXXX kernel: [21716031.967621] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Nov 9 02:50:25 XXXXX kernel: [21716031.967624] sd 0:0:0:0: [sda]
Nov 9 02:50:25 XXXXX kernel: [21716031.967626] Sense Key : Medium Error [current] [descriptor]
Nov 9 02:50:25 XXXXX kernel: [21716031.967629] Descriptor sense data with sense descriptors (in hex):
Nov 9 02:50:25 XXXXX kernel: [21716031.967631] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Nov 9 02:50:25 XXXXX kernel: [21716031.967640] 0b 49 86 18
Nov 9 02:50:25 XXXXX kernel: [21716031.967644] sd 0:0:0:0: [sda]
Nov 9 02:50:25 XXXXX kernel: [21716031.967647] Add. Sense: Unrecovered read error - auto reallocate failed
Nov 9 02:50:25 XXXXX kernel: [21716031.967650] sd 0:0:0:0: [sda] CDB:
Nov 9 02:50:25 XXXXX kernel: [21716031.967652] Read(10): 28 00 0b 49 86 00 00 00 80 00
Nov 9 02:50:25 XXXXX kernel: [21716031.967660] end_request: I/O error, dev sda, sector 189367832
Nov 9 02:50:25 XXXXX kernel: [21716032.005452] ata1: EH complete
Nov 9 02:50:41 XXXXX kernel: [21716048.237709] ata1.00: exception Emask 0x0 SAct 0x380 SErr 0x0 action 0x0
Nov 9 02:50:41 XXXXX kernel: [21716048.272985] ata1.00: irq_stat 0x40000008
Nov 9 02:50:41 XXXXX kernel: [21716048.307238] ata1.00: failed command: READ FPDMA QUEUED
Nov 9 02:50:41 XXXXX kernel: [21716048.341080] ata1.00: cmd 60/00:38:00:92:49/04:00:0b:00:00/40 tag 7 ncq 524288 in
Nov 9 02:50:41 XXXXX kernel: [21716048.341080] res 41/40:00:08:95:49/00:04:0b:00:00/00 Emask 0x409 (media error) <F>
Nov 9 02:50:41 XXXXX kernel: [21716048.473214] ata1.00: status: { DRDY ERR }
Nov 9 02:50:41 XXXXX kernel: [21716048.505636] ata1.00: error: { UNC }
Nov 9 02:50:41 XXXXX kernel: [21716048.572423] ata1.00: configured for UDMA/133
Nov 9 02:50:41 XXXXX kernel: [21716048.572453] sd 0:0:0:0: [sda] Unhandled sense code
Nov 9 02:50:41 XXXXX kernel: [21716048.572457] sd 0:0:0:0: [sda]
Nov 9 02:50:41 XXXXX kernel: [21716048.572459] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Nov 9 02:50:41 XXXXX kernel: [21716048.572462] sd 0:0:0:0: [sda]
Nov 9 02:50:41 XXXXX kernel: [21716048.572464] Sense Key : Medium Error [current] [descriptor]
Nov 9 02:50:41 XXXXX kernel: [21716048.572467] Descriptor sense data with sense descriptors (in hex):
Nov 9 02:50:41 XXXXX kernel: [21716048.572469] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Nov 9 02:50:41 XXXXX kernel: [21716048.572478] 0b 49 95 08
Nov 9 02:50:41 XXXXX kernel: [21716048.572482] sd 0:0:0:0: [sda]
Nov 9 02:50:41 XXXXX kernel: [21716048.572485] Add. Sense: Unrecovered read error - auto reallocate failed
Nov 9 02:50:41 XXXXX kernel: [21716048.572488] sd 0:0:0:0: [sda] CDB:
Nov 9 02:50:41 XXXXX kernel: [21716048.572489] Read(10): 28 00 0b 49 92 00 00 04 00 00
Nov 9 02:50:41 XXXXX kernel: [21716048.572498] end_request: I/O error, dev sda, sector 189371656
Nov 9 02:50:41 XXXXX kernel: [21716048.603930] ata1: EH complete
After seeing that did smartctl -a /dev/sda . Here is the log:
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-36-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST1500DM003-9YN16G
Serial Number: W2F0B7ZT
LU WWN Device Id: 5 000c50 052c400c7
Firmware Version: CC4C
User Capacity: 1,500,301,910,016 bytes [1.50 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Nov 9 21:38:42 2016 CET
==> WARNING: A firmware update for this drive is available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 625) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 209) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 107 091 006 Pre-fail Always - 13530088
3 Spin_Up_Time 0x0003 095 095 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 5
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 082 060 030 Pre-fail Always - 169205210
9 Power_On_Hours 0x0032 078 078 000 Old_age Always - 19694
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 5
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 001 001 000 Old_age Always - 543
188 Command_Timeout 0x0032 100 099 000 Old_age Always - 5 5 5
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 053 041 045 Old_age Always In_the_past 47 (7 6 59 27 0)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 4
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 5
194 Temperature_Celsius 0x0022 047 059 000 Old_age Always - 47 (0 27 0 0 0)
197 Current_Pending_Sector 0x0012 100 099 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 099 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 18785h+58m+38.071s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 114597464908825
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 161059093725236
SMART Error Log Version: 1
ATA Error Count: 351 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 351 occurred at disk power-on lifetime: 19688 hours (820 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: WP at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 08 ff ff ff 4f 00 24d+23:45:23.550 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 24d+23:45:23.528 WRITE FPDMA QUEUED
61 00 28 ff ff ff 4f 00 24d+23:45:23.528 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 24d+23:45:23.516 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 24d+23:45:23.516 WRITE FPDMA QUEUED
Error 350 occurred at disk power-on lifetime: 19688 hours (820 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: WP at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 58 ff ff ff 4f 00 24d+23:45:18.898 WRITE FPDMA QUEUED
60 00 08 ff ff ff 4f 00 24d+23:45:18.299 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 24d+23:45:18.299 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 24d+23:45:18.299 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 24d+23:45:18.298 READ FPDMA QUEUED
Error 349 occurred at disk power-on lifetime: 19688 hours (820 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 00 ff ff ff 4f 00 24d+23:45:15.382 READ FPDMA QUEUED
60 00 80 ff ff ff 4f 00 24d+23:45:14.302 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 24d+23:45:14.300 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 24d+23:45:14.297 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 24d+23:45:14.295 READ FPDMA QUEUED
Error 348 occurred at disk power-on lifetime: 19677 hours (819 days + 21 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 ff ff ff 4f 00 24d+13:00:41.477 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 24d+13:00:41.477 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 24d+13:00:41.477 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 24d+13:00:41.476 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 24d+13:00:41.476 READ FPDMA QUEUED
Error 347 occurred at disk power-on lifetime: 19675 hours (819 days + 19 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 08 95 49 0b Error: WP at LBA = 0x0b499508 = 189371656
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 08 ff ff ff 4f 00 24d+11:11:11.868 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 24d+11:11:11.867 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 24d+11:11:11.867 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 24d+11:11:11.867 WRITE FPDMA QUEUED
61 00 18 ff ff ff 4f 00 24d+11:11:11.867 WRITE FPDMA QUEUED
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 1291 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
enter code here
