1

I have a desktop running as a ubuntu server at another office. Lately its been shutting itself down once in a while, and I'm a bit unsure how to diagnose this. The syslog looks like this:

May 20 15:42:35 hostname sensord: Chip: coretemp-isa-0000
May 20 15:42:35 hostname sensord: Adapter: ISA adapter
May 20 15:42:35 hostname sensord:   Core 0: 67.0 C
May 20 15:42:35 hostname sensord:   Core 1: 66.0 C
May 20 15:42:35 hostname sensord:   Core 2: 61.0 C
May 20 15:42:35 hostname sensord:   Core 3: 58.0 C
May 20 16:04:16 hostname kernel: [ 5243.049529] CPU0: Core temperature above threshold, cpu clock throttled (total events = 1)
May 20 16:04:16 hostname kernel: [ 5243.050011] CPU0: Core temperature/speed normal
May 20 16:05:48 hostname kernel: [ 5335.083540] CPU2: Core temperature above threshold, cpu clock throttled (total events = 1)
May 20 16:05:48 hostname kernel: [ 5335.084028] CPU2: Core temperature/speed normal
May 21 16:06:52 hostname kernel: [ 5399.816039] mce: [Hardware Error]: Machine check events logged

At first i suspected a broken fan or something thermal, and activated sensord. But the temperatures seems stable over time.

Edit: I've install mcelog and the deamon is running. Pretty much waiting for it to happen again to see if the mcelog makes any sense.

Update

The mcelog indicates that it's a thermal issue, I have logs like the one below which match with the times of the Gitlab server backup cron job.

MCE 0
CPU 0 THERMAL EVENT TSC 16ec0aadec3a0
TIME 1401260314 Wed May 28 08:58:34 2014
Processor 0 heated above trip temperature. Throttling enabled.
Please check your system cooling. Performance will be impacted
STATUS 88020003 MCGSTATUS 0
MCGCAP 806 APICID 0 SOCKETID 0
CPUID Vendor Intel Family 6 Model 15
Hardware event. This is not a software error.

I've also done some testing today on stressing the system by stress -c 4 -i 1 -m 1 -t 120 and I very quickly reach 100 C on CPU temp.

coretemp-isa-0000
Adapter: ISA adapter
Core 0:      +100.0°C  (high = +84.0°C, crit = +100.0°C)
Core 1:       +96.0°C  (high = +84.0°C, crit = +100.0°C)
Core 2:       +85.0°C  (high = +84.0°C, crit = +100.0°C)
Core 3:       +79.0°C  (high = +84.0°C, crit = +100.0°C)

I suspect that the heatsink isn't properly mounted, and I will check this when I find the time to.

Solution

I'll check the heatpaste and sink of the cpu, as a quick fix.

I got hold of a used Dell PowerEdge R200 to replace this server, and I will try to get set it up next week. Thank you very much for the advice.

Trygve
  • 187
  • 1
  • 7

2 Answers2

1

The system is telling you it's logging hardware errors, have you looked at them?

May 21 16:06:52 hostname kernel: [ 5399.816039] mce: [Hardware Error]: Machine check events logged

Reference: Ubuntu man page for mcelog.

  • I didn't have mcelog installed, and it was a cold reset so it's gone. I've installed mcelog and I have the deamon running now, so I guess i have to wait for it to happen again to learn mroe – Trygve May 22 '14 at 12:47
  • I'd install it and wait for the next error to happen. My money is on bad RAM. –  May 22 '14 at 12:48
1

You can probably diagnose this... Logs indicate a thermal issue. CPU paste, heatsink, cooling, etc.

But here's the Pro answer:
What's the replacement cost? Is the impact of unexpected downtime worth enough to the end/business-users to just replace the setup with known-good server-class hardware?

Is the outage time needed to do a proper diagnosis of the hardware going to negatively impact users?

If so, replace it...

If not, troubleshoot... Try testing RAM as mentioned earlier. Also, see if you can induce the bad behavior with the stress utility or with a diagnostic Live CD.

ewwhite
  • 194,921
  • 91
  • 434
  • 799