Linux clock loses 10 minutes every week

Question

One of my linux server's clock loses 10 minutes every now and then, nearly every week. I update the time so it stays correct, and although it doesn't really bother me, i'd like to fix it.

I've been searching around a bit. Nothing can be responsible in the crontab, and i can't find any related message in the logs. Some people seem to use ntp to fix that kind of issue, but i'd prefer not to use an unecessary component on it.

Uname result : Linux unis-monitor 2.6.32-5-686 #1 SMP Mon Feb 25 01:04:36 UTC 2013 i686 GNU/Linux

Cat message :

cat messages
Jul 14 06:25:06 unis-monitor rsyslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="882" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Jul 15 06:25:05 unis-monitor rsyslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="882" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.

Cat syslog

cat syslog
Jul 15 06:25:05 unis-monitor rsyslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="882" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Jul 15 06:39:01 unis-monitor /USR/SBIN/CRON[15272]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)
Jul 15 07:09:01 unis-monitor /USR/SBIN/CRON[15465]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)
Jul 15 07:17:01 unis-monitor /USR/SBIN/CRON[15521]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 15 07:39:01 unis-monitor /USR/SBIN/CRON[15662]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)
Jul 15 08:09:01 unis-monitor /USR/SBIN/CRON[15855]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)
Jul 15 08:17:01 unis-monitor /USR/SBIN/CRON[15911]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 15 08:39:01 unis-monitor /USR/SBIN/CRON[16052]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)
Jul 15 09:09:01 unis-monitor /USR/SBIN/CRON[16273]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)

So if you have any clue of where to look or what i could use to monitor those date change ?

Here is some more infos : the server is a virtual server hosted on HyperV on a win 2012 server. Don't know if it changes anything, seen the other servers hosted don't have this issue...

Are you using NTP to maintain time synchronization? If not, clock drift can be expected, but a sudden 10 minute shift is not normal. — Aaron Copley, Jul 15 '13 at 14:19

score 10 · Answer 1 · edited Jul 15 '13 at 14:21

10

NTP is not an unnecessary component.

It's the only sane way to keep a system clock updated against an atomic time source.

You should install NTP, configure it to use your local pool time source, and everything will be fine.

edited Jul 15 '13 at 14:21

MadHatter

78,442
20
178
229

answered Jul 15 '13 at 14:20

Tom O'Connor

27,440
10
72
148

1

It would fix the issue but not the source of it : 10 minutes shifting is not normal, and shouldn't happen even without NTP. – PaKempf Jul 15 '13 at 14:40
2

@PaKempf Time drift is common with VMs. [Read here](http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/11/19/time-synchronization-in-hyper-v.aspx) – colealtdelete Jul 15 '13 at 15:17
Thanks Cole. HyperV has the date service activated on all VMs. Should I unactivate it on this one ? – PaKempf Jul 15 '13 at 15:27
1

@PaKempf no, do not disable time sync in Hyper-V. having time sync turned on is important because it resets the emulated hardware clock in the VM at power-on, resume, and snapshot rollback. – longneck Jul 15 '13 at 17:15

score 4 · Answer 2 · edited Apr 13 '17 at 12:14

In my tests I discovered that Hyper-V virtual machine clocks are very unreliable and frequently have a clock drift exceeding 500ppm. This is enough to cause even ntpd to fail. I had to switch to using chrony in these VMs to provide reasonably accurate wall clocks; it defaults to 1000ppm for this scenario and can be adjusted even further if necessary.

I no longer seriously consider Hyper-V for any application where timekeeping is especially important.

Luke404 · Accepted Answer · 2013-07-17T10:20:15.183

3

HyperV is the disease, NTPd is the cure.

Where to look or what i could use to monitor those date change?

You can query the NTPd daemon (via the ntpq client) to get the difference between local clock and the NTPd servers reference clock. But that implies actually running NTPd so you are not monitoring your changes alone, you're monitoring the combined effect of your running local clock and NTPd keeping it in sync.

I don't actually know if you can configure NTPd to run (and give you the above mentioned metrics) but not to actually adjust the system clock. A different, and less efficient, way would be to periodically (cron?) run ntpdate -q against a set of reference NTPd servers and monitor its output, which will give you the difference between your clock and the reference without actually touching the local clock. Output will be like this:

$ ntpdate -q $YOUR_TLD.pool.ntp.org
[... list of queried servers ...]
17 Jul 12:14:11 ntpdate[42868]: adjust time server 109.168.106.59 offset -0.002517 sec

You could filter the last number and graph it to get a nice view of how much and when your clock jumps around:

$ OFFSET=$( ntpdate -q $YOUR_TLD.pool.ntp.org | grep adjust | awk '{ print $10 }' )
$ echo $OFFSET
0.002970

edited Jul 17 '13 at 10:20

answered Jul 15 '13 at 14:21

Luke404

5,708
3
44
58

How could it not happen on the other VMs then ? – PaKempf Jul 15 '13 at 14:40
You hadn't provided any detail about other VMs so this cannot be answered. And should be posted as a different question anyway. – Luke404 Jul 15 '13 at 14:55
It was a rethorical question. It means that the point you raise (HyperV being the problem...) is unsufficient to solve the problem. – PaKempf Jul 15 '13 at 15:12
1

The point is that under virtualization (be it HyperV, VMware, Xen, KVM, and any other) there could be a large number of factors that cause clock drifts and/or jumps, and in the vast majority of them the "best" solution is to run NTPd inside the VM, just like you would do on non-virtualized systems to achieve a consistent clock across your whole infrastructure. Anyway I'm editing my answer to better fit your question. – Luke404 Jul 17 '13 at 10:10
Well, I finally installed NTP, set it to my timezone an added a crontab to execute it every 6h using ntpdate-debian command. Not sure the cron is mandatory, but seems to work ! – PaKempf Oct 03 '13 at 08:42
2

the recommended way is to run ntpd (the daemon), not ntpdate (the one-shot command). If you run the former you should not use the latter, under certain circumstances it could actually make your accuracy worse. – Luke404 Oct 03 '13 at 12:33

score 2 · Answer 4 · answered Jul 15 '13 at 17:13

Clock drift in a linux vm running in Hyper-V is very common if you don't have the correct integration components installed and enabled. Hyper-V will adjust the hardware clock of any VM it runs, but linux by default does not rely on the hardware clock once the system is booted. The integration components gives the kernel hints about the actual clock time and eliminates this problem.

If integration components are not an option for your distro, then the correct solution is to install something like ntpd and sync to the Hyper-V host or your local clock pool.

score 1 · Answer 5 · answered Jul 15 '13 at 17:21

The "problem" is that your running a version of Linux that does not support Pluggable Time Source Infrastructure. Several distros support it, but commonly only in their 64-bit OSes, and not in their 32-bit.

As others have mentioned, Integration Services allows for clock synchronization, but only if PTSI is supported. Most distros that support PTSI have the HV source already. See if there is a adjtimex port/package available for your distro.

Using NTP is a valid alternative, if you can't get PTSI working correctly. There are also some hacks including the tickadj command, and changing kernel boot variables - these should be avoided.

Linux clock loses 10 minutes every week

5 Answers5

Linked