1

I have a unRAID 6.6.5 server with perfectly correct time and multiple Linux VMs which used to be perfectly synced too. I rebooted one of my VM and suddenly the clock goes twice as fast as the hwclock. I can easily see that by doing these 2 loops in 2 different terminals:

while true; do date; sleep 1; done;
while true; do hwclock --show; sleep 1; done;

If that matters, the hardware behind the unRAID server is an Intel Xeon E3-1245 V6 on a E3C236D2I with ECC RAM. Also, all my VMs are currently running CentOS 7. By default, CentOS 7 comes with chronyd installed. Doing either of these 2 commands will sync the clock, but since the clock goes twice as fast anyway, it's immediately out of sync after:

hwclock --localtime --hctosys
chronyc makestep

I have also tried using ntpd. The result is exactly the same, I can force the sync but the clock still goes too fast. Also, all my VMs are configured the same:

[root@vm ~]# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
tsc acpi_pm
[root@vm ~]# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

I have also tried switching to acpi_pm in the past without seeing any improvement. The unRAID server also runs with TSC:

root@server:~# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc
root@server:~# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
tsc hpet acpi_pm

What am I missing? I've done extensive research on the matter and haven't found a single solution to be working yet.

dan
  • 83
  • 7

1 Answers1

0

While your system is running, the hardware clock is not used. The system clock maintained by the kernel is the only one which matters. The kernel will sync the hardware clock to the system time automatically every 11 minutes anyway, but that's just to deal with the possibility of a clean shutdown not happening.

The questions to ask are:

  1. Is the system time shown by your VM host accurate? (i.e. Do date and chronyc -n tracking show the correct time, and does chronyc -n sources show your system keeping reasonably close with its sources?)
  2. Is the system time shown by your VMs accurate? (Use the same checks as above.)
  3. After you reboot, do your VM host and its VMs come back to the correct time reasonably quickly? (Systems can lose or gain significant time during reboots due to the hardware/system clock difference which you're seeing.)

If the answers to those questions are yes, then your system is doing the right thing. Ignore the hardware clock while your system is running.

Paul Gear
  • 3,938
  • 15
  • 36
  • It's hard to ignore when it runs twice as fast. Makes it hard to read logs. Yes, `date`/`chronyc -n tracking` are perfectly fine on the host. On the VM however, `date` is only correct for the first second after a reboot, it goes twice as fast after that. I can do `chronyc makestep` to make it right again but only for a second. `hwclock` on the other end always returns the correct time. – dan Dec 05 '18 at 09:26
  • Does your VM use open-vm-tools or a similar client which is set to synchronize the VM clock with the host clock? If so, make sure it's turned off. – Paul Gear Dec 10 '18 at 23:09
  • No, it doesn't. It's just a barebone installation of CentOS when it comes to the VMs. unRAID itself doesn't have a lot of tools regarding VMs. – dan Dec 11 '18 at 00:10
  • Which hypervisor are you using? Your VMs should be using a virtualised clock driver for maximum accuracy. On Xen, it's called "xen"; on KVM, "kvm-clock"; on Hyper-V, "hyperv_clocksource_tsc_page". It seems to me your hypervisor and/or guests are not running an ideal kernel for your setup. – Paul Gear Jan 30 '19 at 01:23
  • I'm just using Unraid, might not be ideal but works well enough overall. – dan Jan 30 '19 at 03:58
  • I'm not familiar with Unraid, but you should check out if it has options for running different kernels which are more suited to virtualised servers. – Paul Gear Jan 30 '19 at 04:44
  • Of course, it might just be that you were unlucky and got some hardware with an unstable clock, but the fact that the host is OK and the guests aren't suggests the fault lies in one or both kernels. – Paul Gear Jan 30 '19 at 04:45
  • Sadly Unraid doesn't provide any additional clock. It uses tsc which works well for the host but my guests just have the default CentOS clocks which is also tsc. All of this is actually mentioned in my question already... – dan Jan 30 '19 at 11:39