20

On my host I am using libvirt and a KVM guest. When the host is shutting down, libvirt suspends the guest. When the host is starting up, libvirt resumes the guest. The problem is, if the guest is suspended and resumed after 24 hours for example, then the guest time is 24 hours in the past.

I thought that maybe the problem is with the clocksource, but it is set to "kvm-clock" already.

$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
kvm-clock tsc hpet acpi_pm 

$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
kvm-clock
Hristo Hristov
  • 313
  • 1
  • 2
  • 7

6 Answers6

14

The Problem

I've got the same problem and I haven't found a good solution. Here's what I found:

The problem is that after resume, the system and hardware clock times on the guest are different:

root@guest:~# date; hwclock
Sat Oct 11 13:09:38 UTC 2014
Sat Oct 11 13:10:42 2014  -0.454380 seconds

On the host, they agree:

root@four:~# date; hwclock
Sat Oct 11 13:11:35 UTC 2014
Sat Oct 11 13:11:36 2014  -1.000372 seconds

The solution would be to run hwclock --hctosys on the guest after it's been resumed. However, I haven't found a way to do this with changes on the guest system only, as the guest doesn't notice that it is suspended and resumed.

QEmu Guest Agent

There is the possibility to run a software called QEmu Guest Agent on the guest and notify from the host to update the guest system clock from the guest hardware clock. However, the page mentions that the guest agent makes the host and guest vulnerable to attacks from each other because of issues with a JSON parser (at least I believe that the affected code is also run on the host, I'm not sure about that). Anyway, here's how to set that up:

  1. Set up a virtio serial channel for the agent as mentioned in the libvirt wiki (see also libvirt domain format documentation).

  2. After the serial channel is available, install and start the QEmu Guest Agent on the guest. (Debian: apt-get install --no-install-recommends qemu-guest-agent.)

  3. Trigger the clock offset by suspending, waiting and resuming. Then run the following command on the host to correct it: virsh qemu-agent-command backup '{"execute":"guest-set-time"}' The wiki page that using virsh qemu-agent-command is unsupported, but I haven't found any other command that does the job.

I found two discussions on automating within libvirt the call to guest-set-time on resume from suspend:

However, nothing has been implemented yet as far as I could see.

I found information on how to submit commands to the guest agent on wiki of stoney-cloud.org.

I've also tried setting tickpolicy="catchup" in the libvirt timer configuration but this didn't solve the problem.

NTP

An alternative to using the agent would be to use an ntp daemon or to call ntpdate periodically from a cron job. I wouldn't recommend the latter, as it can cause the time to go backwards, which can confuse programs (for example, the Dovecot IMAP server doesn't try to handle time going backwards and can terminate).

I tried the following ntp daemons:

  • openntpd: Corrects time very slowly at a rate of about 2 seconds per 60 minutes in my test. The time offset was 120 seconds. Also, openntpd throws an error if the time offset is too large and, in my test, completely fails to correct time in that case. Advantages of openntpd: Can run as regular user in chroot.

  • chrony: Corrects a time offset of 120 seconds in 30 minutes in my test. chrony can be configured to run as regular user. chroot support is not implemented. NTP server polling interval can be configured for each NTP server.

  • systemd-timesyncd: Corrects a time offset of 120 seconds in 30 seconds in my test. Runs as regular user by default. However, the polling interval of NTP servers increases up to 2048 seconds, so that a suspend/resume wouldn't be detected until 34 minutes after the resume in the worst case. This does not seem to be configurable. Also, I've observed timesyncd step the time backwards, which causes the same problems as calling ntpdate in a cron (see above).

chrony solves the problem. Openntpd isn't suitable because its correction rate is too low and doesn't seem to be configurable. systemd-timesyncd doesn't entirely solve the problem either, because its polling interval is not configurable.

I tested the following Debian versions of the NTP daemons: openntpd 20080406p-10, chrony 1.30-1 and systemd 215-5+b1.

sampi
  • 158
  • 5
Milan
  • 241
  • 2
  • 4
  • Since systemd release 236 (from December 2017), the timesyncd polling interval can be configured using `PollIntervalMinSec` and `PollIntervalMaxSec`. I can confirm that with `PollIntervalMaxSec=60`, resumed VMs pick up the current time pretty quickly. – F30 Oct 23 '20 at 09:58
4

Many virtualization host operations on the guest can result in a pause - resume. This will negatively effect the system clock at the guest. For example, cloning a VM results in a pause while cloning. The guest clock afterwards is behind. To get NTP to sync the clock you need to then restart the guest--not a good solution in all cases for sure. Alternative you could just restart ntpd in the guest, but that is not optimal either. Ideally there needs to be an event (VM resumed) available that you could optionally use for this type of correction to the guest.

After spending some time researching this I decided to use the host clock directly as a reference for the CentOS 7 guest OS system clock.

Instead of running ntpd in the guest, I decided that every 15 minutes I would set, via crontab, the guest system clock from the guest's hardware clock. The guest's hardware clock reflects the time at the virtualization host which is controlled via ntpd running on the virtualization host. This provides me reliable time in the guest OS. Worst case, the clock may be off for up to 15 minutes before it gets syncronized to the proper time after resuming the guest.

# crontab -e

0,15,30,45 * * * * /sbin/hwclock --hctosys

It would be much better to have an event available at the guest that would initiate a time syncronization when the guest was resumed, but apparently that is not available. The crontab approach is a workaround in that it makes a hwclock call every 15 minutes. It gets the job done, but not as elegantly as I would like.

zman58
  • 41
  • 2
3

libvirt supports guest time sync since 2015. On Debian Stretch and later look for option SYNC_TIME in /etc/default/libvirt-guests:

# If non-zero, try to sync guest time on domain resume. Be aware, that
# this requires guest agent with support for time synchronization
# running in the guest. For instance, qemu-ga doesn't support guest time
# synchronization on Windows guests, but Linux ones. By default, this
# functionality is turned off.
#SYNC_TIME=1

You can test time sync from within host system with: 

virsh qemu-agent-command INSERT_YOUR_DOMAIN_HERE '{"execute":"guest-set-time"}'

This command should return {"return":{}} on success.

Jakob
  • 31
  • 2
  • On at least Debian Bullseye this is not sufficient. A channel also needs to be configured for each guest, as described in the answer by Milan. – sampi Feb 28 '21 at 12:29
2

kvm-clock syncs the guest time to host time on guest startup. You should use and ntp client in the guest, and shutdown/startup instead of using suspend/resume.

dyasny
  • 18,482
  • 6
  • 48
  • 63
  • Yes, I can confirm it syncs on startup, because when I do shutdown/startup of the guest everything is fine. Using ntp is not a solution for many reasons (it is a workaround, it panics when the time difference is huge, it requires access to time server). I am searching for a way to solve the problem with suspend/resume, because this is an interesting, nice and **default** option in libvirt. – Hristo Hristov Nov 25 '11 at 11:39
  • suspend is 1) migrate VM state to file and 2) destroy. When you resume from suspend, the VM state is restored (migrated from file back to VM memory). This state will include the current timestamp. So yes, it is default, but no, timing still matters, and the time has to come from somewhere, and this is where NTP should come in. I doubt another clock source will help but you can try with acpi_pm. – dyasny Nov 25 '11 at 14:19
  • You should [not use NTP in the guest](https://help.ubuntu.com/community/KVM/FAQ#Should_ntp_be_used_for_time_synchronisation.3F). – Brian Cain Nov 25 '11 at 22:55
  • 4
    @Brian Cain this is highly arguable, especially with no explanation or reasoning behind the statement. To provide a profflink: http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Virtualization_for_Servers/2.2/html/Administration_Guide/chap-Virtualization-KVM_guest_timing_management.html – dyasny Nov 26 '11 at 11:08
1

Since Linux Kernel 4.11 there is PTP-KVM (Precision Time Protocol - Kernel Virtual Machine): PTP is comparable to NTP, but with higher precision and for local network. The Linux host system exports its current time to any guest system for efficient reading as /dev/ptp_kvm (or /dev/ptp0). chrony is capable of using that inside the VM:

modprobe ptp_kvm
echo ptp_kvm >/etc/modules-load.d/ptp_kvm.conf
apt-get install chrony
echo "refclock PHC /dev/ptp0 poll 2" >/etc/chrony/conf.d/ptp.conf
systemctl restart chronyd

As a requisite the time on the host must be synchronized too.

You might want to also configure makestep 10 - depending on which difference you are willing to tolerate: for this example a difference below 10s would be adjusted by speeding up the VM clock, while larger differences would we stepped with all (bad) consequences.

I found this in the RedHat documentation, so kudos to them.

pmhahn
  • 96
  • 7
0

I use similar way for syncing time after VM suspend/resume, but i think it's better to try to guess, that it should sync in the right direction and is longer than short difference, that could be fixed by NTPD.

https://gist.github.com/jhrcz/7138803

PS. New centos 6.7 changelog says this could be done automaticaly with only kvm-clock clock source.

Dave M
  • 4,494
  • 21
  • 30
  • 30
Jan Horacek
  • 1