Now, is a Great Time to be Keeping Time

There have been issues in the past where a guest OS's time would drift away from the current time. I have seen instances with ESX Server 2.5 where the clock in a VM would be a day behind or ahead if I let it run over the weekend. For most applications this is bad, but for applications that rely on the ability to keep synched with other applications (whether physical or virtual) this is unusable. Back in the 2.5 days there were ways to address this that helped a lot, but didn't solve the problem entirely  like adding the clock=pit boot option to the guest's kernel command line in the /etc/lilo.conf (See KB Article 1420). These fixes also meant that there had to be some changes to the VM and folks didn't like to change the VM kernel. One of the major reasons for this time drift is that the default Linux kernel timer frequency is 1000 Hz which means that the Linux kernel wanted a timer interrupt 1000 times a second. When you have SMP VMs it doesn't just double, but there is a quadratic increase (2 CPUs = 6000 Hz, 4 CPUs = 20,000 Hz).

 

Making the changes in the aforementioned KB Article 1420 would help with the time drift, but it didn't fix it for all VMs and there could still be some drifting. It also appeared that paravirtualized Xen VM guests didn't have this problem. The reason for this is that when you create the paravirtualized VM, it automatically sets the kernel timer frequency within the VM to 250 Hz, so in a VMware VM, which we used the default kernel timer interrupt of 1000 Hz because one normally wouldn't want to recompile the kernel in the VM to adjust the timer, in Xen you got the benefits of less timer interrupts because they give you no choice and change the kernel timer frequency automatically.

 

With ESX Server 3.0.x, we made some improvements for the VM, and the drift for a lot of workloads went away, but it could still rear its ugly head. With ESX Server 3.5 things got a lot better. Not only that, Linux kernels changed the number of Hz they default with, or they have added kernel boot parameters (divider=10) to allow a person to decrease the number of interrupts necessary for a VM.

 

VMware released a very comprehensive document titled, "Timekeeping in VMware Virtual Machines" (http://www.vmware.com/pdf/vmware_timekeeping.pdf). It goes into what the problem is, why it occurs, and how to get your VMs keeping track of time reliably.

 

So like the title of the blog says, now truly is a great time to be keeping time.