Tuesday, November 25, 2008

ESX Server beats Hyper-V in vConsolidate Performance Benchmark Numbers by up to 125%



The Internet is great. There is a wealth of information up there, and with a little bit of work, you can find the best nuggets of information. You can even find performance benchmarking information comparing VMware ESX Server and Hyper-V. Stick with me here as I try to explain:

Principled Technologies is a company whose mission is: To provide the best possible technology assessment services to our clients. Whether you’re a vendor of technology products—hardware, software, services, or Web sites—or a user of those products, we can help assess the product’s performance, quality, market readiness, and other key characteristics.

They have done a LOT of work benchmarking different products, software, and hardware, and their reports are thorough. Recently, they have published two papers that have some interesting correlations. In August, 2008, Principled Technologies (PT) released a paper commissioned by IBM that was designed to compare IBM and HP servers. To perform this comparison, they used Intel's vConsolidate Benchmarking tool with VMware ESX Server 3.5 U1 as the virtualization backend virtualization platform. In September, 2008, PT released a paper commissioned by Intel to show the differences between their X7460 and X7350 processors. They also used vConsolidate to benchmark these as well. One interesting note, they used Microsoft Windows Server 2008 with Hyper-V as the backend virtualization platform.

Intel's vConsolidate

Intel's vConsolidate is a 3rd party developed benchmarking tool, similar to VMmark, that can be used compare two separate platforms. They do this by creating a group of 5 VMs that consist of a consolidate stack unit or CSU. With the speeds and capacity of modern servers, one CSU will not max out the system resources. Therefore, you stack multiple CSUs on the system until the CPU processing approaches 100% utilization. Once there, you can aggregate and normalize the data to come out with a score. These VMs consist of the following:

  Application OS
WebBench (Web) IIS Windows 32-Bit
LoadSim (mail) Exchange Windows 32-Bit
SysBench (database) SQL Server Windows 64-Bit
SPECjbb2005 (Java) BEA JVM Windows 64-Bit
Idle NA Windows 32-Bit


What was interesting about these two papers is that both of them used a VERY similar machine in their test which was a quad socket, quad core Intel Xeon processor X7350 based server. Both servers had 64GB RAM associated with them, and used vConsolidate Benchmarking to gather the results. The only major difference between these systems is that one used Hyper-V and the other used VMware ESX Server. PT also published the raw data numbers for all of these tests, and some interesting results came out. Here is a table with the average of the data from the outputs of the optimum CSU test results.

5 CSUs VMware ESX Microsoft Hyper-V %gain over Hyper-V
















VMware ESX Server beats Microsoft Hyper-V by up to 125%

First of all, neither of these tests were commissioned by VMware or Microsoft, which means we couldn't influence them in the least. Also, this comparison stressed the entire system by testing all the resources on the system by using a mixed workload of real world enterprise applications instead of simply one type of application. If you look at the results, you can see that VMware outperforms Hyper-V with an average win of 48%. It also shows that VMware ESX Server is the superior choice for high-performing environments.

So, what does this really mean?

  • This comparison proves that VMware ESX server is NOT a commodity!
    • Even though it is free, it provides virtualization capabilities far superior than other virtualization platforms.
  • Don't be afraid to run those Tier 1 applications within a VMware VM.
  • Check out other benchmarking and performance papers at VMware and our Performance Team's Blog, VROOM!, the VMware performance blog, for up to date information on what they are doing.

Full Disclosure

Now, these two papers show VERY similar systems, but they weren't identical. One difference is that the VMware ESX Server was an IBM x3850 M2 server which had 32 x 2GB DIMMS for its 64GB RAM whereas the Hyper-V Server was an Intel Whitebox server with 16 x 4GB DIMMs for its 64GB RAM. The IBM/VMware box used vConsolidate v1, whereas the Intel/Hyper-V system used vConsolidate v2. This use of different versions of the vConsolidate benchmark could introduce the possibility that the CSUs drove different loads on the system. Observation of the differences of vConsolidate aggregate scores between v1 and v2 seem to confirm this. However, our VMware performance team contacted the Intel vConsolidate team and learned that the raw scores reported are comparable between vConsolidate versions one and two. In other words, the RAW data shown in the table above for the Database, Java, Mail, and Web workloads are comparable.

Lastly, your mileage may vary, but I am sure you would see the same results if you set this up.