The Internet is great. There is a wealth of information up there, and with a little bit of work, you can find the best nuggets of information. You can even find performance benchmarking information comparing VMware ESX Server and Hyper-V. Stick with me here as I try to explain:
Principled Technologies is a company whose mission is: To provide the best possible technology assessment services to our clients. Whether you’re a vendor of technology products—hardware, software, services, or Web sites—or a user of those products, we can help assess the product’s performance, quality, market readiness, and other key characteristics.
They have done a LOT of work benchmarking different products, software, and hardware, and their reports are thorough. Recently, they have published two papers that have some interesting correlations. In August, 2008, Principled Technologies (PT) released a paper commissioned by IBM that was designed to compare IBM and HP servers. To perform this comparison, they used Intel's vConsolidate Benchmarking tool with VMware ESX Server 3.5 U1 as the virtualization backend virtualization platform. In September, 2008, PT released a paper commissioned by Intel to show the differences between their X7460 and X7350 processors. They also used vConsolidate to benchmark these as well. One interesting note, they used Microsoft Windows Server 2008 with Hyper-V as the backend virtualization platform.
Intel's vConsolidate
Intel's vConsolidate is a 3rd party developed benchmarking tool, similar to VMmark, that can be used compare two separate platforms. They do this by creating a group of 5 VMs that consist of a consolidate stack unit or CSU. With the speeds and capacity of modern servers, one CSU will not max out the system resources. Therefore, you stack multiple CSUs on the system until the CPU processing approaches 100% utilization. Once there, you can aggregate and normalize the data to come out with a score. These VMs consist of the following:
| Application | OS | |
| WebBench (Web) | IIS | Windows 32-Bit |
| LoadSim (mail) | Exchange | Windows 32-Bit |
| SysBench (database) | SQL Server | Windows 64-Bit |
| SPECjbb2005 (Java) | BEA JVM | Windows 64-Bit |
| Idle | NA | Windows 32-Bit |
Results
What was interesting about these two papers is that both of them used a VERY similar machine in their test which was a quad socket, quad core Intel Xeon processor X7350 based server. Both servers had 64GB RAM associated with them, and used vConsolidate Benchmarking to gather the results. The only major difference between these systems is that one used Hyper-V and the other used VMware ESX Server. PT also published the raw data numbers for all of these tests, and some interesting results came out. Here is a table with the average of the data from the outputs of the optimum CSU test results.
| 5 CSUs | VMware ESX | Microsoft Hyper-V | %gain over Hyper-V |
| Database | 136.028 | 60.526 | 125% |
| Java | 17296.6875 | 15268.49 | 13% |
| | 20.104 | 15.368 | 31% |
| Web | 469.344 | 384.764 | 22% |
VMware ESX Server beats Microsoft Hyper-V by up to 125%
First of all, neither of these tests were commissioned by VMware or Microsoft, which means we couldn't influence them in the least. Also, this comparison stressed the entire system by testing all the resources on the system by using a mixed workload of real world enterprise applications instead of simply one type of application. If you look at the results, you can see that VMware outperforms Hyper-V with an average win of 48%. It also shows that VMware ESX Server is the superior choice for high-performing environments.
So, what does this really mean?
- This comparison proves that VMware ESX server is NOT a commodity!
- Even though it is free, it provides virtualization capabilities far superior than other virtualization platforms.
- Don't be afraid to run those Tier 1 applications within a VMware VM.
- They will run very well, and you will see tremendous ROI by running on VMware Infrastructure 3.
- Check out other benchmarking and performance papers at VMware and our Performance Team's Blog, VROOM!, the VMware performance blog, for up to date information on what they are doing.
Full Disclosure
Now, these two papers show VERY similar systems, but they weren't identical. One difference is that the VMware ESX Server was an IBM x3850 M2 server which had 32 x 2GB DIMMS for its 64GB RAM whereas the Hyper-V Server was an Intel Whitebox server with 16 x 4GB DIMMs for its 64GB RAM. The IBM/VMware box used vConsolidate v1, whereas the Intel/Hyper-V system used vConsolidate v2. This use of different versions of the vConsolidate benchmark could introduce the possibility that the CSUs drove different loads on the system. Observation of the differences of vConsolidate aggregate scores between v1 and v2 seem to confirm this. However, our VMware performance team contacted the Intel vConsolidate team and learned that the raw scores reported are comparable between vConsolidate versions one and two. In other words, the RAW data shown in the table above for the Database, Java, Mail, and Web workloads are comparable.
Lastly, your mileage may vary, but I am sure you would see the same results if you set this up.
8 comments:
Surely one has to use identical hardware and vConsolidate versions before any fair conclusions can be drawn from performance benchmarks?
Don't get me wrong, I am a VMware fan but how valid is this test? Re-run the tests again but this time use Intel & vConsolidate v2for VMware and IBM & vConsolidate v1 for Hyper-V. Now that would be interesting....
Quad Core and 64GB RAM and only 5 VMs? I want to know how they perform with more than 5 VMs.
Another bullshit from "blogger" who writes 11 posts per year. Nothing from his own, - different numbers, for different configurations and different benchmarks.
My Ford car is better then your Gibson guitar, because it is better. That is the way, author thinks.. :)
It's not 5vm's per host, it's 5 vm's per csu, multiple csu's per host. Anyone that has done any work with ESX and HyperV in a serious production environment (1000's of VM's) knows that ESX is far superior.
It would be nice to see identical independant comparisons though.
gogogo5 and Scott:
You are right, in a perfect world, it would be nice if both tests were run exactly the same, but with only different Virtualization Platforms. If we were talking about the final scores in the test (vConsolidate v1 gave VMware ESX Server a 3.04 whereas vConsolidate v2 gave Microsoft Hyper-V a 3.10) I may agree with you, but after talking with the vConsolidate folks, the RAW data numbers (aggregate score for Database, or Web, or Java, etc.) are equivalent AND comparable. Though it isn’t perfect, it is probably as close as one is going to get with published numbers.
Anonymous: What Scott says is correct. There were 5 VMs per Consolidate Stack Unit, and they ran 5 CSUs per machine, so there were actually 25 VMs being run on these 16 core machines (not a ton of consolidation, but that is really 40 Virtual CPUs on the 16 core machine), and these VMs were being stressed with as much workload as they could take, so it is a decent test run.
The other Anonymous: If you don't agree, that is fine, but keep the profanity out of it. I don't want to have to moderate comments, so be respectful, and if you don't like what I say, don't read the blog.
In other word to Anonymous "Get lost as we dont care about you.Be happy with your ford car and keep thinging that it is the best car ever made :) "
VMware ESX Server beats Microsoft Hyper-V by up to 125%.
About a decade ago we heard almost the same. E-directory and active directory.
Do I need to say more.
I just finished migrating our SQL server from VMware to Hyper-V and have notice a significant increase in performance. Both hardware are identical so I'm guessing Hyper-V is much more optimized to run Windows than VMware is. VMware has more focus on Linux OS in my opinion.
Post a Comment