VMware ESXi 5.1 can run Microsoft Hyper-V Server 2012 VMs, nice!

You heard all about guestOS = "winhyperv" here first!

What if you just want to virtually "kick the tires" on the latest Microsoft Hyper-V Server 2012, without committing dedicated hardware, to focus more on learning the features than the performance? Well, if you already have the prerequisites:

a) an ESXi 5.1 lab where you're allowed to tweak the Hyper-V VM's vmx file slightly (not changing anything in the ESXi host configuration)
b) a current subscription to MSDN or TechNet Professional, or even the free eval obtained by clicking the Free Download button here

then you're in luck. Just leave your VMware VMs going, and add a Hyper-V VM without the server UI stuff (called the Core version), for a smaller resources footprint, and the ability to "nest" Hyper-V VMs.

The video walk-through seen at the end is 32 minutes, but you can skip the video and deploy this in your lab in well under 15 minutes, if you already have the 1.7GB ISO file downloaded that is. Just follow along with the screenshots below!

Backstory:

Windows Server 2012 with Hyper-V includes these lofty claims:
Why Hyper-V? Competitive Advantages of Windows Server 2012 Hyper-V over VMware vSphere 5.1.

More info here:
Microsoft Hyper-V Server 2012
and in TechNet here:
technet.microsoft.com/en-us/library/hh833684.aspx.

Yes, the initial learning curve for Hyper-V without the full GUI is a bit rough:
Standalone Hyper-V is too painful to use
buy hey, all you really care about is getting it to run multiple VMs, so do you really need a fancy UI, do you? I finally got this all working just this past weekend, so I didn't waste any time getting this word out there (early Monday).

In other words, if you got VMware ESXi 5.1 working on your hardware, that means your BIOS and CPU are capable with Intel VT-x or AMD-V explained here, and you will very likely be able to get ESXi 5.1 to run Windows Server 2012 with Hyper-V.

Back on ESXi 5.0, luck with running nested Hyper-V VMs didn't work out well. By nesting, I mean getting ESXi 5.0 to run Hyper-V to run Hyper-V VMs at all, explained here:
tinkertry.comhyper-v-server-2012-release-candidate-x64-runs-fine-on-esxi-5-update-1

Doing a little research that included combing the best of these recent articles:

How to Enable Nested ESXi & Other Hypervisors in vSphere 5.1, Wednesday, August 29, 2012
www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html

How To Install CentOS 6.3 KVM on vSphere 5.1, by Chris Colotti
www.chriscolotti.us/vmware/nicira-nvp/how-to-install-centos-6-3-kvm-on-vsphere-5-1

coupled with the incentive that this really should work, based on folks' discussions over here (that didn't exactly spell out the exact steps):

Server 2012 Hyper-V under ESXi 5.x??, Sep 14, 2012 7:41 PM
communities.vmware.com/message/2115423

Running Nested VMs, Created on: Nov 30, 2008 9:17 PM by jmattson - Last Modified: Sep 12, 2012 2:06 PM by jmattson
communities.vmware.com/docs/DOC-8970

I was finally able to prototype and get this whole procedure going , and boiled it down to the simplest method I could come up with. I even managed to avoid the step of having to use the vSphere Web Client to expose hardware assisted virtualization to the guest OS, to speed you along toward success.

VMware ESXi 5.1 can run Microsoft Hyper-V Server 2012 VMs Comprehensive Implementation Guide

1) Install VMware ESXi 5.1:
Assuming your hardware platform is compatible, such as TinkerTry.com/vzilla, installation and configuration instructions of ESXi 5.1 can be seen here:
TinkerTry.com/install-esxi-5-1-flash-drive-then-configure-and-run-vms

2) Turn on SSH on ESXi using the vSphere Client:
If you haven't done this already, the steps are shown on VMware's site here, or a more visual approach here:
www.thomasmaurer.ch/2011/08/enable-ssh-on-esxi-5-via-vsphere-client

3) Download the 1.7GB ISO:
Microsoft Hyper-V Server 2012 Release Candidate (x64) - DVD (English)
Visit MSDN's download site, with the permalink to the x64 English version here:
msdn.microsoft.com/subscriptions/downloads#FileId=50558

The actual file you're grabbing is:
en_microsoft_hyper-v_server_2012_x64_dvd_915600.iso
or from the free eval site here by clicking the Free Download button, or just use the direct download link here, with a file named:
9200.16384.WIN8_RTM.120725-1247_X64FRE_SERVERHYPERCORE_EN-US-HRM_SHV_X64FRE_EN-US_DV5.ISO

These 2 files are identical, so doesn't matter which you choose (seen pictured below):

4) Use Datastore Browser (or Veeam FastSCP) to transfer the ISO file to your ESXi storage:
Consider creating a folder named ISO, then use Datastore Browser to upload your ISO file to that new folder

5) Configure your ESXi host network properly, turning on "Promiscuous Mode":
You'll need to enable "Promiscuous Mode" on your vSwitch that this Hyper-V VM network will use (there are a variety of ways to layout your network, including dedicating a NIC for this Promiscous mode Hyper-V that will still be accessible by clients), also seen in the video at this exact spot.

Source:
www.ivobeerens.nl/2012/06/28/nest-windows-server-2012-hyper-v-in-vmware-esxi-5

If you forget this step, you may get the Hyper-V Manager RPC Error "Cannot connect to the RPC service on computer. Make sure your RPC service is running."

6) Configure a new VM:
In the video, you'll see me create a new VM called "TEMPLATE-hyperv-Microsoft Server 2012 with Hyper-V" using the Typical wizard, pictured below.

The finalized product name was actually "TEMPLATE-hyperv-Microsoft Hyper-V Server 2012" so despite the screenshots, you may wish to go with that name instead.

Navigate to the ISO file, and be sure Connect at power on checkbox is on, click "Finish":

7) Upgrade Virtual Hardware to Version 9:

8) Tweak the VM's .vmx File:
Use free WinSCP to edit the .vmx file for the VM you just created. VMware's jmattson has informed me that these 2 tweaks are all that's needed, in the VMware Nested Virtualization forum:

Change this one line from:
guestOS = "windows8srv-64"
to
guestOS = "winhyperv"

and add the following line to the very end
featMask.vm.hv.capable = "Min:1"

followed by a carriage return, then save your changes.

Those changes, along with VM Hardware version 9 upgrade,  are the "secret sauce" to success. And potentially a bit easier than trying to do this from vSphere Web Client or VMware Workstation 9.

Sources:

The definitive simplification was from jmattson's response at http://communities.vmware.com/message/2119895#2119895, and his Running Nested VMs overview.
www.virtuallyghetto.com/2011/07/how-to-enable-support-for-nested-64bit.html
communities.vmware.com/docs/DOC-8970
tw.linkedin.com/in/aliao and www.youtube.com/watch?v=I-rjR4OawvQ
and Chris Colloti's article giving me the idea to compare the vmx file before and after editing with the vSphere Web Client to "Expose hardware assisted virtualization to the guest OS" as Chris explains here:
www.chriscolotti.us/vmware/nicira-nvp/how-to-install-centos-6-3-kvm-on-vsphere-5-1

9) Power on the VM, and install Hyper-V:
Proceed with a normal install

10) Configure Hyper-V:
Configure basic Hyper-V settings if you'd like, including allow Remote Management, and creating an administrator account for the remote client system that you'll be connecting, using the same username and password.

11) Install Hyper-V Manager on Windows 8 Release Preview:
Go to Add/Remove Programs, Add/Remove Windows Components, then turn on the Hyper-V Management Tools checkbox. I used my convenient, 3-minutes-to-deploy Windows 8 VM for the client, because the Hyper-V management UI is extremely easy to install.

If you're on Windows 7, get the equivalent function from
Remote Server Administration Tools for Windows 7 with Service Pack 1 (SP1).

12) Run Hyper-V Manager and connect to Hyper-V:
Now you're ready to hit the Windows key on your keyboard, and type "hyper" and hit enter, and the Hyper-V Manager should start right up and allow you to connect, or at least try to, but you'll get this error:

"Access denied. Unable to establish communication between"

So finding COM Management to fix this in Windows 8 wasn't easy like it was on Windows 7:
tokyoimage.wordpress.com/2010/10/15/hyper-v-manageraccess-denied-unable-to-establish-communication-between
but I finally found this post:
http://www.calvinirwin.net/tag/hyper-v
which told me to use Start, Run, DCOMCNFG, then click "Properties", "COM Security", "Edit Limits...", turn on the "Remote Access" checkbox and click OK

13) Created a simple "nested" VM:
Use all the defaults for a VM you create, but on the Connect Virtual Hard Disk, choose "Attach a virtual hard disk later". Why? We're just trying to test if a nested VM is allowed to boot succesfully, either it works, or it doesn't.

14) Power on to test the "nested" VM:
It not only works, it seems to work pretty well, with less CPU load noticed on the ESXi 5.1 host than the betas, this is a good sign.

Of course, future tests will exercise this new nested lab of mine considerably harder, with nested VMs with full operating systems, and Hyper-V live migrations. I'll also test conversions of VMware VMs to Hyper-V with Microsoft's tools here, just to compare with VMware Converter. And I need to test if Hyper-V VMs can actually see the network properly, and if the better vmxnet3 network driver would work.

But I need to save some fun for another day, don't I?

Sep. 24, 2012 01:28pm ET Update:

There might be an even simpler way, described here:
http://communities.vmware.com/message/2119892#2119892

where instead of those 4 lines in the original article/video:

mce.enable = TRUE
hypervisor.cpuid.v0 = FALSE
featMask.vm.hv.capable = "Min:1"
vhv.enable = TRUE

you keep only one line and change another. I will test further, and revise the above instructions as needed, details to follow.

Sep. 24, 2012 05:18pm ET Update:
Confirmed, this tip works! Seen below was a reply to my thread by jmattson, a VMware employee moderating the Nested Virtualization Forum:

Sep 24, 2012 4:02 PM
Re: VMware ESXi 5.1 can run Microsoft Hyper-V Server 2012 VMs too, nice!

pbraren wrote:
tweak the VMX, adding these 4 lines:
mce.enable = TRUE
hypervisor.cpuid.v0 = FALSE
featMask.vm.hv.capable = "Min:1"
vhv.enable = TRUE

jmattson replied:
Alternatively, it should suffice to change the guestOS line to
guestOS = "winhyperv"

...

Sep 24, 2012 5:18 PM in response to: pbraren
Re: VMware ESXi 5.1 can run Microsoft Hyper-V Server 2012 VMs too, nice!

I’m glad that you find this capability useful.

There is one line in the long version that isn’t subsumed by guestOS type. That’s featMask.vm.hv.capable = “Min:1″. With this setting, VC knows that the VM requires virtualized HV, and it won’t let you try to power on the VM on a host that isn’t capable of virtualized HV support. VC doesn’t know that guestOS=”winhyperv” implies the same constraint. Without this line, you’ll get a more unfriendly failure if you try to power on the VM on a host that can’t handle it.

It seems jmattson has kindly revealed this previously undocumented winhyperv parameter to us, and I'm very thankful. And he goes on to explain why featMask.vm.hv.capable should be left there as well.

His revelation does not imply any sort of formal VMware support, but he sure does a nice job keeping the Nested Virtualization Forum going, with >2000 posts.

I find it funny how Googling for "hyper-v"+winhyperv+esxi today, you get no relevant hits that have all 3 words in the results (at least not today). I would guess that won't last long.

The nested VMs still fire up just fine, with just as much speed as far as I can tell. And we have the reassurance that we've actually told ESXi 5.1 that it's winhyperv that we're running as a VM, without having to make any host-wide (config file) changes to ESXi 5.1 itself. Nice!

Instructions updates required for Step 8, along with comments updates below YouTube video DONE.

That's a wrap, put a fork in it! Finally, I can now play with Hyper-V, easily, any time I wish. vZilla just became even more versatile.

Here's the full video:

Jul 07 2013 Update:
Alternative approach documented over here:
http://wahlnetwork.com/2012/09/05/installing-hyper-v-server-2012-on-esxi-5/

Oct 03 2013 Update:
With ESXi 5.5 now out, can it run the latest Hyper-V? Apparently it might, with a new vSphere API, read more here:
virtuallyghetto.com/2013/09/quick-tip-new-hyper-v-guestos.html