Possible to run nested ESXi 5.1 within vCenter Server / ESXi 5.0?

[ma11achy]

Hi all,

I have sucessfully created a nested VM of ESXi 5.1 with vCenter Sevrer 5.1 VM within my vCenter Server / ESXi 5.0 physical  environment.

I have sucessfully powered on the nested ESXi 5.1 and added it to the vCenter Server 5.1 VM.

However, I am having trouble getting Virtual Machines to run within the nested virtual environment, such as dropped iSCSI connections, VM's suddenly becoming unresponsive and most of all - Purple Screens of Death on the nested ESXi 5.1.

Is what I am attempting to do just impossible, or are there tweaks to the 5.0 physical environment that I can add to make this possible?

Many thanks,

Martin.

[vmroyale]

Note: Discussion successfully moved from VMware vCenter™ Server to Nested Virtualization

[admin]

Please post the vmware.log file for the outer VM.

[ma11achy]

I've added the ESX outer VM and vCenter outer VM log files.

Cheers,

Martin.

[admin]

You should add the following option to /etc/vmware/config on the ESXi 5.0 host:

vhv.allow = TRUE

In the configuration file for the ESXi 5.1 VM, change

guestOS = windows7srv-64

to

guestOS = vmkernel5

For more information on running nested VMs, see .

[ma11achy]

Hi Jim,

Thanks - already had vhv.allow on the physical host, as we run many nested ESX VM's within our environment.

Added the guestOS statement, but no joy however, still having problems.

Cheers for the assistance, but I think I'm going to call nested ESX 5.1 on physical ESX 5.0 nested a "not possible" for the moment.

I'll go and build a 5.1 physical and see how things go.

Cheers,

Martin.

[admin]

ma11achy wrote:

Hi Jim,

Thanks - already had vhv.allow on the physical host, as we run many nested ESX VM's within our environment.

According to the log file you provided, you don't have vhv.allow in /etc/vmware/config:

2012-11-23T15:59:12.076Z| vmx| DICT --- HOST DEFAULTS /etc/vmware/config

2012-11-23T15:59:12.076Z| vmx| DICT                    libdir = /usr/lib/vmware

2012-11-23T15:59:12.076Z| vmx| DICT           authd.proxy.vim = vmware-hostd:hostd-vmdb

2012-11-23T15:59:12.076Z| vmx| DICT           authd.proxy.nfc = vmware-hostd:ha-nfc

2012-11-23T15:59:12.076Z| vmx| DICT        authd.proxy.nfcssl = vmware-hostd:ha-nfcssl

2012-11-23T15:59:12.076Z| vmx| DICT   authd.proxy.vpxa-nfcssl = vmware-vpxa:vpxa-nfcssl

2012-11-23T15:59:12.076Z| vmx| DICT      authd.proxy.vpxa-nfc = vmware-vpxa:vpxa-nfc

2012-11-23T15:59:12.076Z| vmx| DICT            authd.fullpath = /sbin/authd

2012-11-23T15:59:12.076Z| vmx| DICT          authd.soapServer = TRUE

2012-11-23T15:59:12.076Z| vmx| DICT vmauthd.server.alwaysProxy = TRUE

What am I missing?

[ma11achy]

vhv.allow is on the physical host - allowing us to run nested ESXi 5.1

There shoudl be no need to add vhv.allow to the nested ESXi 5.1 to run VM's. It would be a backwards step for VMWare to have to add a line to the config file just to run VM's?

[admin]

ma11achy wrote:

vhv.allow is on the physical host - allowing us to run nested ESXi 5.1

The log file I quoted above is from your physical host and reports the entire contents of /etc/vmware/config on the physical host.  I don't see vhv.allow in there.

Note that you can run 32-bit nested VMs under an ESXi 5.1 guest even without vhv.allow.  However, performance will be poor, and as a result of the poor performance, you can encounter stability issues.  For 64-bit nested VMs, or for good performance of 32-bit nested VMs, you do need vhv.allow = TRUE.

Internally, we run tens of thousands of nested VMs with ESXi 5.1 as a guest of ESXi 5.0.  I can assure you that this configuration works fairly well.

[Schorschi]

I have another interesting wrinkle, I have 5.1 on bare metal, 5.1 running in to VMs, the VM based ESXi hosts have 64-bit VMs running.  However, the 64-bit VMs running in the nexted 5.1, cannot receive DHCP offers nor does PXE/TFTP work.  The nested ESXi hosts can use DHCP/PXE/TFTP just fine.  However any VMs on the nested ESXi hosts can't.   PXE-E11 error, TFTP timeout, no PXELinux menu for example.  If I set the VMs to static IP they work to a degree, but they can't ping the default gateway of the environment, nor does any name resolution work, simple ping to default gateway returned unavailable route?  I know for a fact the physical network, DHCP, DNS, etc. work, as well as PXE/TFTP.  So what would cause TFTP not to work?  Or the VMs in the nested ESXi hosts, fail to see their default gateway even if configured static?

If I take a completely functional VM from the ESXi host, bare metal and add it to one of the nested ESXi hosts, it fails to work, if I take a VM from a nested ESXi host, to a bare metal ESXi host, it works, meaning it can find and reference the default gateway, so the issue is in the network definition of the nested ESXi hosts?  No customization has been done, other than enablement of vMotion support.  Weird.

True, maybe this issue should be a separate issue/discussion, but since it appears to be an issue that nested ESXi 5.1, since I never saw this issue with nested 5.0, here, made some logical sense, at least to me. 

[admin]

Have you enabled promiscuous mode for the vSwitch that the virtual ESXi guests are attached to on the physical ESXi host?

[Schorschi]

That was it... I swear we had turned it on... but it was not enabled correctly on the bare-metal ESXi instance/vSwitch used by the nexted ESXi instances.  Late last night I started double checking, and found the gap in the configuration.  For future reference, Iooking at the vmkernel output (ALT-F12), the requests for promiscuous mode are echoed in log as well if not enabled.

[admin]

Jim,

Just wanted to check if the the 5.0 physical ESXi needs to be rebooted after this vhv.allow option is added ?

[admin]

No.  The /etc/vmware/config file is read whenever a VM is powered on.

[admin]

Thanks for the quick response!