That's the CPUID output from the host, which doesn't really tell me what I need to know. Thank you, though, for providing the CPUID output from a kvm guest in a private thread.
It appears that kvm reports inconsistent CPUID information with '-cpu host' if the host supports XSAVE. Though kvm reports XSAVE support, it hides CPUID leaf 0xD, which contains information on how to use XSAVE. As a result, ESXi tries to use XSAVE incorrectly, and the CPU raises a #GP fault. This leads to a PSOD.
The workaround, for the present, is to hide XSAVE from the ESXi guest. This can be done with '-cpu host,-xsave'.
The full command line is:
qemu-system-x86_64 -enable-kvm -cpu host,-xsave -machine vmport=off -m 4096 -smp 2 -hda esxi00.img -netdev tap,helper=/usr/lib/qemu-bridge-helper,id=hostnet0 -device vmxnet3,netdev=hostnet0,id=net0 -cdrom VMware-VMvisor-Installer-6.0.0.update01-3029758.x86_64.iso
I wrote a blog post with step-by-step directions for getting this to work. Note that the latest QEMU doesn't need the patch anymore, as they've added a "-machine vmport=off" option.
These directions specifically cover setting up ESXi in a guest managed by libvirt / virt-manager. Hope it helps.
I've discovered the following additional issues with ESXi on kvm:
1. At present, kvm does not appear to support XSAVE, AVX, AVX2, or BMI2. On my (now fairly ancient) Haswell system, I find I need to invoke qemu-system-x86_64 with "-cpu host,-xsave,-avx,-avx2,-bmi2"
2. There appear to be some issues regarding the communication of CPU topology. For an 8-vCPU ESXi VM, I find that I need to invoke qemu-system-x86_64 with "-smp cores=8,sockets=1"
3. When running under kvm, ESXi 6.0 cannot run VMs with debugging enabled, because it tries to verify the value of a VMCS field that kvm does not implement.
This configuration is still not very stable, but some things do work.
hello all
read this topic and thanks for info
in my case
qemu 2.5
ubuntu 14.04
after pre installing esxi , I see last message kernel up
then first yellow screen and stuck initializing chipset
using -cpu host
what can be wong
Please connect a virtual serial device and post the serial device log.
hello!
Compiled against library: libvirt 1.3.2
Using library: libvirt 1.3.2
Using API: QEMU 1.3.2
Running hypervisor: QEMU 2.5.50
Running against daemon: 1.3.2
cat /sys/module/kvm_intel/parameters/nested
Y
cat /sys/module/kvm_intel/parameters/ept
Y
cat /sys/module/kvm/parameters/ignore_msrs
Y
qemu-system-x86_64 -enable-kvm -cpu host,-xsave,-avx,-avx2,-bmi2 -m 4096 -smp cores=2,sockets=1 \
-machine vmport=off \
-boot d -cdrom ESXI5.iso \
-drive file=esxi.qcow2,format=qcow2,if=virtio,bus=0,unit=0,cache=none \
-serial mon:stdio \
-vnc :smileylaugh:
no network, for testing not need.
sorry for stupid questiong how i can connect and take this log?
like this ? -serial mon:stdio \
hmm in my case, when i use cpu host all time stuck on initializing chipset, and HOST very unstable state or stuck too...
i change to core2duo and ESXI installing
why? i am use 2630L 2.4Ghz
hello againg) dont know get i luck or not for answer
now i can run ESXI under KVM
but lol
when VM start long LOGO in BIOS VMWARE and then ( in my case ) virtual machine start boot up
initrd -> kernel up -> stuck
install windows....first loading succesfull then stuck....
i try local hard ( ide kvm )
and iscsi
all time VM stuck when i starting install or start boot up VM machine
Nice to see this sort of working. Can you power off the VM and add the following settings to the VMs .vmx file:
vmx.buildType=debug
monitor_control.log_vmsample = "TRUE"
this will enable additional debugging info that will help us to root cause.
Thanks for answer
first step
run vm
qemu-system-x86_64 -enable-kvm -cpu IvyBridge,+vmx,-xsave,-avx,-avx2,-bmi2,-erms,-xsaveopt -m 8192 -smp cores=8,sockets=1 \
-machine q35,vmport=off \
-drive file=esxi.raw,format=raw,if=ide,bus=0,unit=0,cache=none \
-device vmxnet3,netdev=pnet0,mac=00:AA:AA:AA:AA:AB -netdev tap,id=pnet0 \
-vnc :smileylaugh:
why i not use -cpu host or -cpu pass-throught, cuz dont know why, all time ESXI stuck on initializing chipset
work only if i change to other CPU ( core2duo for example )
i am using Xeon 2630L v2 2.4GHz ( i am try VSA on nested ESXI and all work correctly on those cpus )
from cpu_map.xml i see my IvyBridge E3-12xx ( not my, but good too )
now ESXI run correctly, and installing correctly, no warrning see
i can run 64 bit Guest without error or someing else
add to .vmx
vmx.buildType= "debug"
monitor_control.log_vmsample = "TRUE"
cat >> /vmfs/volumes/VMFS/VM/VM.vmx << EOF
vmx.buildType=debug
monitor_control.log_vmsample = "TRUE"
EOF
cat /vmfs/volumes/VMFS/VM/VM.vmx
1. Black Screen 20-30 sec
2. Logo VMWARE 20-30 sec
3. boot up and stuck
cannot reset machine, only power off
this is part of log can be helpfully
ubuntu in CD
boot up -> Try ubuntu without install
VM power off
2001-01-01T00:04:02.261Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:02.262Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=9
2001-01-01T00:04:02.267Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:02.268Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=10
2001-01-01T00:04:02.273Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:02.274Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=11
2001-01-01T00:04:02.277Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:02.277Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=12
2001-01-01T00:04:02.585Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:02.585Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=13
2001-01-01T00:04:02.758Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:02.759Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=14
2001-01-01T00:04:02.765Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:02.765Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=15
2001-01-01T00:04:02.769Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:02.769Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=16
2001-01-01T00:04:02.935Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:02.936Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=17
2001-01-01T00:04:03.056Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:03.057Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=18
2001-01-01T00:04:03.062Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:03.062Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=19
2001-01-01T00:04:03.064Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:03.065Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=20
2001-01-01T00:04:03.067Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:03.067Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=21
2001-01-01T00:04:03.069Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:03.069Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=22
2001-01-01T00:04:03.071Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:03.072Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=23
2001-01-01T00:04:03.075Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:03.075Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=24
2001-01-01T00:04:03.086Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4af4
2001-01-01T00:04:03.086Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x8101 (sinking): rip=0x4aff count=25
2001-01-01T00:04:03.897Z| vcpu-0| I120: BIOS-UUID is 56 4d 46 3f e6 75 79 d9-e0 67 11 90 f6 d2 2e 5a
2001-01-01T00:04:04.797Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:04.798Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=26
2001-01-01T00:04:04.812Z| vcpu-0| I120: GetMSR: MSR[0x11e] (read as 0x708101): rip=0x4a13
2001-01-01T00:04:04.812Z| vcpu-0| I120: SetMSR: unknown MSR[0x11e]:=0x708001 (sinking): rip=0x4a18 count=27
2001-01-01T00:04:06.119Z| vcpu-0| I120: SVGA enabling SVGA
2001-01-01T00:04:17.919Z| mks| I120: SVGA disabling SVGA
2001-01-01T00:04:18.498Z| vcpu-0| I120: Unknown int 10h func 0x0000
2001-01-01T00:04:18.500Z| vcpu-0| I120: GetMSR: unknown MSR[0x1a0] (read as zero): rip=0x73526a count=1
2001-01-01T00:04:18.502Z| vcpu-0| I120: Intel VT enabled.
2001-01-01T00:04:18.582Z| vcpu-0| W110: MONITOR PANIC: vcpu-0:ASSERT public/x86vtinstr.h:252
2001-01-01T00:04:18.582Z| vcpu-0| I120: Core dump with build build-2068190
2001-01-01T00:04:18.582Z| vcpu-0| W110: Writing monitor corefile "/vmfs/volumes/3a4fc99d-f6437147-561e-00aaaaaaaaab/123/vmmcores.gz"
2001-01-01T00:04:18.628Z| vcpu-0| I120: Counting amount of anonymous memory
2001-01-01T00:04:18.632Z| vcpu-0| I120: Total Count of Anon Pages and CR3 pages 6976
2001-01-01T00:04:18.632Z| vcpu-0| W110: Dumping core for vcpu-0
2001-01-01T00:04:18.632Z| vcpu-0| I120: CoreDump: dumping core with superuser privileges
2001-01-01T00:04:18.632Z| vcpu-0| I120: VMK Stack for vcpu 0 is at 0x4123a7bd5000
2001-01-01T00:04:18.632Z| vcpu-0| I120: Beginning monitor coredump
2001-01-01T00:04:19.362Z| vcpu-0| I120: End monitor coredump
2001-01-01T00:04:19.363Z| vcpu-0| W110: Dumping extended monitor data
2001-01-01T00:04:19.695Z| vcpu-0| I120: CoreDump: ei->size 30707712 : len = 30707712
2001-01-01T00:04:31.781Z| vcpu-0| W110: A core file is available in "/vmfs/volumes/3a4fc99d-f6437147-561e-00aaaaaaaaab/123/vmx-debug-zdump.001"
2001-01-01T00:04:31.793Z| vcpu-0| I120: Msg_Post: Error
2001-01-01T00:04:31.793Z| vcpu-0| I120: [msg.log.error.unrecoverable] VMware ESX unrecoverable error: (vcpu-0)
2001-01-01T00:04:31.793Z| vcpu-0| I120+ vcpu-0:ASSERT public/x86vtinstr.h:252
2001-01-01T00:04:31.793Z| vcpu-0| I120: [msg.panic.haveLog] A log file is available in "/vmfs/volumes/3a4fc99d-f6437147-561e-00aaaaaaaaab/123/vmware.log".
2001-01-01T00:04:31.793Z| vcpu-0| I120: [msg.panic.requestSupport.withoutLog] You can request support.
2001-01-01T00:04:31.793Z| vcpu-0| I120: [msg.panic.requestSupport.vmSupport.vmx86]
2001-01-01T00:04:31.793Z| vcpu-0| I120+ To collect data to submit to VMware technical support, run "vm-support".
2001-01-01T00:04:31.793Z| vcpu-0| I120: [msg.panic.response] We will respond on the basis of your support entitlement.
2001-01-01T00:04:31.793Z| vcpu-0| I120: ----------------------------------------
clean install ubuntu 14.04
this try with ESXI6
cat /sys/module/kvm_intel/parameters/nested
cat /sys/module/kvm_intel/parameters/ept
cat /sys/module/kvm/parameters/ignore_msrs
all Y
latest update for
Compiled against library: libvirt 1.3.2
Using library: libvirt 1.3.2
Using API: QEMU 1.3.2
Running hypervisor: QEMU 2.5.50
Running against daemon: 1.3.2
cat /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1
cat /etc/modprobe.d/kvm-intel.conf
options kvm-intel nested=y ept=y
i am testing all supported CPU
Broadwell
Broadwell-noTSX
Haswell
Haswell-noTSX
qemu32
kvm32
cpu64-rhel5
cpu64-rhel6
kvm64
qemu64
Conroe
Penryn
Nehalem
Westmere
SandyBridge
IvyBridge
no one work normal....
cpu host ( not work )
cpu-passthrough ( via virt-manger not work too )
all time stuck in chipset initzialization
1. Clean VM witout HDD minuim hardware, only CD\ROM with Ubuntu 14.04
UNREAL SLOW WORK, really 20-30 min for "TRY UBUNTU" in normal case, this work instantly ( for this type installation all work in RAM )
2. External .OVF template ASAv ( cisco virtual machine ) all time stuck, all night wait, no work at all all time stuck after initrd\kernel initsialization
3. Windows load from CD, after frist menu stuck
really dont know, what i doing wrong, or why other ppl can work with nested esxi under kvm ))
in my case i am build lab, and very important run ESXI in single machine with QEMU
perfomance dont worry, but it this case or stuck or unreal slow work....
i am have vmmcores but dont know how i can read this dump
The following error:
2001-01-01T00:04:31.793Z| vcpu-0| I120: [msg.log.error.unrecoverable] VMware ESX unrecoverable error: (vcpu-0)
2001-01-01T00:04:31.793Z| vcpu-0| I120+ vcpu-0:ASSERT public/x86vtinstr.h:252
indicates that there is a problem with our software using KVMs virtual VT HW. I asked around and was told by the folks who have tried it that this does not quite work yet. Can you upload the vmmcores.gz file and the corresponding vmware.log? For now you can run 32-biit guests using software virtualization if you add the following options to your 32-bit VMs config file:
monitor.virtual_exec = "software"
monitor.virtual_mmu = "software"
this options not work at all
i am try on
Ubuntu 14.04 ( IvyBridge Xeon )
RHEL7 ( IvyBridge Xeon )
Ubuntu 14.04 ( Core i7 4770 )
cpu host not work at all
nested ESXi not work at all
same problem on both CPU
don't know why
mb need use older QEMU with patch.... but... dont know really where i have problem
I looked into the following error and found it was fixed last November by Jim:
2001-01-01T00:04:18.502Z| vcpu-0| I120: Intel VT enabled.
2001-01-01T00:04:18.582Z| vcpu-0| W110: MONITOR PANIC: vcpu-0:ASSERT public/x86vtinstr.h:252
2001-01-01T00:04:18.582Z| vcpu-0| I120: Core dump with build build-2068190
I have requested the fix to be backported to the next patch of ESX 6.0.
Thanks,
Doug
Hi,
trying to set up ESXi inside of proxmox. And got tha attached screenshot.
Wich features I need to enable/disable?
Is it real? As I know Proxmox is actually guy for KVM.
Here is the answer to my question.
For the all who wants to build this: http://www.marksei.com/install-esxi-nested-inside-proxmox-ve/