Mac rebooting with GPU passthrough

[needshelp]

I'm in the process of enabling GPU passthrough on a 10.8.5 partition on a EsXi 5.1.  I'm able to successfully enable it on my Windows machine but I'm having issues enabling GPU passthrough on the Mac.  I've enabled GPU passthrough on it and everytime it boots it goes to the apple logo start up screen and then reboots itself continuously.  I even tried adding the pciHole.start and pciHole.end since its using 6GB of ram but it keeps rebooting when it tries to log on.  If I take out the gpu passthrough pci-e card then the vm would start up fine.  I'm hoping someone with more knowledge on this issue can shine some light.

The GPU is a Quadro K5000 tested on a MacPro4,1

Windows VM works fine on it with GPU passthrough enabled but Mac can't boot pass the startup apple logo screen.

[Linjo]

Can you attach (not post) the vmware.log and the vmx-file?

[needshelp]

I just attached the files.

[Linjo]

Try to modify the pcihole.start to 2048 and remove the pcihole.end

Was it any difference from when you added the pcihole-parameters? Its not totally clear in your description.

You could also try to add this parameter in the vmx-file:

pciPassthru0.msiEnabled = FALSE

// Linjo

[needshelp]

There wasn't any difference when I changed the pcihole parameters.  I'll try the 2048 plus the other parameters and get back to you in a min.

[needshelp]

So I removed the pcihole.end and added pciPassthru0.msiEnabled = "FALSE".  Also changed pciHole.start = "2048" but still no luck.

[Linjo]

Could you also add "pciPassthru1.msiEnabled = "FALSE"

Any way you upgrade to ESX 5.5 and hardware-version 10? (Not sure that this will make any difference but could not hurt to try if nothing else)

// Linjo

[needshelp]

Adding the passthru1.msienabled line didn't help.  I'm not sure if I can risk updating the esxi box currently since I have production VMs running on the machine. 

[Linjo]

Ensure that GPUs are mapped below the 4G boundary by disable your server’s SBIOS option that controls 64-bit memory-mapped I/O support

This option may be labeled “Enable >4G Decode”, “Enable 64-bit MMIO”, “Above 4G Decoding”, it should be set to “Disabled”

[needshelp]

I'm running the esxi host machine on a Mac Pro 4,1.  I'm not exactly sure how to get into the SBIOS.  Correct me if I'm wrong but the Mac host would be running EFI so I didn't know if there is any sort of SBIOS option.  Can you clarify on how to get to this setting?  A quick google search didn't help me unfortunately. 

[Linjo]

Of course, sorry for that. Was digging through some of my troubleshooting notes and pasted in that sentence without thinking.

I have not other ideas at the moment, I hope that someone else have.

// linjo

[needshelp]

Just an update:  I went back and reinstalled 10.6.8 Server and it works with GPU passthrough.  However installing a fresh version of 10.8 did not work (keeps rebooting).  Has there been any update to vmware esxi that addressed this issue?

[Linjo]

Hmm, interesting. We are adding support as soon as we can but 10.8 should be supported with esxi 5.1

Could also be something in 10.6.8 vs 10.8 or in the K5000 driver

Maybe trying an update to 5.5...

Here are some links:

Mac OSX | Guest OS Install Guide - VMware Blogs

VMware Documentation for OS X 10.8

[needshelp]

Just updated to EsXi 5.5 but the problem still persists.

Has anyone gotten GPU passthrough to work with an NVIDIA card using a guest OSX 10.8 with EsXi 5.5?

Can someone at vmware try this on their end to see if it is actually working correctly?  The reason why I'm asking is because I've tried it with 3 different cards and all 3 fails to boot when using 10.8.  Surprisingly 10.6. and 10.7 loads, but unfortunately for me our production is going to need 10.8.  I'm hoping vmware can fix this issue or give me a workaround that would work.