Hello,
I'm trying to boot a Mac VM actually running in VMware Workstation Pro 17.01.
I tried two avenues:
first I powered it on to firmware and then create a boot option into recovery mode. I tried a few options among the <.diagnosis> options available, but none worked. (It booted into normal OS, no matter what changes I made).
Second I modified the vmx file to include there a directive to boot into recovery (one line at the end of the file). In this case, the MacOS VM falls into a loop at the time of boot and remains there until powered off. (a message is displayed, but the loop is too fast and it can't be read).