I was wondering if it was possible to install a mainframe OS, such as z/os, as a guest machine on VMWare Player
If it supports x86 (Intel) architecture you are on your own to get it to work. If it is NOT x86 compatible, then you can't virtualize it. Meaning if you can install it on your desktop / laptop you can probably stick it in a VM, it just may not be supported.