Is what is stated in the title possible?
Alternatively is it possible to add a datastore through the VIX C API?
No, you must first move the virtual machine to a directory within the datastore before you can register it. VIX does not provide a way to move a virtual machine, but it shouldn't be too hard to move a directory in the OS of your choosing.
VixHost_RegisterVM() adds a VM to a datastore. It currently doesn't allow you to specify the datastore; it uses the first one it finds.
There's no way in Vix to create a new datastore.
Is it possible to add a vmx file outside a datastore to a datastore?
No, you must first move the virtual machine to a directory within the datastore before you can register it. VIX does not provide a way to move a virtual machine, but it shouldn't be too hard to move a directory in the OS of your choosing.