VMware has a tool that let's you migrate a VM from a hyper-v server to a ESXI one but my problem is that I want o remove Windows Server 2012 from a server and install ESXI on the same server, so I would like to back up my VM's before, but now just backup, backup in a ESXI format so I can use them on ESXI after installing it. Is there any way to do it?
Yes you can. You can pretty much use Free windows server backup using Vembu ImageBackup. you can install Vembu ImageBackup on the running OS, and take image of the whole server to a local Vembu BDR server. On restore, You have the option to convert to vmware supported file format while restoring your image file Then install ESXi on the baremetal server, and restore the vmdk file on to the newly installed ESXi server.
If you don't have target server ready before formating this with ESXI, you can use Vmdk2Phys
This will save your physical disk converted as VMDK to your external drive. After installing ESXI you can create new VM and can attch those VMDK to VM.
You can use starwind v2v converter. StarWind V2V Converter - Free Tool from StarWind! For me it did the job well when I used it for similar purposes.