I guess this comes down to how much you are willing to pay for your solution. Both ESXi and VMware Server are free products, but in their free versions there are limitations. With ESXi there is a limited hardware set that you can run, and with VMware Server you are not truly running an enterprise class virtualization hypervisor. You can license ESXi and use Virtual Center for management which will give you all of the benefits of ESX 3.5, but there is still the limited hardware set that your can use.
In your particular situation I would recommend a few solutions:
1. Install ESX3.5 and test to see if the same performance issue exists. You can evaluate ESX 3.5 for 60 days without having to license and that should give you enough time to determine if performance is acceptable. If the performance measures up then you will need to license the product.
2. Find hardware that is on the ESXi compatibility list and test for performance issues. If performance is acceptable you can use ESXi in the free capacity until you want to use the advanced features that come with licensing.
3. Migrate your VM's to VMware Server. I don't expect that performance will improve with this solution and you will have to deal with the overhead of the guest OS. Again I would not recommend running VMware Server in a production environment. It may be suitable for test/dev work but in the long run either of the ESX platforms will be more reliable.
Free software is nice and can be a fit in some situations, however when you reach a certain scale licensing the product to obtain the advanced features is beneficial and will utimately save time and money.
