While both ESX Server and VMware Server are server virtualization products, the difference is that VMware ESX installs and runs on the bare metal of a physical server where as VMware Server needs a base operating system. In other words, VMware ESX has a type 1 hypervisor where as VMware Server has a type 2 hypervisor.
You will obtain must better performance from ESX Server as it has much less overhead. ESX Server also has many features available such as VMFS, VMotion, VMHA, and DRS. On the other hand, ESX Server is also a commercial product that must be purchased where as VMware Server is a free product. VMware Server is an excellent option to choose to slowly migrate to server consolidation at a low cost. VMware Server is also an excellent way to learn about virtualization as well as a way to run multiple operating systems on your desktop PC, at no cost.
For more information on this topic, see Choosing between VMware Server and ESX Server