Running VMware ESX inside a virtual machine is a great way to experiment with different configurations and features without building out a whole lab full of hardware and storage. It is pretty common to do this on VMware Workstation nowadays — the first public documentation of this process that I know of was published by Xtravirt a couple of years ago.
But what if you prefer to run ESX on ESX instead of Workstation?
You may be pleased to know that the GA build of ESX 4 allows installing ESX 4 as a virtual machine as well as powering on nested virtual machines — VMs running on the virtual ESX host.  You can even VMotion a running virtual machine from the physical ESX to a virtual ESX — on the same physical server!
The extra tweaks to make it all work are minimal, and I will show you how without even opening up a text editor.
After installing ESX 4 onto your real hardware, configure as desired and enable promiscuous mode on a vSwitch:
Create a new VM with the following guidance (choose “Custom”):
  • Virtual Machine Version 7
  • Guest OS: Linux / Red Hat Enterprise Linux 5 (64-bit)
  • 2 VCPUs, 2GB RAM
  • 2 NICs – e1000
  • LSI Logic Parallel
  • New disk – reasonable size
After you have the VM ready, simply attach a VMware ESX 4 ISO p_w_picpath, power on, and install ESX as a guest OS.
After installation, add the new virtual ESX to vCenter 4 and create a new VM.
If you do not need to run VMs on your virtual ESX, you can stop there.  However, if you try to power on that nested VM, you will see the following error:
To prevent this, just one tweak is needed:
  • Shut down the virtual ESX VM
  • Click Edit Settings
  • Click the Options tab
  • Click Advanced / General / Configuration Parameters…
  • Click Add Row
  • For the Name/Value enter: monitor_control.restrict_backdoor / TRUE
The above procedure is just an alternative to hand-editing the .vmx file — if you prefer to do it that way, feel free.
Now you are ready to power your virtual ESX VM back on, as well as the nested VMs.  This capability should come in handy as you start investigating the new features of vSphere 4.
You may be interested in this related post where a VM is migrated between the physical and virtual ESX hosts.