You're trying to understand the problem from the most difficult end, CPU resources. Approach it from Memory instead and it's a bit easier to understand.
The host has x amount of memory, and the host reserves y amount of memory for it's own nefarious purposes (which by the way follows the formula a MB initially and the b MB for every VM that's powered on). If you don't allow over-subscription/allocation, you assign a total of x - y to your VMs, and every VM can always get the memory it's allocated. This is however very inefficient, so you generally over-allocate memory to your machines. If your policy states that 3:1 is ok, then you can assign 3*(x-y) to your VMs (or practically 3x as the amount VMware ESXi need is nothing compared to total memory).
If the host runs out of memory, it first tries ballooning (which starts at 95% utilization), where it via VMtools starts asking the guest OS for memory, and then it deletes all the pages it gets (as whatever the OS gives the balloon driver is not being currently used by the OS, so it's safe to delete). If this doesn't help, ESXi does what every other operating system does when it runs out of memory, it starts swapping memory to disk. ESXi though does it per VM (as it has to take shares, reservations and limits into account).
vCPUs work a bit differently, as it's not technically a resource the way memory and disc is (I mean, a 20 core 2 GHz CPU isn't the same as a 1 core 40 GHz CPU now is it?). First of all, Hyperthreading doesn't give you twice the amounts of cores, official VMware figures is that HT gives you abouts a 10-30% performance gain. Secondly, vCPUs are multiplexed, so the scheduler gives out time slots on the pCPUs for the different vCPUs. So if you have an 8 core CPU on your host, it can run either 8 1-vCPU VMs at the same clock cycle, or 1 8-vCPU. This gives you a problem that on a highly loaded host, you might see performance drops by assigning more cores to a VM, as the scheduler has a hard time finding a large enough time slot to run the larger VM.
Shares, reservations and limits are a way around some of this.
Reservations are a guarantee, aka. if you give a VM a reservation of 500 MHz, the hypervisor will give it 500 MHz regardless of what's happening on the host. If the reservation isn't being used at the moment though, it will give the resources to another VM for the time being (so you're never wasting resources by setting reservations).
Limits on the other hand set a hard "roof" or cap for a resource. This is useful if you want a VM to believe that it has more resources than it actually can get. This has really no practical use (it was used in earlier ESXi versions as a way to get hot-add of memory by removing the limit) today other than tricking software vendors, and it's not very useful at that either. Limits are enforced ruthlessly, so if you assign a limit of 1 GB memory to a machine with 4 GB configured, that machine will never get more than 1 GB of physical memory.
Shares are a bit different, they allow you to "weight" the "importance" of a particular VM when it comes to different resources. In ESXi every VM is equal, and when competing for resources 2 VMs with 1000 shares each will get the resource 50% of the time. By altering the shares to say 3000 vs 1000 you can change the divide to 75%/25%. Shares are only used when VMs are competing for resources.
So, to actually answer your question, over-allocation means that you assign for example 32 vCPUs (2 GHZ each) to your VMs on a host that has a 16 core CPU without HT. Your CPU has 32 GHZ "available" but you have assigned 64 GHz to your VMs. This is 2 to 1 over-allocation/subscription. But if you know anything about servers it's that they are idle most of the time, so your host might not see more than 16 GHz of load. But, theoretically, this setup could produce a demand for 64GHz, which would bog down your host. HT isn't very useful for these calculations, but you could maybe add 25% on top of your "non-HT" GHz value to get some idea of it's benefits.
