There are two factors involved here. In order:
CPU Emulation
It's possible that a mismatch between the host and the guest will force the VM to use CPU emulation mode. I don't expect this happen, but depending on your hypervisor, operating system, and architecture, it's possible. You want to avoid this at all costs. The other part of this that you want your VM hypervisor to also match up with your host system. So if you have 64bit Windows, look for a 64bit edition of VirtualBox, VMWare, etc. Once you're sure this is not a problem, it comes down to the next factor:
RAM
If you'll generally be giving your guests less than 4GB of RAM, than stick with 32bit for now. There are other changes in the move to x64 also, but for the most part those won't help your VMs as much as the RAM saved from a smaller pointer/word size will, because VMs have more I/O issues and need the RAM more to help save going to disk as often. Don't forget to allow for growth here. You don't want to have to rebuild a VM from scratch. For example, I'd probably set up a VM acting as a database server as 64bit, even if right now it only has 2GB of RAM. I'd do the same for a Windows 7 test image. But if I had a machine that exists for something like testing a web site under IE7 on Windows XP, I'd leave that at 32bit.
As a final note, you mention that you have hyperthreading turned off. You probably want that on if you'll be doing a lot of VM work. The reason you sometimes want to turn this off is that it gives more cache per core. Most people with quad core processors don't even use all the cores a lot of the time, and additional core provided by hyperthreading just aren't that powerful because of conflict issues in the pipeline where the two instructions happening in the same physical might be related in some way. Thus, the gain from larger cache sizes on few cores can sometimes outweigh the gain from additional weak cores. VMs, however, can use those fake extra cores and be sure of fairly clean pipelines with few of the issues that normally hold this feature back.
I've run both and I never seem to notice any significant preformance difference, I guess if you want to give the vm more than 4gigs then the 64 bit might be faster. However I have never had a need for a vm with more than 4gigs of RAM that's not on bare metal hyper visor. – Supercereal – 2011-03-31T16:01:55.850