I asked why you needed such a disparate set of qualities in a server... Something will be missing.
(Is this for a VMWare host server? NAS storage server? It's not clear.)
Small is possible, but performance and expansion suffer.
Quiet is possible, but size or RAM capacity may not be where you wish.
It's easy to find systems that meet your RAM specs. But if you're looking for something that can support 32GB of RAM, you probably care about performance more than the size of the system.
A serious recommendation would be something like an HP 300-series tower. ML350 and ML370 are whisper-quiet. I use one at home. These are no-compromises servers that may be larger than you want, but are pro-recommended.
If you need small and low power, but can live with smaller CPU and RAM capacity, buy an HP Microserver - http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-4237916-4237918-4237917-4248009.html?dnr=1
Maybe the right answer lies in between the two, but I'm more curious about the reasoning for your requirements.