In essentiality, what swap gives you is protection against burst memory consumption - if some application of yours decides to allocate too much memory for your system's good (memory consumption approaches total available memory), what you are going to get is termination of the said application or (best case) or termination of other applications (not good at all), either way, in 99% cases, the application will not get to do whatever it was going to do. With swap, there is a possibility, that the said application will do whatever it intended to do, release the memory and the normal operation will be resumed. With no swap there is no such possibility. Can you really not spare 4Gb of your hard drive for swap? We definitely DO have swap enabled on our systems even though they have at the very least 32Gb RAM.
Generally, what is your problem with swap? The only "real" downside I see is, that you are shaving some gigabytes off of your usable disk space, but thats all there is about it. Or am I missing something?