I do second Tero Kikanen's answer. It looks very much like your system is doing exactly what it is supposed to do. It makes best possible use of your RAM (whatever RAM is not useful elsewhere gets used as cache, reducing disk access). Any data in RAM not used for a while gets swapped out, so you have more cache available, while some never-to-be-used-after-boot binary is sitting in your swap partition waiting to get swapped in later - which might happen only tomorrow or next week.
There are situations, though, where ths doesn't work quite well. For example a nearly idling machine transfers more data from one disk to another, than it has RAM available. Any binaries and their data not being used during that disk-to-disk transfer will get swapped out. Not a real problem, most of the time, but an inconvenience, when you return to your computer and want to continue browsing the web. Your browser might appear to hang for 10 or more seconds, and your productive but bored web server will take a few seconds until all it's binaries and data are back to RAM.
On a busy server, though, this doesn't happen, because before half of your cache is filled with data-being-copied, your binaries are used again, your cached data is used again, and your data-being-copied gets older than your actively used data. So the copies are dropped from cache, not your productively used data.
So don't empty your buffers and caches, you would slow down your machine.
TomTomTom