Well, one answer is that pages are thrown out of physical ram (paged out) because they haven't been used lately, so using that memory for disk-cache makes more sense. (because that cache is used more often.) You could try
swapoff -a
Which turns off paging. (Actually, it unmounts all paging files/partitions.) That usually causes programs to crowd out the caches, but it might also make the Out-Of-Memory killer kill a few of your hungriest processes. (Like firefox.) When you're fed up, just do
swapon -a
(Notice I said "when", not "if"...)