Google information on resource quotas on Linux. I'd also look to see what process is ballooning up to take up all that memory. You don't mention how much memory you have on that system, either; is it just a case where you need to get more memory for the server? Without watching the resource usage and seeing what is hitting what process, you don't know if it's just gradual memory usage taking it up or if there's a resource leak/application issue causing it.
How many users are hitting that server? What is the workload? How much memory does it have in it already? What errors, if any, are in the log just before, during, and after the memory issues?
I'd also point out that using resource limits/quotas will mean setting up a process to act as a watchdog to restart processes that are killed, and it's not very clean to do so, as it may force-kill processes and things that rely on disk i/o or databases probably won't like being killed off quite suddenly. You might want instead to focus on finding which process is the troublemaker and fix or isolate it. Worse, you may have an attack going on that is causing the memory usage to balloon, or an out of control cron job, something like that. Killing the processes or rebooting isn't fixing the problem, it's helping hide the problem.