How to force the Linux kernel reclaiming cache memory rather than using swap?

Question

I have a PostgreSQL-driven web service on two Debian servers, one a vhost running Linux kernel 3.2.0-2 with 2GB RAM and the other a dedicated server running Linux kernel 2.6.26-1 with 6 GB RAM. Although the application has a very small memory footprint, over the time the cache fills up to a certain amount at which processes gets swapped out increasing the system's load, finally run out of memory and eventually crash the kernel (same behavior on both systems, only different time periods). Before the crashes, free -m reports something simliar to:

$ free -m
             total       used       free     shared    buffers     cached
Mem:          1978       1583        394          0        147       1208
-/+ buffers/cache:        227       1751

If I reboot the system, it runs smoothly for some time (often weeks or months, but sometimes only days) until the cache got filled up again. After the reboot, free -m reports:

$ free -m
             total       used       free     shared    buffers     cached
Mem:          1978        452       1526          0          6        302
-/+ buffers/cache:        143       1835

It would be acceptable for me if the server slows down somewhat due to disk I/O if there are peaks due to excessive usage, but it is not acceptable that the system crashes then and when because disk cache memory is not reclaimed to processes if they temporarily need more memory.

So on one server I disabled swapping at all (since with swapping the load goes up to 300 - 400 (in sort of a chain reaction if processes are being killed) rendering the system unuseable even for a remote login to initiate a reboot. In addition I did set the kernel tuning parameter vm.swappiness=0 and vm.overcommit_memory=2 as recommended in the PostgreSQL High Performance Tuning guide to force the kernel to reclaim memory from the cache and avoid over-committing.

But this seems to not have the desired effect - the above output from free did not show a shrinking cache at all while programs again begin to crash with no memory available error messages (the output above from free was produced just seconds after the first no memory available messages appeared in the syslog after about two months of smooth operation).

Is there any chance to avoid this excessive caching to be able to use the memory for the applications rather than for disc I/O?

Increasing physical memory didn't help since on the 6GB-equipped system I get the same results sooner or later when more clients connect to the service (it's a hotspot auth server for free public use with an unknown number of users and I need to stick with the two servers for some time because of various reasons). On the vhost I first used 1GB, now 2GB - being the limit for my vhost - and all I got so far was an increase in cache storage, but not in memory for the processes. Is this the way its supposed to work on Linux? Just 12% of memory for processes? Or are the above suggestions from the HPT book incorrect?

Thanks in advance for any hint!

Hello Michael, yes, it seems so. I also thought that kernel cache memory should be freed if userland applications need more memory. But look at this output from free(1) I did occasionaly note when the system became unresponsive again, which is a sign that the situation leads to the OOM killer getting kicked in sooner or later:

             total       used       free     shared    buffers     cached
Mem:          1978        981        997          0        142        638
-/+ buffers/cache:        200       1777
Swap:            0          0          0

I can see the free Mem going down to zero while -/+ buffers/cache still shows some high value before the systems eventually crashes. I immediately cleared the cache by syncing and writing 3 to /proc/sys/vm/drop_caches resulting in kernel memory beeing freed:

             total       used       free     shared    buffers     cached
Mem:          1978        481       1496          0          1        283
-/+ buffers/cache:        196       1782
Swap:            0          0          0

So as a work-around I am clearing the cache manually from time to time. Unfortunately I cannot just update the OS on the production system, but have to change the systems before updates to ensure 99% availability. Testing kernels on our staging system is possible, but useless since the situation described will only appear after a certain time on our production system serving tens of thousands of login requests each day.

BTW, this is the output of free -m for today:

             total       used       free     shared    buffers     cached
Mem:          1978       1107        871          0        146        753
-/+ buffers/cache:        207       1771
Swap:            0          0          0

If I understand correctly, applications and kernel uses about 190 to 227 MB, while buffers and cache consume the rest of the memory. I am sure that the problem is not related to some application since nothing changes if I restart the applications, but it helps to clear the cache manually.

Answer 1

Kernel 3.15 add more control over page cache. Many application servers does not require heavy disk io. If that's the case, just add a cron job doing:

    echo 1 > /proc/sys/vm/drop_caches