1
I have a PostgreSQL-driven web service on two Debian servers, one a vhost running Linux kernel 3.2.0-2 with 2GB RAM and the other a dedicated server running Linux kernel 2.6.26-1 with 6 GB RAM. Although the application has a very small memory footprint, over the time the cache fills up to a certain amount at which processes gets swapped out increasing the system's load, finally run out of memory and eventually crash the kernel (same behavior on both systems, only different time periods). Before the crashes, free -m
reports something simliar to:
$ free -m
total used free shared buffers cached
Mem: 1978 1583 394 0 147 1208
-/+ buffers/cache: 227 1751
If I reboot the system, it runs smoothly for some time (often weeks or months, but sometimes only days) until the cache got filled up again. After the reboot, free -m
reports:
$ free -m
total used free shared buffers cached
Mem: 1978 452 1526 0 6 302
-/+ buffers/cache: 143 1835
It would be acceptable for me if the server slows down somewhat due to disk I/O if there are peaks due to excessive usage, but it is not acceptable that the system crashes then and when because disk cache memory is not reclaimed to processes if they temporarily need more memory.
So on one server I disabled swapping at all (since with swapping the load goes up to 300 - 400 (in sort of a chain reaction if processes are being killed) rendering the system unuseable even for a remote login to initiate a reboot. In addition I did set the kernel tuning parameter vm.swappiness=0
and vm.overcommit_memory=2
as recommended in the PostgreSQL High Performance Tuning guide to force the kernel to reclaim memory from the cache and avoid over-committing.
But this seems to not have the desired effect - the above output from free
did not show a shrinking cache at all while programs again begin to crash with no memory available
error messages (the output above from free
was produced just seconds after the first no memory available
messages appeared in the syslog after about two months of smooth operation).
Is there any chance to avoid this excessive caching to be able to use the memory for the applications rather than for disc I/O?
Increasing physical memory didn't help since on the 6GB-equipped system I get the same results sooner or later when more clients connect to the service (it's a hotspot auth server for free public use with an unknown number of users and I need to stick with the two servers for some time because of various reasons). On the vhost I first used 1GB, now 2GB - being the limit for my vhost - and all I got so far was an increase in cache storage, but not in memory for the processes. Is this the way its supposed to work on Linux? Just 12% of memory for processes? Or are the above suggestions from the HPT book incorrect?
Thanks in advance for any hint!
Hello Michael, yes, it seems so. I also thought that kernel cache memory should be freed if userland applications need more memory. But look at this output from free(1)
I did occasionaly note when the system became unresponsive again, which is a sign that the situation leads to the OOM killer getting kicked in sooner or later:
total used free shared buffers cached
Mem: 1978 981 997 0 142 638
-/+ buffers/cache: 200 1777
Swap: 0 0 0
I can see the free Mem
going down to zero while -/+ buffers/cache
still shows some high value before the systems eventually crashes. I immediately cleared the cache by sync
ing and writing 3 to /proc/sys/vm/drop_caches
resulting in kernel memory beeing freed:
total used free shared buffers cached
Mem: 1978 481 1496 0 1 283
-/+ buffers/cache: 196 1782
Swap: 0 0 0
So as a work-around I am clearing the cache manually from time to time. Unfortunately I cannot just update the OS on the production system, but have to change the systems before updates to ensure 99% availability. Testing kernels on our staging system is possible, but useless since the situation described will only appear after a certain time on our production system serving tens of thousands of login requests each day.
BTW, this is the output of free -m
for today:
total used free shared buffers cached
Mem: 1978 1107 871 0 146 753
-/+ buffers/cache: 207 1771
Swap: 0 0 0
If I understand correctly, applications and kernel uses about 190 to 227 MB, while buffers and cache consume the rest of the memory. I am sure that the problem is not related to some application since nothing changes if I restart the applications, but it helps to clear the cache manually.
Wait, you're saying that the OOM killer kicks in because kernel cache memory is using too much? That doesn't make sense; cache should be evicted when the system comes under memory pressure. If that's really the case, you've run into a kernel bug, and probably should file a bug report with whoever packages the kernel you're running. Are you sure it's not something else that is gobbling up memory? – a CVn – 2014-08-04T18:28:51.567
Please [edit] your question to include additional information. Comments are second-class citizens on Stack Exchange, as well as that they can become almost unreadable in cases like this (or worse). – a CVn – 2014-10-06T09:20:27.057
@MichaelKjörling I have the same issue with an embedded system IO operations, unfortunately no one will help me due to use of non-free third party software (tainted kernel). So I can confirm that topic author problem does happen. Problem still exists in kernel 3.3.8. – RoughTomato – 2016-06-24T12:38:45.130
@RoughTomato Linux 3.3.8 is ancient (specifically, from the first week of June 2012; the commit is dated June 1, the tarball is dated June 4). That version is four years old.
– a CVn – 2016-06-24T13:43:03.883@MichaelKjörling charms of working in embedded I guess. I'm working with a project that's years old but still supported and unfortunately can't be moved to newer linux which causes all sorts of problems (and headaches for me :( ). Still, hoping to find out whether somebody noticed that and maybe fixed it somehow in newer versions and maybe made my work a bit easier. I don't believe this problem exists on new 4.6.2 x86_64 which I'm currently running at home. Although it might be platform specific in my case (mips). If I ever figure it out I'll make sure to share! maybe it'll help. Cheers :) ! – RoughTomato – 2016-06-24T14:45:45.307