Memory used by unknown processes on OEL6 - without cache and slabcache

Question

I've searched through "Where is my Linux memory?" questions, but I couldn't find a similar scenario for my kind of leak:

OEL 6.3 x86_64 Linux with 6GB of RAM
without any of my main components running (2 tomcat, MySQL, Grizzly-based web container) there are around 2.3GB of memory eaten by some unknown part of the system
hardly afew MB filecache
hardly afew MB slab cache
even after reboot
system is a virtual host on VMWare

Here is a top output ordered by RES desc:

top - 15:57:26 up  3:26,  1 user,  load average: 0.27, 0.26, 0.76
Tasks: 192 total,   1 running, 191 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   5993276k total,  2257256k used,  3736020k free,    23772k buffers
Swap:  4063224k total,    89644k used,  3973580k free,    98960k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3236 root      20   0  123m 8384 1520 S  0.0  0.1   0:01.05 Xorg
 3373 gdm       20   0  369m 3840 2464 S  0.0  0.1   0:00.69 gdm-simple-gree
 3358 gdm       20   0  337m 2916 1940 S  0.0  0.0   0:05.30 gnome-settings-
16380 root      20   0  157m 2708 2192 S  0.0  0.0   0:00.01 su
20174 root      20   0  157m 2704 2192 S  0.0  0.0   0:00.01 su

free -m:

-/+ buffers/cache:       2083       3769

/proc/meminfo for slab:

MemTotal:        5993276 kB
MemFree:         3722140 kB
Slab:              50724 kB

Any hint on finding out whats eating that 2 gigs here is appreciated!

I'm pretty sure I know what's going on. Do you have access to the vSphere or vCenter Management for the host/cluster? — ewwhite, Aug 01 '14 at 14:34

score 1 · Accepted Answer · edited Apr 13 '17 at 12:14

1

One thing that many people miss, especially when running Tomcat and other Java-based applications on VMware, is the impact that this software has on VMware's memory allocation.

So first:

Do you have access to the vSphere or vCenter Management for the host/cluster?
Do you have the VMware tools installed?

I often see unexplained memory issues inside of Linux VMs under the following conditions:

Extreme RAM overcommitment at the host level.
Heavy memory ballooning on the host/cluster.
If there are RAM limits on the VM. (e.g. 6GB allocated to the VM, but with a 4GB host limit)
Bad resource pool/shares design.

Basically, you need to look for contention at the host/cluster level.

A super-terrible example of this:
vSphere education - What are the downsides of configuring VMs with *too* much RAM?

edited Apr 13 '17 at 12:14

Community

1

answered Aug 01 '14 at 14:43

ewwhite

194,921
91
434
799

Thanks a lot, it seems to be a reasonable cause, unfortunately I will only have access to the vSphere next week, so I will let you know if this was the cause. – TewWe Aug 04 '14 at 11:31
@TewWe What was the cause of the issues? – ewwhite Aug 12 '14 at 12:44
I was able to acces the env from vSphere and there were a 4G memory resource limit set on the VM. Raising it solved th problem, thank you! – TewWe Aug 12 '14 at 12:44
@TewWe Wonderful! Yeah, people really shouldn't use resource limits on virtual machines... especially Linux systems, as it causes this type of problem when the VM hits the limit. You're lucky it was only one virtual machine. I had to deal with it on *hundreds* of incorrectly configured VMs. – ewwhite Aug 12 '14 at 12:46

score 0 · Answer 2 · edited Apr 13 '17 at 12:14

try

grep VmPea /proc/*/status|sort -n -k+2 |tail

as written by polynomial in the link below

serverfault almost no free memory

You did read Serverfault Canonical question about Memory usage in linux right?

Additionally you might add threads in top via pressing H (at least for me on ubuntu 12.04)

For more debug:

search for ps_mem.py one Python memory script

With that you can see which process is actually using the RAM (private/shared) not 100% accurate but i guess good enough for your "problem."

There are a lot of other ways to actually detect an issue but i suggest first checking if it is actually used by something and is not just buffered which is totally normal.

Thanks, but as I mentioned the memory does not seem to be mapped to any process, and your script is showing the virtual memory, not the reserved system memory - but still no sign of where the gigs are gone. The cause probably will be what ewwhite mentioned below. — TewWe, Aug 04 '14 at 11:34

Memory used by unknown processes on OEL6 - without cache and slabcache

2 Answers2