System thinks it is swapping, but isn't

Question

I have a linux server (Ubuntu 10.04) which has no swap space (swapon -l reports nothing, 0kb swap memory listed in /proc/meminfo). The beginning of the output of top (sorted by %mem) is reproduced below:

top - 04:18:28 up  7:31,  2 users,  load average: 0.93, 0.76, 0.71
Tasks:  25 total,   2 running,  23 sleeping,   0 stopped,   0 zombie
Cpu(s): 15.0%us,  0.8%sy,  0.0%ni, 84.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4195200k total,  4177028k used,    18172k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND
30075 admin     20   0 4026m 1.4g 9648 S   31 35.7 271:03.97 2.5g java
 7700 root      15   0  170m 9152 4360 S    0  0.2   0:00.96 161m apache2
 7730 www-data  15   0  171m 6268 1168 S    0  0.1   0:00.00 165m apache2
 7726 www-data  15   0  171m 6232 1108 S    0  0.1   0:00.01 165m apache2
 9520 www-data  15   0  171m 6228 1100 S    0  0.1   0:00.00 165m apache2

My confusion is how 4 GB of memory could be used with only about 40% of the usage reported (ps agrees with these numbers). Further, I am confused how more virtual memory could be allocated than resident when there is no swap space.

As reported, there is no free memory, apt-get failed due to a bad allocation. This too confuses me, as I thought the OOM killer would have sprung into action before this could happen.

Answer 1

The number in VIRT column doesn't mean all that memory is not actually used, it's just marked as allocated. Java programs tend to allocate lot of virtual space.

Sometimes this can be bit messy; by default Linux kernel allows programs to over-commit memory - that is, to allocate more memory than there's available in the system. This is kind of the same than e-mail providers give you gigabytes and gigabytes of free e-mail space, because they know/hope that not everybody actually use that much space, but only fraction of it.

How much over-commit is allowed? In typical distributions the default settings is to use vm.overcommit_memory=0 mode, which means Linux kernel uses some heuristics to determine that amount. Check the value you now have set with sysctl vm.overcommit_memory

If that is 0 and you want to change the kernel behaviour to be "Don't allow any over-commiting to happen", then add these lines to /etc/sysctl.conf:

vm.overcommit_memory=2
vm.overcommit_ratio=0

After you give command sudo sysctl -p the new settings will be loaded.

But what do those lines mean?

vm.overcommit_memory=2 changes the over-commit model from heuristics to user-definable value.

vm.overcommit_ratio=0 is the percentage of memory that is allowed to over-committed. Zero means no over-commit is allowed.

But, if you want to instead allow about 2 gigabytes of over-committing in your case, then

vm.overcommit_memory=2
vm.overcommit_ratio=50

would do that.