What's using my swap (Ubuntu)?

Question

I have an Ubuntu 8.04 server which is running a database and a bunch of Java application servers. Its memory configuration and usage is:

             total       used       free     shared    buffers     cached
Mem:      16456176   15930028     526148          0      81372    9674196
-/+ buffers/cache:    6174460   10281716
Swap:      1951888     366100    1585788

I noticed that swap goes up every afternoon, and is released some time at night. The demand is not the same on all days, and starts at different times. So it's pretty random, except for the fuzzy "afternoon-night" boundary.

Load on this machine varies during the day. It's very low between midnight and 6-7 AM, much higher (but stable) until 6-8 PM, then dropping gradually.

Now I have the following questions:

1. How can I see which processes are using swap?
2. Why does it prefer to swap out rather than take some memory from cache?

Answer 1

The swap usage patterns you describe don't sound surprising. They're consistent with some permanently-running processes having rarely-used pages. During the day, due to the high activity, the rarely-used pages are almost always in the swap. At night, there's more room for them in RAM.

You can get a glimpse of how much memory of various kinds each process is using in top or htop. Neither show swap usage by default, but both can be configured to (top: press f and switch on the SWAP column; htop: press F2, add the NSWAP column). You can get more information about a particular process with cat /proc/12345/vmstat where 12345 is the process ID. Note that “how much swap a program is using” is not completely well-defined, as some pages are shared by several processes.

There are two major kinds of competitors for RAM: process memory (which can be swapped out) and disk caches (which can be re-read from a file). There is no reason to always prioritize process memory over disk caches: it's better to swap out a rarely-used portion of process memory than keep reading a file into memory again and again. The figures you give, with about half (say 30%–70%) of the memory devoted to disk caches, is typical for systems that have a reasonable amount of RAM for the tasks they're supposed to do.

Answer 2

With htop v1.01, I pressed "S" to add NSWAP column instead of "F2" (as Gilles suggested), then Columns > Available Columns > and F5 to add it.

Answer 3

Without polling and parsing the outputs of tools like vmstat, free and top, the best place to look might be crontabs of the root user or other users on the system. If the general load on the system spikes reliably at a certain time, chances are there is a process in cron that creates the need for resources. Otherwise you can always create a ghetto throwaway logging utility with the script utility, which just records everything that goes to the STDOUT.

So if I needed to do a one-off throwaway script to catch the output for later inspection, in one terminal I would type:

script /tmp/free.txt

and then

while (true); do date; free; sleep 30; done

and in another terminal

script /tmp/top.txt

and then

while (true); do date; top -n 1; sleep 30; done

and then the next morning, kill both script invocations and match the output of top and free

once again this is a ghetto approach but it sounds like you only need a one-off

you might also like to read the ubuntu swap guide

Answer 4

On StackOverflow, there is already an excellent answer on your first question ("Which process is actually using swap?").

https://stackoverflow.com/a/7180078/1442301

In a nutshell, tools like top or htop does not provide accurate information at all about swap usage. You should dive into the /proc folder (in the pseudo file /proc/$PID/smaps) to get more accurate information.

The post mentioned above contains a short shell script to get this information echoed in a nice way.