5

We're in the middle of a project to move our infrastructure from a co-lo situation into Amazon EC2 and we've noticed some weird memory characteristics of the processes in our setup. Without going into too much detail about the specifics of our processes, we've noticed that on our EC2 instances "top" will show processes using a lot of swap space -- in fact, much greater than the amount of available swap or (if you add it all up) more than the available disk.

Here's a sample top output:

Mem:   7136868k total,  5272300k used,  1864568k free,   256876k buffers
Swap:  1048572k total,        0k used,  1048572k free,  2526504k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND                                                                            
 4121 jboss     20   0 5913m 603m  14m S  0.7  8.7   3:59.90 5.2g java                                                                               
22730 root      20   0 2394m 4012 1976 S  2.0  0.1   4:20.57 2.3g PassengerHelper                                                                    
20564 rails     20   0 2539m 220m 9828 S  0.3  3.2   0:23.58 2.3g java                                                                               
 1423 nscd      20   0  877m 1464  972 S  0.0  0.0   0:03.89 876m nscd

You can see, for instance, that jboss is reportedly using 5.2 gigs of swap space which is definitely impossible since there's only 1G allocated and none is being used (probably because there's still 1.8G of RAM free).

And here's the results of uname -a:

Linux xxx.yyy.zzz 2.6.35.14-106.53.amzn1.x86_64 #1 SMP Fri Jan 6 16:20:10 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

We're running an AMI based off of the default Amazon Linux AMI (Amazon Linux AMI release 2011.09, so some RHEL5 and RHEL 6) with not too many customizations and definitely no kernel-level customizations.

Something here tells me that on this particular kernel/distribution, the reporting of swap or maybe even total memory usage isn't what it appears to be...

Any help would be appreciated!

rusty
  • 163
  • 6

1 Answers1

5

In fact, jboss is using 5.9GB of virtual memory and no swap space. The top tool uses a broken formula to calculate what it erroneously reports as swap space. It is in fact the result of subtracting the resident set size from the address space size. This is a boneheaded thing to do since one is a measure of virtual memory and the other is a measure of physical memory. So it's not entirely clear what the result is even a measure of at all. The number is as meaningless as the missing dollar in the old riddle.

(Actually, it's not completely meaningless. It's the maximum amount of swap space the program's current mappings could need if they were all dirtied and the program's resident set size didn't change. That's really, really close to meaningless though it doesn't take into account whether those mappings are even writable at all.)

David Schwartz
  • 31,215
  • 2
  • 53
  • 82
  • 4
    The OP's problem is that the column that purports to show "swap" (far right hand side) is actually showing no such thing (it's actually just VIRT-RSS). That's where the confusion will be coming from, and is one of the things that really fries my grits about top. – womble Mar 20 '12 at 05:44
  • @womble: You're right. I never noticed that. It seems the misunderstanding is on the part of the authors of `top`, not the OP. – David Schwartz Mar 20 '12 at 09:13
  • 1
    Yarp. I have no idea what the top authors were thinking there, and I don't comprehend the rationale of including it in the default set of fields, as it is on some Linux distros. – womble Mar 20 '12 at 09:47
  • Thanks David, I am sure this explains what we're seeing! – rusty Mar 22 '12 at 15:20