Linux top command. Memory usage

Question

I am testing my web server with Jmeter. I launch 40 users test, then dump top command. What i see, is 40 (+1 host) apache processes. Each process uses appr. 7mb of RES memory. But 7*40 is 280 mb of memory. But top shows that there are 508mb total and 345mb free. So only 163mb used... Why i have such strange stuff?

top - 04:49:24 up 1 day, 10:49,  1 user,  load average: 0.28, 0.18, 0.16
Tasks: 107 total,   2 running, 105 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.4%us,  0.4%sy,  0.0%ni, 97.6%id,  0.5%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    508132k total,   162428k used,   345704k free,    28340k buffers
Swap:   916476k total,    21800k used,   894676k free,    63480k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 9816 www-data  20   0 69232 7260 3232 S  1.9  1.4   0:00.69 apache2            
 9890 www-data  20   0 69232 7260 3232 S  1.9  1.4   0:00.06 apache2            
 9900 www-data  20   0 69232 7260 3232 S  1.9  1.4   0:00.04 apache2            
 9906 www-data  20   0 69232 7256 3232 S  1.9  1.4   0:00.04 apache2            
 9908 www-data  20   0 69232 7256 3232 S  1.9  1.4   0:00.06 apache2            
    1 root      20   0  2836  760  460 S  0.0  0.1   0:01.50 init               
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.01 kthreadd           
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.99 ksoftirqd/0        
    4 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0        
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/0         
    6 root      20   0     0    0    0 S  0.0  0.0   0:04.20 events/0           
    7 root      20   0     0    0    0 S  0.0  0.0   0:00.00 cpuset             
    8 root      20   0     0    0    0 S  0.0  0.0   0:00.00 khelper            
    9 root      20   0     0    0    0 S  0.0  0.0   0:00.00 netns              
   10 root      20   0     0    0    0 S  0.0  0.0   0:00.00 async/mgr          
   11 root      20   0     0    0    0 S  0.0  0.0   0:00.00 pm                 
   12 root      20   0     0    0    0 S  0.0  0.0   0:00.45 sync_supers        
   13 root      20   0     0    0    0 S  0.0  0.0   0:00.62 bdi-default        
   14 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/0      
   15 root      20   0     0    0    0 S  0.0  0.0   0:05.89 kblockd/0          
   16 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kacpid             
   17 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kacpi_notify       
   18 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kacpi_hotplug      
   19 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ata_aux            
   20 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ata_sff/0          
   21 root      20   0     0    0    0 S  0.0  0.0   0:00.00 khubd              
   22 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kseriod            
   23 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kmmcd              
   25 root      20   0     0    0    0 S  0.0  0.0   0:00.08 khungtaskd         
   26 root      20   0     0    0    0 S  0.0  0.0   0:08.30 kswapd0            
   27 root      25   5     0    0    0 S  0.0  0.0   0:00.00 ksmd               
   28 root      20   0     0    0    0 S  0.0  0.0   0:00.00 aio/0              
   29 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ecryptfs-kthrea    
   30 root      20   0     0    0    0 S  0.0  0.0   0:00.00 crypto/0           
   35 root      20   0     0    0    0 S  0.0  0.0   0:00.00 pciehpd            
   37 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_0          
   38 root      20   0     0    0    0 S  0.0  0.0   0:00.02 scsi_eh_1          
   41 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kstriped           
   42 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kmpathd/0          
   43 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kmpath_handlerd    
   44 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksnapd             
   45 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kondemand/0        
   46 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kconservative/0    
  239 root      20   0     0    0    0 S  0.0  0.0   0:03.95 mpt_poll_0         
  240 root      20   0     0    0    0 S  0.0  0.0   0:00.00 mpt/0              
  241 root      20   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_2          
  258 root      20   0     0    0    0 S  0.0  0.0   0:05.60 jbd2/sda1-8        
  259 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ext4-dio-unwrit    
  304 root      20   0  2392   68   68 S  0.0  0.0   0:00.04 upstart-udev-br    
  306 root      16  -4  2440   72   68 S  0.0  0.0   0:00.06 udevd              
  414 root      18  -2  2328   64   60 S  0.0  0.0   0:00.00 udevd              
  415 root      18  -2  2328   64   60 S  0.0  0.0   0:00.00 udevd              
  518 root      20   0     0    0    0 S  0.0  0.0   0:02.87 vmmemctl           
  526 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kpsmoused          
  556 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kgameportd         
  618 syslog    20   0 33976  440  440 S  0.0  0.1   0:00.83 rsyslogd           
  689 root      20   0  1856   84   80 S  0.0  0.0   0:00.00 getty              
  693 root      20   0  1856   84   80 S  0.0  0.0   0:00.00 getty              
  697 root      20   0  1856   84   80 S  0.0  0.0   0:00.00 getty              
  698 root      20   0  1856   84   80 S  0.0  0.0   0:00.00 getty              
  701 root      20   0  1856   84   80 S  0.0  0.0   0:00.00 getty              
  703 memcache  20   0 54192  128  128 S  0.0  0.0   0:07.77 memcached          
  705 root      20   0  2456  268  204 S  0.0  0.1   0:00.42 cron               
  706 daemon    20   0  2316    0    0 S  0.0  0.0   0:00.00 atd                
  722 root      20   0  5640  360  256 S  0.0  0.1   0:00.53 sshd               
  753 mysql     20   0  153m  456   32 S  0.0  0.1   0:20.57 mysqld             
 9280 root      20   0  2780 1408 1064 S  0.0  0.3   0:00.05 login              
 9292 zim32     20   0  8828 6068 1536 S  0.0  1.2   0:00.26 bash               
 9324 root      20   0  7268 2968 2180 S  0.0  0.6   0:03.50 mc                 
 9326 root      20   0  6252 3544 1588 S  0.0  0.7   0:00.21 bash               
 9735 root      20   0     0    0    0 S  0.0  0.0   0:00.00 flush-8:0          
 9808 root      20   0 68892 8624 4828 S  0.0  1.7   0:00.15 apache2            
 9814 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:01.03 apache2            
 9827 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:00.97 apache2            
 9842 www-data  20   0 69232 7264 3236 S  0.0  1.4   0:00.40 apache2            
 9844 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:00.40 apache2            
 9870 www-data  20   0 69232 7264 3236 S  0.0  1.4   0:00.22 apache2            
 9872 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:00.23 apache2            
 9877 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:00.18 apache2            
 9878 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:00.18 apache2            
 9888 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:00.07 apache2            
 9889 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:00.06 apache2            
 9891 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:00.08 apache2            
 9892 www-data  20   0 69232 7256 3232 S  0.0  1.4   0:00.05 apache2            
 9893 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:00.06 apache2            
 9894 www-data  20   0 69232 7256 3232 S  0.0  1.4   0:00.08 apache2            
 9895 www-data  20   0 69232 7256 3232 S  0.0  1.4   0:00.06 apache2            
 9896 www-data  20   0 69232 7256 3232 S  0.0  1.4   0:00.04 apache2            
 9897 www-data  20   0 69232 7248 3228 S  0.0  1.4   0:00.06 apache2            
 9898 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:00.03 apache2            
 9899 www-data  20   0 69232 7260 3236 S  0.0  1.4   0:00.06 apache2            
 9901 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:00.04 apache2            
 9902 www-data  20   0 69232 7256 3232 S  0.0  1.4   0:00.04 apache2            
 9903 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:00.03 apache2            
 9904 www-data  20   0 69232 7256 3232 S  0.0  1.4   0:00.08 apache2            
 9905 www-data  20   0 69232 7256 3232 S  0.0  1.4   0:00.05 apache2            
 9907 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:00.05 apache2            
 9909 www-data  20   0 69232 7256 3232 S  0.0  1.4   0:00.08 apache2            
 9911 www-data  20   0 69232 7256 3232 S  0.0  1.4   0:00.04 apache2            
 9912 www-data  20   0 69232 7248 3228 S  0.0  1.4   0:00.04 apache2            
 9913 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:00.06 apache2            
 9914 www-data  20   0 69232 7260 3232 S  0.0  1.4   0:00.04 apache2            
 9915 www-data  20   0 69232 7260 3232 R  0.0  1.4   0:00.04 apache2            
 9916 www-data  20   0 69232 7256 3232 S  0.0  1.4   0:00.04 apache2            
 9917 www-data  20   0 69232 7256 3232 S  0.0  1.4   0:00.06 apache2            
 9918 www-data  20   0 69232 7248 3228 S  0.0  1.4   0:00.02 apache2            
 9919 root      20   0  2632 1068  816 R  0.0  0.2   0:00.02 top

score 6 · Answer 1 · answered Sep 25 '11 at 13:09

It's a case of top reporting memory usage per process as if it where the only process running. In reality all the apache processes have shared memory (common linked libraries), and this shared memory is being reported for each process so the total usage across all apache processes appears to be more than it really is.

There are numerous posts that discuss how to underdstand and interpret the output of top and ps.

process memory usage on Linux is a complex matter; you can't just run ps and know what is going on. This is especially true when you deal with programs that create a lot of identical children processes, like Apache. ps might report that each Apache process uses 10 megabytes of memory, when the reality might be that the marginal cost of each Apache process is 1 megabyte of memory

score 4 · Answer 2 · answered Sep 26 '11 at 06:53

4

You forgot to calculate the SHR, or Shared portion.

Try (7*40) - (3*40) = 160, which quite closely matches your used memory.

Shared memory is the memory which is shared among the processes; the Apache libraries and stuff like that.

answered Sep 26 '11 at 06:53

Janne Pikkarainen

31,454
4
56
78

Ye.. but 160 Mb used only by apache. But we see that 163 Mb is used by all processes in system. 160 mb apache and 3 mb others... – zim32 Sep 26 '11 at 13:06
Where did you get the numbers `(7*40) - (3*40)` from? – Click Upvote Sep 14 '13 at 11:23

Linux top command. Memory usage

2 Answers2