my webserver with 16GB ram shows all RAM as used, but is it really, see the 'top'

Question

I have some questions about my web server. Its a LAMP web server running centos 5.5 and php5, mysql5. The server gets hundreds (maybe thousand) of concurrent users during peak hours.

I'm trying to optimize a little and understand "top". From what I can see:

all 16GB of my ram have been used up?
does that mean that my server needs more memory?
My swap is only 2GB, should it be increased?
usually during peak hours my server load average first number is about 2.5-3. What could I do to optimize the server so that the load average even during peak doesn't go above 1? In the past I was told a good working server should stay under 1 load, is this still true? Although even during load of 2.5-3, server pages and applications seem to load with pretty good speed.
what should the memory size in php.ini be set to?

    top - 14:30:18 up 2 days, 12:41,  5 users,  load average: 1.25, 1.74, 2.92                                                                                                                                                                                      
    Tasks: 305 total,   2 running, 302 sleeping,   0 stopped,   1 zombie                                                                                                                                                                                            
    Cpu(s):  6.3%us,  0.9%sy,  0.0%ni, 92.5%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st                                                                                                                                                                                  
    Mem:  16427200k total, 16111472k used,   315728k free,  3120316k buffers                                                                                                                                                                                        
    Swap:  2104496k total,      268k used,  2104228k free,  6216756k cached                                                                                                                                                                                         

      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                            29080 apache    15   0  358m  36m 5192 S 20.2  0.2   2:08.40 httpd                                                                                                                                                                                              
    29093 apache    18   0  357m  36m 5192 S 18.2  0.2   2:02.52 httpd                                                                                                                                                                                              29079 apache    15   0  370m  49m 5832 S 10.0  0.3   2:32.14 httpd                                                                                                                                                                                              
     1812 apache    15   0  370m  49m 5196 S  7.3  0.3   2:25.30 httpd                                                                                                                                                                                               5204 apache    15   0  358m  36m 5168 S  5.3  0.2   0:59.28 httpd                                                                                                                                                                                              
    29075 apache    15   0  370m  48m 5184 S  3.3  0.3   2:15.93 httpd                                                                                                                                                                                               9712 apache    15   0  360m  38m 5180 S  3.0  0.2   0:54.81 httpd                                                                                                                                                                                              
    29072 apache    16   0  358m  36m 5192 S  2.7  0.2   2:24.43 httpd                                                                                                                                                                                               6310 apache    17   0  388m  67m 5180 S  2.3  0.4   0:58.85 httpd                                                                                                                                                                                              
     8674 apache    15   0  343m  21m 4980 S  2.0  0.1   0:07.91 httpd                                                                                                                                                                                              29085 apache    15   0  371m  49m 5224 S  2.0  0.3   2:16.86 httpd                                                                                                                                                                                              
    29083 apache    15   0  370m  48m 5196 S  1.7  0.3   2:10.64 httpd                                                                                                                                                                                               5575 apache    15   0  357m  36m 5228 S  1.3  0.2   0:53.78 httpd                                                                                                                                                                                              
    29066 apache    15   0  379m  59m 5860 R  1.3  0.4   2:11.93 httpd                                                                                                                                                                                              29078 apache    15   0  370m  48m 5188 S  1.3  0.3   2:14.52 httpd                                                                                                                                                                                              
    29084 apache    15   0  370m  48m 5208 S  1.0  0.3   2:02.49 httpd                                                                                                                                                                                              29089 apache    15   0  370m  48m 5188 S  1.0  0.3   2:27.61 httpd                                                                                                                                                                                              
    29082 apache    15   0  390m  68m 5188 S  0.7  0.4   2:32.48 httpd                                                                                                                                                                                              29984 apache    15   0  358m  36m 5228 S  0.7  0.2   2:08.32 httpd                                                                                                                                                                                              
     3571 root      16   0 13400 1792  848 S  0.3  0.0   2:37.89 top                                                                                                                                                                                                 4419 mysql     15   0  668m 175m 7204 S  0.3  1.1   3:32.25 mysqld                                                                                                                                                                                             
    28181 root      15   0 90460 3624 2680 S  0.3  0.0   0:17.60 sshd                                                                                                                                                                                               29091 apache    15   0  390m  69m 5196 S  0.3  0.4   2:29.99 httpd                                                                                                                                                                                              
    32476 root      15   0 12900 1320  848 R  0.3  0.0   0:06.46 top                                                                                                                                                                                                    1 root      15   0 10372  680  572 S  0.0  0.0   0:02.01 init                                                                                                                                                                                               
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.51 migration/0                                                                                                                                                                                            3 root      34  19     0    0    0 S  0.0  0.0   0:00.07 ksoftirqd/0                                                                                                                                                                                        
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0                                                                                                                                                                                             5 root      RT  -5     0    0    0 S  0.0  0.0   0:00.12 migration/1                                                                                                                                                                                        
    6 root      34  19     0    0    0 S  0.0  0.0   0:00.03 ksoftirqd/1                                                                                                                                                                                            7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/1                                                                                                                                                                                         
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:00.06 migration/2                                                                                                                                                                                            9 root      34  19     0    0    0 S  0.0  0.0   0:00.03 ksoftirqd/2                                                                                                                                                                                        
    10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/2                                                                                                                                                                                            11 root      RT  -5     0    0    0 S  0.0  0.0   0:00.06 migration/3                                                                                                                                                                                        
    12 root      34  19     0    0    0 S  0.0  0.0   0:00.04 ksoftirqd/3                                                                                                                                                                                           13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/3                                                                                                                                                                                         
    14 root      RT  -5     0    0    0 S  0.0  0.0   0:01.45 migration/4                                                                                                                                                                                           15 root      34  19     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd/4                                                                                                                                                                                        
    16 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/4                                                                                                                                                                                            17 root      RT  -5     0    0    0 S  0.0  0.0   0:00.22 migration/5                                                                                                                                                                                        
       18 root      34  19     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd/5                                                                                                                                                                                           19 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/5                                                                                                                                                                                         
       20 root      RT  -5     0    0    0 S  0.0  0.0   0:00.15 migration/6                                                                                                                                                                                           21 root      34  19     0    0    0 S  0.0  0.0   0:00.02 ksoftirqd/6                                                                                                                                                                                        
       22 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/6                                                                                                                                                                                            23 root      RT  -5     0    0    0 S  0.0  0.0   0:00.15 migration/7                                                                                                                                                                                        
       24 root      34  19     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd/7                                                                                                                                                                                           25 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/7                                            
       26 root      RT  -5     0    0    0 S  0.0  0.0   0:00.19 migration/8                                              27 root      34  19     0    0    0 S  0.0  0.0   0:00.04 ksoftirqd/8                                           
       28 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/8                                               29 root      RT  -5     0    0    0 S  0.0  0.0   0:00.34 migration/9                                           
       30 root      34  19     0    0    0 S  0.0  0.0   0:00.03 ksoftirqd/9                                              31 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/9                                            
       32 root      RT  -5     0    0    0 S  0.0  0.0   0:00.16 migration/10                                             33 root      34  19     0    0    0 S  0.0  0.0   0:00.04 ksoftirqd/10                                          
       34 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/10                                              35 root      RT  -5     0    0    0 S  0.0  0.0   0:00.12 migration/11                                          
       36 root      34  19     0    0    0 S  0.0  0.0   0:00.05 ksoftirqd/11                                             37 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/11                                           
       38 root      RT  -5     0    0    0 S  0.0  0.0   0:00.35 migration/12

And here is another top paste, but its not peak time for traffic yet. Isn't it a problem that httpd is using 100% CPU?

top - 16:35:57 up 2 days, 14:47,  4 users,  load average:  4.34, 3.60, 3.09
Tasks: 321 total,   3 running, 317 sleeping,   0 stopped,   1 zombie
Cpu(s): 15.0%us,  1.1%sy,  0.0%ni, 65.7%id, 18.1%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  16427200k total, 16235036k used,   192164k free,  3129048k buffers
Swap:  2104496k total,      268k used,  2104228k free,  6184496k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                            
  540 apache    25   0  385m  63m 4720 R 100.1  0.4   0:22.11 httpd                                                                                                                                                                                             
16194 apache    18   0  358m  36m 5188 S 29.6  0.2   2:42.62 httpd                                                                                                                                                                                              
 1759 apache    15   0  391m  69m 5188 S 20.6  0.4   3:32.70 httpd                                                                                                                                                                                              
 1761 apache    17   0  388m  67m 5844 S 20.0  0.4   3:32.70 httpd                                                                                                                                                                                              
 5420 apache    16   0  357m  36m 5816 S 18.0  0.2   3:29.58 httpd                                                                                                                                                                                              
 1758 apache    16   0  358m  37m 5188 S 15.3  0.2   3:11.11 httpd                                                                                                                                                                                              
26922 apache    16   0  366m  45m 4148 R 12.6  0.3   0:29.59 httpd                                                                                                                                                                                              
13655 apache    15   0  371m  49m 5840 S  6.3  0.3   5:22.93 httpd                                                                                                                                                                                              
26894 apache    15   0  358m  36m 5124 S  6.0  0.2   1:00.43 httpd                                                                                                                                                                                              
26925 apache    15   0  358m  36m 5156 S  2.0  0.2   0:29.61 httpd                                                                                                                                                                                              
13654 apache    15   0  358m  36m 5200 S  1.3  0.2   5:06.32 httpd                                                                                                                                                                                              
23337 apache    15   0  359m  37m 5184 S  1.3  0.2   3:02.68 httpd                                                                                                                                                                                              
24108 apache    16   0  357m  35m 5180 S  1.3  0.2   0:48.09 httpd                                                                                                                                                                                              
28473 apache    15   0  357m  35m 5128 S  1.3  0.2   0:25.66 httpd                                                                                                                                                                                              
  584 apache    15   0  357m  35m 5068 S  1.0  0.2   0:12.12 httpd                                                                                                                                                                                              
 2687 apache    15   0  370m  48m 5168 S  1.0  0.3   3:26.85 httpd                                                                                                                                                                                              
26893 apache    15   0  357m  35m 5144 S  1.0  0.2   0:33.74 httpd                                                                                                                                                                                              
26923 apache    15   0  357m  35m 5072 S  1.0  0.2   0:33.96 httpd                                                                                                                                                                                              
26924 apache    15   0  360m  38m 5156 S  1.0  0.2   0:58.06 httpd                                                                                                                                                                                              
  585 apache    15   0  357m  35m 5096 S  0.7  0.2   0:11.08 httpd                                                                                                                                                                                              
17329 apache    15   0  369m  48m 5176 S  0.7  0.3   1:15.52 httpd                                                                                                                                                                                              
 1760 apache    15   0  389m  68m 5856 D  0.3  0.4   3:36.45 httpd                                                                                                                                                                                              
 4419 mysql     15   0  668m 175m 7204 S  0.3  1.1   4:04.57 mysqld                                                                                                                                                                                             
    1 root      15   0 10372  680  572 S  0.0  0.0   0:02.04 init

score 6 · Answer 1 · answered Jun 28 '11 at 15:00

Mem:  16427200k total,  <--- You have 15.6GB total RAM.
      16111472k used,   <--- Of which, Linux is using  15.4GB of it for various things.
      315728k free,     <--- The remainder .3GB, is officially unused by anything.
      3120316k buffers  <--- 3GB is used in Buffers of various kinds.         
      6216756k cached   <--- 5.9GB is used in the block and file caches.

Should you panic? No. Memory in Buffers and Cache can be consumed by applications on demand. In your case, you have just shy of 9GB in this state, you have plenty of RAM.

Cache is used to make disk-accesses faster by bringing disk data in to memory. If you were running a large file-server, for instance, you'd want as much memory in the Cache state as you could get. "Free" memory in Linux is fully unused memory, which is going to waste.

jamespo · Answer 2 · 2011-06-28T16:14:55.383

3

http://www.linuxatemyram.com/ ?

Edited summary: Spare memory is being used as cache

edited Jun 28 '11 at 16:14

answered Jun 28 '11 at 14:43

jamespo

1,698
12
12

1

-1 Link-and-run answers are strongly discouraged here. Please include at least a summary of the linked-to article. – sysadmin1138 Jun 28 '11 at 15:00
I know its bad to link & run, but I like this answer ;) – Jakub Jun 28 '11 at 15:39

score 2 · Answer 3 · answered Jun 28 '11 at 15:00

Your RAM isn't really all used up, 6gb of it is being used as cache. This is normal, linux will fill available ram with cache. It will use this RAM for applications if needed.

You're only using 268k of swap, you don't need to allocate any more.

A load of 2/3 seems fine for a busy webserver. In general if the load is less than the number of CPU cores you have it doesn't indicate a problem.

The memory limit in the php.ini depends on the requirements of whatever applications your webserver runs.

score 0 · Answer 4 · answered Aug 29 '13 at 23:36

Try free -m for a quick look at memory/swap utilized, free, and cached.

 ~# free -m
             total       used       free     shared    buffers     cached
 Mem:         70000      69650        349          0         38      21361
 -/+ buffers/cache:      48250      21750
 Swap:            0          0          0

my webserver with 16GB ram shows all RAM as used, but is it really, see the 'top'

4 Answers4