-2

Top reports that the system is using 93% (7.2GB) of the total 7.6GB RAM. However, the sum of all running processes as calculated below is 2.2GB! What is wrong? Could be something to do with the fact that it is a VMWare virtual server? Edit: I added the output of free.

$ top

top - 12:00:28 up 116 days, 22:10,  2 users,  load average: 0.70, 0.24, 0.14
Tasks: 134 total,   1 running, 133 sleeping,   0 stopped,   0 zombie 
Cpu(s): 35.2%us,  9.1%sy,  0.0%ni, 16.6%id,  8.6%wa,  0.5%hi, 30.0%si,  0.0%st
Mem:   8061584k total,  7513516k used,   548068k free,     9352k buffers
Swap:  2047996k total,  1559912k used,   488084k free,    80004k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
10478 mysql     20   0 4181m 1.0g 3300 S 37.0 13.7  12212:52 mysqld
32309 app        0 -20  552m 108m 4520 S  0.0  1.4   0:02.60 ruby
32199 root      20   0 1026m  99m 5680 S 12.2  1.3   0:07.35 ruby
32526 app       20   0  661m  99m 6112 S 52.0  1.3   0:12.77 ruby
32303 app        0 -20  549m  96m 4324 S  0.0  1.2   0:03.58 ruby
32300 app        0 -20  550m  93m 4304 S  0.0  1.2   0:03.15 ruby
22472 app       20   0 1811m  64m 2420 S 11.3  0.8 152:50.74 java
32306 app        0 -20  534m  33m 1920 S  0.0  0.4   0:00.03 ruby
24600 app        0 -20  534m  28m 1584 S  0.0  0.4   0:12.29 ruby
17998 root      20   0 1125m  25m 1284 S  0.0  0.3  30:33.06 ruby
18089 root      20   0  953m  17m 1868 S  0.0  0.2   0:40.92 node

$ free -m

             total       used       free     shared    buffers     cached
Mem:          7872       7372        499          0          1         51
-/+ buffers/cache:       7319        553
Swap:         1999       1436        563

$ sudo ps -A -o rss | awk '{ sum += $1} END { print sum }'

2300228 (2.2GB)

$ sudo ps -A -o size | awk '{ sum += $1} END { print sum }'

12359284 (11.8GB)
sina
  • 189
  • 1
  • 2
  • 9
  • I don't know about how this machine is built at the moment, except that it is a CentOS. I used `cat /proc/scsi/scsi` to find out that it is a VMWare. Let me know if there is anything else I could do to find more details. – sina Oct 29 '15 at 09:23
  • This may be a duplicate. But not of the question it is currently marked as a duplicate of. I am guessing used memory is occupied by slabs or a balloon. Neither is mentioned in http://serverfault.com/q/67759/214507 – kasperd Oct 29 '15 at 10:53
  • I don't know why I am getting downvoted. The linked duplicate doesn't answer my question. This is not normal, I am maintaining other Linux servers and none of them have this issue. In this case MySQL will probably crash very soon as Kernel (or whatever is using the RAM) is not freeing RAM. – sina Oct 29 '15 at 14:24
  • @sina dont worry this is a common shun on this type of question, because for most cases it can be answered with `free -m` and undestanding how linux uses memory. but not always, e.g. I have still more memory really "used" but with no processes or system to underlay it........... – jave.web Jan 05 '21 at 15:51

1 Answers1

2

Linux uses unused RAM for caching purposes that can be freed instantly if needed. Summing up process memory to get total RAM usage is a useless approach, use free -m to get the picture.

The output of free indicates that most of the RAM is indeed used, and not as cache. However, summing up the output of top to come up with a number for the used RAM isn't possible. First, by default top sorts by CPU consumption, not be RAM consumption, and second the list is usually not complete, as it is limited to the screen length. So, it's entirely possible you have a mostly dormant process actually using 5 GB of RAM.

Top lists different memory usage types. RES is the amount of memory used exclusive by a process (this number is used by the memory order), and SHR is the shared memory used by multiple processes, which is likely the culprit as the app process uses a lot of shared memory.

Sven
  • 97,248
  • 13
  • 177
  • 225
  • Thanks, I added the output of `free -m` to the question. Which one shows the real RAM usage in this case? It should probably be between 1-2 GB. – sina Oct 29 '15 at 09:20
  • Did you check the `buffers/cache` numbers I added to the question? According to the duplicate question you linked my actual used memory is 7319 (93%) which is incorrect because the actual sum is about 20% of total RAM. – sina Oct 29 '15 at 10:26
  • `...actual sum is about 20%`. That's *just* the running processes. the Kernel uses RAM for other things too. – GregL Oct 29 '15 at 11:26
  • @GregL Thanks. Why is kernel using RAM without showing it as any process? Why is it not the case in my other servers. I am running out of RAM, and this is bad for my apps, MySQL will probably crash soon, as it has happened. – sina Oct 29 '15 at 14:20
  • Because the rest of your RAM isn't being used by a particular process, but rather by the filesystem cache, as stated by Sven and reported by `free`. Please carefully read both this answer, as well as the linked one and you'll find it's right. – GregL Oct 29 '15 at 14:28
  • @GregL As I noted in the above comment, according to the linked answer `free` is reporting that running proceses (`used` column `buffers/cache` row) are using 7319 (93%). Shouldn't this number be around 2GB? – sina Oct 29 '15 at 14:37
  • @abc see my edit. – Sven Oct 29 '15 at 14:55
  • @Sven Thank you for the update! I added two more reports to the question (total sum of `rss` and `size` for all processes as reported by `ps`). The `size` sum is very large. I need to look up about how to interpret the `SHR` values as well, but they don't seem very large. – sina Oct 29 '15 at 20:04