Fresh Debian Squeeze (6) Installation - Very High Ram Usage

Question

We've got a new server provisioned, and are experiencing high ram usage, without information on the usage in either htop / top nor in a python memory script found (http://www.pixelbeat.org/scripts/ps_mem.py). Here is the output from free -m:

             total       used       free     shared    buffers     cached
Mem:          7195       1347       5848          0          3         28
-/+ buffers/cache:       1315       5879
Swap:         3813          0       3813

And from top, sorted by memory usage:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                           
1586 root      20   0  200m  14m 1836 S    0  0.2   0:00.75 xend                                                                               
1585 root      20   0 95892 9.9m  832 S    0  0.1   0:00.00 xend                                                                               
1855 root      20   0 51728 2996 2368 S    0  0.0   0:00.00 sshd                                                                               
1822 ntp       20   0 38332 2176 1596 S    0  0.0   0:00.03 ntpd                                                                               
1858 psadmin   20   0 17728 1932 1436 S    0  0.0   0:00.00 bash                                                                               
1071 root      20   0 52416 1852 1032 S    0  0.0   0:00.02 rsyslogd                                                                           
1857 psadmin   20   0 51880 1396  756 S    0  0.0   0:00.00 sshd                                                                               
1866 psadmin   20   0 19064 1364 1000 R    0  0.0   0:00.04 top                                                                                
1564 root      20   0 49168 1140  592 S    0  0.0   0:00.00 sshd       (etc)

And the aforementioned python script which says what applications are using what:

Private  +   Shared  =  RAM used   Program 

180.0 KiB +  71.5 KiB = 251.5 KiB   xenconsoled
228.0 KiB +  32.0 KiB = 260.0 KiB   acpid
204.0 KiB +  58.0 KiB = 262.0 KiB   init
252.0 KiB +  75.5 KiB = 327.5 KiB   cron
332.0 KiB +  44.0 KiB = 376.0 KiB   dbus-daemon
380.0 KiB +  53.5 KiB = 433.5 KiB   xenstored
556.0 KiB + 138.0 KiB = 694.0 KiB   sudo
612.0 KiB + 255.0 KiB = 867.0 KiB   getty (6)
604.0 KiB + 582.0 KiB =   1.2 MiB   udevd (3)
  1.1 MiB + 103.5 KiB =   1.2 MiB   rsyslogd
956.0 KiB + 267.5 KiB =   1.2 MiB   ntpd
  1.2 MiB +  73.0 KiB =   1.3 MiB   bash
  1.5 MiB +   1.3 MiB =   2.8 MiB   sshd (3)
16.2 MiB +   3.8 MiB =  20.0 MiB   xend (2)
---------------------------------
                         31.0 MiB
=================================

Private  +   Shared  =  RAM used   Program

So, two questions really:

1) What is this RAM going to?

2) Why would I be seeing only 7195MB of RAM from an installed 8GB?

This is on a Dell Poweredge R210, with SAS drives in RAID0.

Any help with this would be very greatly appreciated!

Answer 1

Transpires that it was the same issue seen here: Debian Squeeze and available memory (1GB absent), the Xen kernel makes ~1GB 'absent' shown in dmesg.

Updated question with specifics: Xen Kernel on Debian Squeeze making 1GB memory 'absent'

Answer 2

I see xend and xenconsoled running, and you used the word "provisioning" when discussing the deployment, so I'm assuming that this is a virtualized system running under Xen?

That's your problem, though it isn't much of a problem, just the way that it works. dom0 is using the missing RAM. The additional RAM usage is likely the balloon driver attempting to conserve resources. There's some additional explanation available here, here, and some possible fixes here.

Answer 3

Question 1

Don't Panic!

It is being used for your file cache buffer. Linux will continually cache files that it reads from the hard drive to improve performance. It is nothing to worry about, as soon as the memory is required for something else it will dump the cache and make the memory available to running processes. It is actually telling you this when you are running free -m:

• total memory used is 1347 total
• memory used for your buffers/cache is 1315

Basically it is doing what it is supposed to do.

Question 2

It could well be due to the video card using memory, check your bios to see if there is a setting that allocates a specific amount of memory to the GPU.

Answer 4

tbh, idk.

http://www.mjmwired.net/kernel/Documentation/IO-mapping.txt

127 NOTE NOTE NOTE! The above is only one part of the whole equation. The above
128 only talks about "real memory", that is, CPU memory (RAM). 
129 
130 There is a completely different type of memory too, and that's the "shared
131 memory" on the PCI or ISA bus. That's generally not RAM (although in the case
132 of a video graphics card it can be normal DRAM that is just used for a frame
133 buffer), but can be things like a packet buffer in a network card etc. 
134 
135 This memory is called "PCI memory" or "shared memory" or "IO memory" or
136 whatever, and there is only one way to access it: the readb/writeb and
137 related functions. You should never take the address of such memory, because
138 there is really nothing you can do with such an address: it's not
139 conceptually in the same memory space as "real memory" at all, so you cannot
140 just dereference a pointer. (Sadly, on x86 it _is_ in the same memory space,
141 so on x86 it actually works to just deference a pointer, but it's not
142 portable). 

-- try disabling framebuffer completely. see if that does anything at all.

And if that was insufficiently mystifying you can go ahead and dive into:

http://www.tldp.org/LDP/tlk/mm/memory.html