2

I have an Ubuntu server with 320GB of memory. I installed xen 4.4.1 on this machine, and run 2 Debian VMs. One with +-100GB of memory and one with +-200GB. Everything worked fine, until at one point, the 200GB machine reports having only 128GB. The server had an uptime of 144 days and somewhere within the last month, more than 70GB of memory went missing.

on the dom0:

$ sudo xl info
...
total_memory           : 327634
free_memory            : 16547
...

$ sudo xl list
Name                               ID   Mem  VCPUs      State   Time(s)
Domain-0                            0    510    32     r-----      54.4
mycroft                             1 102400    16     -b----      33.3
adler                               2 204000    16     -b----      34.5

$ uname -a
Linux moriarty 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

on the VM having 204000MB according to xl list:

$ free -m
             total       used       free     shared    buffers     cached
Mem:        128404       6220     122184          0         10         56
-/+ buffers/cache:       6152     122251
Swap:            0          0          0

$ uname -a
Linux adler 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux

$ cat /proc/meminfo
MemTotal:       131486352 kB
MemFree:        125117048 kB
Buffers:           11216 kB
Cached:            58016 kB
SwapCached:            0 kB
Active:          6057868 kB
Inactive:          47632 kB
Active(anon):    6036284 kB
Inactive(anon):      324 kB
Active(file):      21584 kB
Inactive(file):    47308 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                12 kB
Writeback:             0 kB
AnonPages:       6036296 kB
Mapped:            14740 kB
Shmem:               344 kB
Slab:              20024 kB
SReclaimable:       6504 kB
SUnreclaim:        13520 kB
KernelStack:        2728 kB
PageTables:        14824 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    65743176 kB
Committed_AS:   91568356 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      214612 kB
VmallocChunk:   34359523687 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:    208896000 kB
DirectMap2M:           0 kB

I already rebooted both servers without any result: the dom0 keeps reporting 204gB, the machine itself reports 128gB. What's the cause of the difference and how can I fix it?

EDIT

The dmesg output gives me this

[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000002000000000 (usable)
[    0.000000]  Xen: 0000002000000000 - 00000031ce000000 (unusable)

The range of the last line seems to correspond with the missing memory.

bmesuere
  • 123
  • 3

2 Answers2

2

I had the same problem with debian wheezy 7.8 guests. Installing the wheezy backports kernel 3.16.0-0.bpo.4-amd64 solved it for me. This was on the guests, I didn't touch the host.

Add the following line to /etc/apt/sources.list:

deb http://ftp.uk.debian.org/debian/ wheezy-backports main

Then run

apt-get update
apt-get -t wheezy-backports install linux-image-amd64
reboot
Andrew Schulman
  • 8,561
  • 21
  • 31
  • 47
Matt
  • 36
  • 1
0

Did you have memory ballooning in place? If so, the "missing" memory should be the memory reclaimed by the balloon driver.

Can you post the output cat /proc/meminfo on the machine with the "missing" memory?

EDIT

From your /proc/meminfo output, it seems that balloon is at work indeed.

Have a look at "DirectMap4k" value: it reports that about 200 GB of RAM are managed by the MMU at 4k granularity. In other word, the virtualized hardware see the full 200 GB RAM.

However, the "MemTotal" value clearly shows that total available memory is "only" 135 GB.

This means that something at kernel/driver level "stolen" some memory for other use. Such a big amount of free memory is the perfect target for ballooning. You can find more information here.

shodanshok
  • 44,038
  • 6
  • 98
  • 162