Xen 中关于 32G 内存的两个问题

一台 48G 物理内存的服务器,上面做了 xen 虚拟化,重启之后不论是 free 还是 top 发现都只有 32G 的内存。

# free -m
total       used       free     shared    buffers     cached
Mem:         32768        961      31806          0         13        122
-/+ buffers/cache:        824      31943
Swap:         8095          0       8095

但是使用 xm info 查看得出的确实理想的结果:

# xm info

total_memory           : 49139
free_memory            : 15425

# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0    32768     4 r—–     30.3

g 了几个帖子,下面直接给出原文了。

"Yes, this is normal, or rather it is a limitation of the xen kernel. Note
that it probably is not a real issue in practice, as any VM's that you
create will come from the "missing" 32GB, only overflowing into the "real"
32GB when the first 32GB is used. I have a similar system with 48GB
physical memory, and with 26 guests, each having 1 GB, the amount of
memory shown in Dom0 is around 22GB.

Dom0 shouldn't see all the memory !!
You should limit dom0 memory to, say, 1 GB.

Add dom0_mem=1G option for xen.gz in /boot/grub/grub.conf and reboot.

then your dom0 should have around 1 GB in /proc/meminfo.

You can check the Xen hypervisor memory info with "xm info". It'll list
the hypervisor available memory.

And "xm list" will list how much memory each domain/guest is using.

If you have full amount (32G or 64G) of memory visible to dom0 linux
kernel, then linux will set network, memory etc defauls based on that
(huge) boot time amount of memory.. and then when you create guests,
that dom0 memory will be "ballooned" down, and later when you have
much smaller amount of memory in dom0, it'll have problems since many
internal kernel parameters were based on having 32G or 64G of memory..
and weird things start to happen.

So it's best to limit and dedicate for example 1 GB of memory to dom0
and prevent ballooning that way.

Xen always dedicated physical memory to each guest.. dom0 is also a guest. "

解决该问题最佳的方式是给 Dom0 固定内存,在启动的时候为 kernel 加上如下的参数:

title CentOS (2.6.18-238.el5xen)
   root (hd0,0)
   kernel /xen.gz-2.6.18-238.el5 dom0_mem=4G
module /vmlinuz-2.6.18-238.el5xen ro root=LABEL=/
module /initrd-2.6.18-238.el5xen.img

同时需要在 xend-config.sxp 文件中固定 (dom0-min-mem 4096) 参数的值(这里设置的是 4096),该值表示当 dom0 的内存“缩小”时的最低限度,如果为 0 的话,表示 dom0 永远不会缩小,也就是 domU 不会占用到 dom0 的内存。

# free -m
total       used       free     shared    buffers     cached
Mem:          4096        656       3439          0        181        199
-/+ buffers/cache:        276       3819
Swap:         8095          0       8095

# cat /proc/meminfo

MemTotal:      4194304 kB
MemFree:       3521696 kB

# xm info
total_memory           : 49139
free_memory            : 44097

# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     4096     4 r—–    199

如果你想将 Dom0 固定至 40G,跟上面一样的修改方式:

# free -m
total       used       free     shared    buffers     cached
Mem:         40960       1098      39861          0         13        122
-/+ buffers/cache:        962      39997
Swap:         8095          0       8095

这时候我们将其中的一个 domU 的 .cfg 中的 memory 改为 15240。启动 domU,使用 xm 观察内存的使用情况:

# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0    32952     4 r—–     22.4
ubuntu_openvpn                             1    15240     1 -b—-      3.1

可以发现 Dom0 的内存被挤压了。接着将该 DomU 关闭,xm 发现:

# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0    32952     4 r—–     31.1

内存就由最初的 40960 挤压到了 32952 了,可以想象,这样的动态内存变化对 Dom0 来说并不好。

上面是 Dom0 出现的 32G 内存的问题,下面的这个是在 domU 上出现的 32G 内存的问题,到目前为止我们还没遇到这种情况(准确的讲是还没有给 domU 分配这么大的空间过),暂且记录于此。

问题大致的意思是说,给 domU 分配了 64G 的内存,但是在 domU 中只能看到 32G 的内存。

该问题发生在 debian amd64 squeeze 2.6.32-5-xen-amd64 的内核中,该内核在设计之初就只给 domU 分配了最大为 32G 的内存,要修复此问题,只能使用新内核。在 domU 的内核中:

| CONFIG_XEN_MAX_DOMAIN_MEMORY:
|
| The pseudo-physical to machine address array is sized
| according to the maximum possible memory size of a Xen
| domain.  This array uses 1 page per gigabyte, so there's no
| need to be too stingy here.
|
| Symbol: XEN_MAX_DOMAIN_MEMORY [=32]
| Prompt: Maximum allowed size of a domain in gigabytes
|   Defined at arch/x86/xen/Kconfig:16
|   Depends on: PARAVIRT_GUEST [=y] && XEN [=y]
|   Location:
|     -> Processor type and features
|     -> Paravirtualized guest support (PARAVIRT_GUEST [=y])|
|     -> Xen guest support (XEN [=y])

参考:

http://xen.1045712.n5.nabble.com/DomU-max-32GB-memory-td4498208.html
http://www.mentby.com/norman-gaywood/more-than-32g-on-memory-in-a-domu-guest.html
http://wiki.xensource.com/xenwiki/XenBestPractices