[Qemu-discuss] Too much memory overhead?

[Mike Leong]

We use Openstack.  Openstack is configured to reserve 32GB of memory for the hypervisor OS, day to day operation, etc.  However, even w/ 32G reserved, the system as very little free memory available.  Can someone help me figure out where is my memory used?  I'm suspecting qemu-system-x86_64 is using a lot more memory that allocated.

Here's my setup:

Openstack Release: Ice House

Server mem: 256G

Qemu version: 2.0.0+dfsg-2ubuntu1.1

Networking: Contrail 1.20

Block storage: Ceph 0.80.7

Hypervisor OS: Ubuntu 12.04

memory over-provisioning is disabled

kernel version: 3.11.0-26-generic

Info on instances:

- root volume is file backed (qcow2) on the hypervisor local storage

- each instance has a rbd volume mounted from Ceph

- no swap file/partition

Each hypervisor hosts about 45-50 instances.

address@hidden:/etc/libvirt/qemu# free -g

             total       used       free     shared    buffers     cached

Mem:           251        250          1          0          0          1

-/+ buffers/cache:        248          2  <------------ this number

Swap:           82         25         56

RSS sum of all the qemu processes:

address@hidden:/etc/libvirt/qemu# ps -eo rss,cmd|grep qemu|awk '{ sum+=$1} END {print sum}'

204191112

RSS sum of the non qemu processes:

address@hidden:/etc/libvirt/qemu# ps -eo rss,cmd|grep -v qemu|awk '{ sum+=$1} END {print sum}'

2017328

As you can see, the RSS total is only 196G.

slabtop usage:

 Active / Total Objects (% used)    : 473924562 / 480448557 (98.6%)

 Active / Total Slabs (% used)      : 19393475 / 19393475 (100.0%)

 Active / Total Caches (% used)     : 87 / 127 (68.5%)

 Active / Total Size (% used)       : 10482413.81K / 11121675.57K (94.3%)

 Minimum / Average / Maximum Object : 0.01K / 0.02K / 15.69K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME

420153856 420153856   7%    0.02K 18418442      256  73673768K kmalloc-16

55345344 49927985  12%    0.06K 864771       64   3459084K kmalloc-64

593551 238401  40%    0.55K  22516       28    360256K radix_tree_node

1121400 1117631  99%    0.19K  26700       42    213600K dentry

680784 320298  47%    0.10K  17456       39     69824K buffer_head

 10390   9998  96%    5.86K   2078        5     66496K task_struct

1103385 901181  81%    0.05K  12981       85     51924K shared_policy_node

 48992  48377  98%    1.00K   1531       32     48992K ext4_inode_cache

  4856   4832  99%    8.00K   1214        4     38848K kmalloc-8192

 58336  33664  57%    0.50K   1823       32     29168K kmalloc-512

 13552  11480  84%    2.00K    847       16     27104K kmalloc-2048

146256  81149  55%    0.18K   3324       44     26592K vm_area_struct

113424 109581  96%    0.16K   2667       48     21336K kvm_mmu_page_header

 18447  13104  71%    0.81K    473       39     15136K task_xstate

 26124  26032  99%    0.56K    933       28     14928K inode_cache

  3096   3011  97%    4.00K    387        8     12384K kmalloc-4096

106416 102320  96%    0.11K   2956       36     11824K sysfs_dir_cache

Using virsh dommemstat, I'm only using 194GB:

rss:

address@hidden:/etc/libvirt/qemu# for i in instance-0000*.xml; do inst=$(echo $i|sed s,\.xml,,); virsh dommemstat $inst; done|awk '/rss/ { sum+=$2} END {print sum}'

204193676

allocated:

address@hidden:/etc/libvirt/qemu# for i in instance-0000*.xml; do inst=$(echo $i|sed s,\.xml,,); virsh dommemstat $inst; done|awk '/actual/ { sum+=$2} END {print sum}'

229111808

Basically, the math doesn't add up.  The qemu processes are using less that what's allocated to them.  In the example above, node-2 has 250G, with 2G free.

qemu has been allocated 218G, w/ 194G used in RSS.  That means 24G is not used yet (218 - 194) and I only have 2G free.  You can guess what would happen if the instances decided to use that 24G...

thx