Re: [Qemu-discuss] Increasing number of threads when using ceph

[Jan Schermer]

My experience is the same, threads only grow.
The only way to decrease the number of threads is to detach the block device 
completely.
Note that if you need to support VMs with many virtual block devices, then each 
block device will need hundreds of threads and possibly also hundreds of 
connections (depending on how many OSDs you have).
Basically, the qemu process will open new threads and connection(s?) for every 
OSD serving the block device (which is likely all of them).
In practice I think the number was something like:

total = $no_of_osds * $number_of_devices * 3

Thus, in a standard configuration you will then soon run out of PIDs, file 
descriptors, connections, memory...
<rant>... then you'll start loosing sleep, patience and finally data, which is 
when you'll finally start moving to a sane storage backend</rant>

Jan


> On 1 Jun 2017, at 15:44, Tyanko Aleksiev <address@hidden> wrote:
> 
> Dear QEMU users,
> 
> On our OpenStack installation we use QEMU/KVM (v1:2.5 on Ubuntu 14.04, kernel 
> 3.13) controlled by libvirt (v1.3.1). The storage back-end for the VMs is 
> provided via ceph (v0.94.10).
> 
> Whenever a new instance is spawned the number of tasks (threads) initially 
> started by libvirt-qemu is around 200. Big fraction of those are used to 
> access the OSDs of the ceph cluster. Additionally, when some I/O operations 
> are performed within the instance the number of threads will increase which 
> is also an expected behavior.
> 
> However, we noticed is that the number of threads will never decrease over 
> time. In particular, I generated some I/O by creating files of different 
> size. After the write operation was completed the number of threads did not 
> decrease. Then, I deleted the files and reclaimed (discarded/trimed) the 
> space which again did not result in any thread being released.
> 
> I'm wondering if this is something we should expect? If not, eventually, how 
> this issue could be addressed?
> 
> Thank you very much for your time and help!
> 
> Cheers,
> Tyanko
> 
> -- 
> Tyanko Aleksiev
> System Administrator
> 
> Service and Support for Science IT,
> University of Zurich
> Winterthurerstrasse 190
> CH-8057 Zurich Switzerland
> Tel: +41 44 635 42 22
>