Re: [Qemu-discuss] Increasing number of threads when using ceph

[Tyanko Aleksiev]

Hi Jan,

On 06/01/2017 06:44 PM, Jan Schermer wrote:
> My experience is the same, threads only grow.
> The only way to decrease the number of threads is to detach the block device 
> completely.

And thank you for the information.

> Note that if you need to support VMs with many virtual block devices, then each 
> block device will need hundreds of threads and possibly also hundreds of 
> connections (depending on how many OSDs you have).
> Basically, the qemu process will open new threads and connection(s?) for every 
> OSD serving the block device (which is likely all of them).
> In practice I think the number was something like:
>
> total = $no_of_osds * $number_of_devices * 3
>
> Thus, in a standard configuration you will then soon run out of PIDs, file 
> descriptors, connections, memory...
> <rant>... then you'll start loosing sleep, patience and finally data, which is when 
> you'll finally start moving to a sane storage backend</rant>

For us this is rather an isolated case than a common issue since it only 
happens on heavily oversubscribed compute nodes (3 out of 300).

Cheers,
Tyanko

> Jan
>
>
> > On 1 Jun 2017, at 15:44, Tyanko Aleksiev <address@hidden> wrote:
> >
> > Dear QEMU users,
> >
> > On our OpenStack installation we use QEMU/KVM (v1:2.5 on Ubuntu 14.04, kernel 
> > 3.13) controlled by libvirt (v1.3.1). The storage back-end for the VMs is 
> > provided via ceph (v0.94.10).
> >
> > Whenever a new instance is spawned the number of tasks (threads) initially 
> > started by libvirt-qemu is around 200. Big fraction of those are used to access 
> > the OSDs of the ceph cluster. Additionally, when some I/O operations are 
> > performed within the instance the number of threads will increase which is also 
> > an expected behavior.
> >
> > However, we noticed is that the number of threads will never decrease over 
> > time. In particular, I generated some I/O by creating files of different size. 
> > After the write operation was completed the number of threads did not decrease. 
> > Then, I deleted the files and reclaimed (discarded/trimed) the space which 
> > again did not result in any thread being released.
> >
> > I'm wondering if this is something we should expect? If not, eventually, how 
> > this issue could be addressed?
> >
> > Thank you very much for your time and help!
> >
> > Cheers,
> > Tyanko
> >
> > --
> > Tyanko Aleksiev
> > System Administrator
> >
> > Service and Support for Science IT,
> > University of Zurich
> > Winterthurerstrasse 190
> > CH-8057 Zurich Switzerland
> > Tel: +41 44 635 42 22

--
Tyanko Aleksiev
System Administrator

Service and Support for Science IT,
University of Zurich
Winterthurerstrasse 190
CH-8057 Zurich Switzerland
Tel: +41 44 635 42 22