Re: [Qemu-discuss] qemu with librbd vs krbd - huge performance difference

[Fam Zheng]

On Tue, 09/04 08:48, Nikola Ciprich wrote:
> Hi,
> 
> I've seen this discussed few times without any real conclusion,
> so I'd like to try asking again..
> 
> we're  seeing very bad performance on ceph volumes mapped to qemu using 
> librbd,
> while block access to krbd is multiple times better (when comparing fast 
> storage iops).
> 
> rough numbers I'm getting:
> 
> qemu, block access to KRBD mapped volume:
> read: > ~100K IOPS
> write: > ~100K IOPS
> 
> qemu: librbd access to volume:
> read: ~25K IOPS
> write: ~20K IOPS
> 
> 
> when I run FIO on host, using librbd, I'm getting ~140K IOPS on reads, ~50K 
> IOPS
> on write.
> 
> trying different cache modes does not have any important impact. I'm using 
> virtio-scsi,
> 
> no special settings, host is 4.14.60 x86_64, qemu-kvm-2.12.0, guest has  16 
> cores assigned.
> 
> fio cmdline:
> 
> on host:
> fio --readwrite=randread/randwrite --randrepeat=1 --ioengine=rbd --direct=1 
> --numjobs=16 --gtod_reduce=1 --name=test --pool=${pool} --rbdname=${vol} 
> --invalidate=0 --bs=4k --iodepth=64 --time_based --runtime=60 
> --group_reporting
> 
> 
> in guest:
> fio --readwrite=randread/randwrite --filename=/dev/sdx --randrepeat=1 
> --ioengine=libaio --direct=1 --numjobs=16 --gtod_reduce=1 --name=test 
> --invalidate=0 --bs=4k --iodepth=64 --time_based --runtime=60 
> --group_reporting
> 
> trying numjobs=1 leads to similar results (qemu krbd much faster then librbd)
> 
> no special rbd features enabled on volume, apart from layering.
> 
> any ideas on why the difference might be so huge?

Any different pattern in host mpstat comparing the krbd and librbd runs? IIUC
even with numjobs=1, iodepth=64 may still allow krbd to benefit from blk-mq's
scalability.

Fam

> 
> thanks a lot in advance
> 
> BR
> 
> nik
> 
> 
> -- 
> -------------------------------------
> Ing. Nikola CIPRICH
> LinuxBox.cz, s.r.o.
> 28.rijna 168, 709 00 Ostrava
> 
> tel.:   +420 591 166 214
> fax:    +420 596 621 273
> mobil:  +420 777 093 799
> www.linuxbox.cz
> 
> mobil servis: +420 737 238 656
> email servis: address@hidden
> -------------------------------------
>