[Qemu-discuss] qemu with librbd vs krbd - huge performance difference

[Nikola Ciprich]

Hi,

I've seen this discussed few times without any real conclusion,
so I'd like to try asking again..

we're  seeing very bad performance on ceph volumes mapped to qemu using librbd,
while block access to krbd is multiple times better (when comparing fast 
storage iops).

rough numbers I'm getting:

qemu, block access to KRBD mapped volume:
read: > ~100K IOPS
write: > ~100K IOPS

qemu: librbd access to volume:
read: ~25K IOPS
write: ~20K IOPS


when I run FIO on host, using librbd, I'm getting ~140K IOPS on reads, ~50K IOPS
on write.

trying different cache modes does not have any important impact. I'm using 
virtio-scsi,

no special settings, host is 4.14.60 x86_64, qemu-kvm-2.12.0, guest has  16 
cores assigned.

fio cmdline:

on host:
fio --readwrite=randread/randwrite --randrepeat=1 --ioengine=rbd --direct=1 
--numjobs=16 --gtod_reduce=1 --name=test --pool=${pool} --rbdname=${vol} 
--invalidate=0 --bs=4k --iodepth=64 --time_based --runtime=60 --group_reporting


in guest:
fio --readwrite=randread/randwrite --filename=/dev/sdx --randrepeat=1 
--ioengine=libaio --direct=1 --numjobs=16 --gtod_reduce=1 --name=test 
--invalidate=0 --bs=4k --iodepth=64 --time_based --runtime=60 --group_reporting

trying numjobs=1 leads to similar results (qemu krbd much faster then librbd)

no special rbd features enabled on volume, apart from layering.

any ideas on why the difference might be so huge?

thanks a lot in advance

BR

nik


-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: address@hidden
-------------------------------------