[Qemu-discuss] Significantly better IOP performance in Windows 2008r2 vs 2016

[Matthew Schumacher]

I'd appreciate a little help here.  I'm trying to figure out why I get 
so much better disk performance in 2008r2 vs 2016 and I'm just out of 
ideas.  Here is the issue:

In windows 2008r2 I get:

IOMeter, 4k aligned 50% read, 100% random, 8 outstanding I/O: 18000 IOPS 
or 74MB/sec
CrystalDiskMark 4KiB Q8T8 298.2 read 152.8 Write

Under the covers, it's a zfs zvol, and zpool iostat shows ~2.8K to 3.5k 
iops when iometer is running.

In Windows 2016 I get:

IOMeter, 4k aligned 50% read, 100% random, 8 outstanding I/O: 8500 IOPS 
or 35MB/sec
CrystalDiskMark 4KiB Q8T8 79.48 read 71.52 Write

Under the covers, it's the same exact zfs zvol config on the same zpool, 
and zpool iostat shows ~2.0K to 2.6k iops

I can't for the life of me figure out why the performance gap.  Here is 
what I've looked at:

1.  Partitioning on the virtual disks (zvol) to see if they are aligned 
the same.  Found:

win2008r2
/dev/zvol/datastore/testwindows1 *       2048  1026047  1024000 500M  7 
HPFS/NTFS/exFAT
/dev/zvol/datastore/testwindows2      1026048 41940991 40914944 19.5G  7 
HPFS/NTFS/exFAT

Windows 2016
/dev/zvol/datastore/testwindows2p1 *      2048   206847   204800 100M  7 
HPFS/NTFS/exFAT
/dev/zvol/datastore/testwindows2p2      206848 41940991 41734144 19.9G  
7 HPFS/NTFS/exFAT

Both seem to align if I multiply the start sector * 512, then divide by 
4096, I get even number either way.

2.  Virtual machine settings.  Virsh dumpxml reports identical config 
except name, mac, and zvol device.

3.  Zvol config is identical, same block size, compression, everything.  
Both zvols live in the same pool.

4.  Windows drivers are from the same virtio driver iso.  Windows 2016 
reports version 100.76.104.14900 and Windows 2008r2 reports 
61.76.104.14900 both 3/20/2018.

More information.

Storage is a 4k block size zvol on top of a zpool in raid-10 (striped 
mirrors) on HBA controller.
qemu command lines:

/usr/bin/qemu-system-x86_64 -name guest=testwindows,debug-threads=on -S 
-object 
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-8-testwindows/master-key.aes 
-machine pc-1.1,accel=kvm,usb=off,dump-guest-core=off -cpu 
core2duo,nx=on,ss=on,monitor=off,vmx=off,cx16=on,hypervisor=on,lahf_lm=on 
-m 4096 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -object 
iothread,id=iothread1 -uuid 5e5e3048-7ea1-4069-8efd-df214724bfc8 
-no-user-config -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-8-testwindows/monitor.sock,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc 
base=localtime,clock=vm,driftfix=slew -global 
kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on 
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 
file=/dev/zvol/datastore/testwindows,format=raw,if=none,id=drive-virtio-disk0,cache=none,aio=native 
-device 
virtio-blk-pci,iothread=iothread1,ioeventfd=on,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2,write-cache=on 
-drive if=none,id=drive-ide0-1-0,readonly=on -device 
ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 
-netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=25 -device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:34:20:f7,bus=pci.0,addr=0x3 
-device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:1 -device 
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -sandbox 
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny 
-msg timestamp=on

/usr/bin/qemu-system-x86_64 -name guest=testwindows2,debug-threads=on -S 
-object 
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-9-testwindows2/master-key.aes 
-machine pc-1.1,accel=kvm,usb=off,dump-guest-core=off -cpu 
core2duo,nx=on,ss=on,monitor=off,vmx=off,cx16=on,hypervisor=on,lahf_lm=on 
-m 4096 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -object 
iothread,id=iothread1 -uuid b854c09e-5190-4adf-bcbe-501baa02d9e7 
-no-user-config -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-9-testwindows2/monitor.sock,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc 
base=localtime,clock=vm,driftfix=slew -global 
kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on 
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 
file=/dev/zvol/datastore/testwindows2,format=raw,if=none,id=drive-virtio-disk0,cache=none,aio=native 
-device 
virtio-blk-pci,iothread=iothread1,ioeventfd=on,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2,write-cache=on 
-drive if=none,id=drive-ide0-1-0,readonly=on -device 
ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 
-netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=26 -device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:82:5b:c7,bus=pci.0,addr=0x3 
-device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:2 -device 
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -sandbox 
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny 
-msg timestamp=on