NVME: Multiple IRQs but only failed FLUSH completion visible in CQ

[Maciej Leks]

I'm observing unexpected behavior with NVMe command completions in
QEMU. Here's what happens:

I issue a single READ command (NVME_NVM_CMD_READ) with:

CID: 8
NSID: 1
LBA: 0x802
NLB: 1

The trace log shows two IRQ notifications::
pci_nvme_irq_msix raising MSI-X IRQ vector 1
apic_deliver_irq dest 0 dest_mode 0 delivery_mode 0 vector 33 trigger_mode 0
[...]
pci_nvme_irq_msix raising MSI-X IRQ vector 1
apic_deliver_irq dest 0 dest_mode 0 delivery_mode 0 vector 33 trigger_mode 0

However, when checking the completion queue, I only see a failed FLUSH command:

CID: 0
Status: 0x400b (sct=1, sc=11)
This FLUSH command was not issued by my code.

The trace shows this unexpected FLUSH command:
pci_nvme_io_cmd cid 0 nsid 0x0 sqid 1 opc 0x0 opname 'NVME_NVM_CMD_FLUSH'
pci_nvme_enqueue_req_completion cid 0 cqid 1 dw0 0x0 dw1 0x0 status 0x400b

I suspect this might be related to automatic flushing because of the
last CQ element (SQ and CQ len is set to 8) and starting a new cycle,
but I'm not sure if this behavior is correct. Could someone explain
why I'm seeing this FLUSH command and why my READ completion isn't
visible in the CQ?

Full Trace log:
pci_nvme_mmio_write addr 0x1008 data 0x0 size 4
pci_nvme_mmio_doorbell_sq sqid 1 new_tail 0
pci_nvme_io_cmd cid 8 nsid 0x1 sqid 1 opc 0x2 opname 'NVME_NVM_CMD_READ'
pci_nvme_read cid 8 nsid 1 nlb 1 count 512 lba 0x802
pci_nvme_map_prp trans_len 512 len 512 prp1 0x144000 prp2 0x0 num_prps 1
pci_nvme_map_addr addr 0x144000 len 512
pci_nvme_io_cmd cid 0 nsid 0x0 sqid 1 opc 0x0 opname 'NVME_NVM_CMD_FLUSH'
pci_nvme_enqueue_req_completion cid 0 cqid 1 dw0 0x0 dw1 0x0 status 0x400b
pci_nvme_err_req_status cid 0 nsid 0 status 0x400b opc 0x0
pci_nvme_irq_msix raising MSI-X IRQ vector 1
apic_deliver_irq dest 0 dest_mode 0 delivery_mode 0 vector 33 trigger_mode 0
pic_register_write register 0x0b = 0x0
pci_nvme_rw_cb cid 8 blk 'drv0'
pci_nvme_rw_complete_cb cid 8 blk 'drv0'
pci_nvme_enqueue_req_completion cid 8 cqid 1 dw0 0x0 dw1 0x0 status 0x0
pci_nvme_irq_msix raising MSI-X IRQ vector 1
apic_deliver_irq dest 0 dest_mode 0 delivery_mode 0 vector 33 trigger_mode 0
apic_register_write register 0x0b = 0x0

Additional info:
qemu version: 9.1, 9.2.

Best Regards,
Maciek Leks