RE: ENQCMD

[Tian, Kevin]

> From: Stefan Hajnoczi <stefanha@redhat.com>
> Sent: Friday, October 30, 2020 3:51 PM
> 
> Hi,
> The "Scalable Work Submission in Device Virtualization" talk at KVM
> Forum 2020 was interesting and I have some beginner questions about
> ENQCMD:
> https://static.sched.com/hosted_files/kvmforum2020/22/Scalable_Work_Su
> bmission_In_Device_Virtualization.pdf
> 
> Security
> --------
> If the ENQCMD instruction is allowed for userspace applications, how can
> they be prevented from writing to the MMIO address directly (without the
> ENQCMD instruction) and faking the 64-byte enqueue register data format?
> For example, they could set the PRIV bit or an arbitrary PASID.

ENQCMD payload is transmitted through DMWr transactions (slide 10), which
cannot be triggered through other memory instructions. The device portal
only handles DMWr transactions.

PRIV cannot be set by ENQCMD. Only by ENQCMDS (in ring0).

PASID is copied from a MSR when ENQCMD is executed. the MSR is managed
by the kernel. Userspace cannot specify it directly.

> 
> Work Queue Design
> -----------------
> Have you looked at extending existing hardware interfaces like NVMe or
> VIRTIO to support enqueue registers?

The first devices supporting ENQCMD are DSA/IAX. The native support 
just landed in the kernel. In concept any device which wants to support
shared work queue can implement this capability.

> 
> Have you benchmarked NVMe or VIRTIO devices using ENQCMD instead of
> the traditional submission queuing mechanism?

No. We don't have such devices yet.

> 
> Is ENQCMD faster than traditional I/O request submission? If not, then I
> guess it's only intended for shared queues where the PASID translation
> is needed?

ENQCMD is non-posted transaction. So it's a little bit slower than normal
writes. The point is to carry 'retry' signal back in case of resource 
contention.

and 64bytes payload is also an advantage. e.g. for DSA/IAX the whole
work descriptor can be included in the ENQCMD payload.

> 
> A few thoughts come to mind:
> 
>  * Traditional submission queues are no longer needed and can be
>    replaced by an enqueue register. NVMe sqs and VIRTIO avail rings
>    aren't needed anymore, although the sqes and vring descriptors are
>    still needed to represent commands and buffers.

yes, this is one possible enhancement.

> 
>    Or the enqueue register can be used simply as a doorbell to start DMA
>    reading requests from a traditional submission queue. In this case
>    the advantage is that a single shared hardware unit (ADI) can emulate
>    multiple queues at the same time.
> 
>  * In order to support submitting multiple requests in a single enqueue
>    register access there needs to be some kind of chaining mechanism.
>    For example, the Device Specific Command field contains a num_reqs
>    field telling the device how many requests to DMA.

yes, this leaves to device specific format. e.g. DSA defines a chaining
descript type to chain multiple requests together.

> 
> I don't know much about ENQCMD and am trying to figure out where it fits
> in. Please let me know if this matches how this feature is intended to
> be used.
> 

I think you captured most of the intention of  ENQCMD. 😊

Thanks
Kevin