Re: [Qemu-devel] [RFC] docs: vhost-user: add in-band kick/call messages

[Stefan Hajnoczi]

On Mon, Sep 2, 2019 at 2:13 PM Johannes Berg <address@hidden> wrote:
>
> From: Johannes Berg <address@hidden>
>
> For good reason, vhost-user is currently built asynchronously, that
> way better performance can be obtained. However, for certain use
> cases such as simulation, this is problematic.
>
> Consider an event-based simulation in which both the device and CPU
> have are scheduled according to a simulation "calendar". Now, for
> example, consider the CPU sending a command to the device, over a
> vring in the vhost-user protocol. In this case, the CPU must wait
> for the vring update to have been processed, so that the device has
> time to put an entry onto the simulation calendar to obtain time to
> handle the interrupt, before the CPU goes back to scheduling and
> thus updates the simulation calendar or even has the simulation
> move time forward.
>
> This cannot easily achieved with the eventfd based vring kick/call
> design.
>
> Extend the protocol slightly, so that a message can be used for kick
> and call instead, if the new VHOST_USER_PROTOCOL_F_KICK_CALL_MSGS is
> negotiated. This in itself doesn't guarantee synchronisation, but both
> sides can also negotiate VHOST_USER_PROTOCOL_F_REPLY_ACK and thus get
> a reply to this message by setting the need_reply flag, and ensure
> synchronisation this way.
>
> Signed-off-by: Johannes Berg <address@hidden>
> ---
>  docs/interop/vhost-user.rst | 36 ++++++++++++++++++++++++++++++++++--
>  1 file changed, 34 insertions(+), 2 deletions(-)
>
> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> index 7827b710aa0a..1270885fecf2 100644
> --- a/docs/interop/vhost-user.rst
> +++ b/docs/interop/vhost-user.rst
> @@ -2,6 +2,7 @@
>  Vhost-user Protocol
>  ===================
>  :Copyright: 2014 Virtual Open Systems Sarl.
> +:Copyright: 2019 Intel Corporation
>  :Licence: This work is licensed under the terms of the GNU GPL,
>            version 2 or later. See the COPYING file in the top-level
>            directory.
> @@ -785,6 +786,7 @@ Protocol features
>    #define VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD  10
>    #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER  11
>    #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD 12
> +  #define VHOST_USER_PROTOCOL_F_KICK_CALL_MSGS 13
>
>  Master message types
>  --------------------
> @@ -946,7 +948,9 @@ Master message types
>    Bits (0-7) of the payload contain the vring index. Bit 8 is the
>    invalid FD flag. This flag is set when there is no file descriptor
>    in the ancillary data. This signals that polling should be used
> -  instead of waiting for a kick.
> +  instead of waiting for the call. however, if the protocol feature
> +  ``VHOST_USER_PROTOCOL_F_KICK_CALL_MSGS`` has been negotiated it instead
> +  means the updates should be done using the messages.
>
>  ``VHOST_USER_SET_VRING_CALL``
>    :id: 13
> @@ -959,7 +963,9 @@ Master message types
>    Bits (0-7) of the payload contain the vring index. Bit 8 is the
>    invalid FD flag. This flag is set when there is no file descriptor
>    in the ancillary data. This signals that polling will be used
> -  instead of waiting for the call.
> +  instead of waiting for the call; however, if the protocol feature
> +  ``VHOST_USER_PROTOCOL_F_KICK_CALL_MSGS`` has been negotiated it instead
> +  means the updates should be done using the messages.
>
>  ``VHOST_USER_SET_VRING_ERR``
>    :id: 14
> @@ -1190,6 +1196,19 @@ Master message types
>    ancillary data. The GPU protocol is used to inform the master of
>    rendering state and updates. See vhost-user-gpu.rst for details.
>
> +``VHOST_USER_VQ_CALL``

"call" should be "kick".  "kick" is the driver->device request
submission notification and "call" is the device->driver request
completion notification.

> +  :id: 34
> +  :equivalent ioctl: N/A
> +  :slave payload: vring state description
> +  :master payload: N/A
> +
> +  When the ``VHOST_USER_PROTOCOL_F_KICK_CALL_MSGS`` protocol feature has
> +  been successfully negotiated, this message may be submitted by the master
> +  to indicate that a buffer was added to the vring instead of signalling it
> +  using the vring's event FD or having the slave rely on polling.
> +
> +  The state.num field is currently reserved and must be set to 0.

Please include an explanation of how this message is different from
the existing methods.  Maybe something like:

  Unlike eventfd or polling, this message allows the master to control
precisely when virtqueue processing happens.  When the master uses
``need_reply`` to receive a reply, it knows the device has become
aware of the virtqueue activity.

>  Slave message types
>  -------------------
>
> @@ -1246,6 +1265,19 @@ Slave message types
>    ``VHOST_USER_PROTOCOL_F_HOST_NOTIFIER`` protocol feature has been
>    successfully negotiated.
>
> +``VHOST_USER_VQ_KICK``

s/KICK/CALL/