[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH v8] introduce vfio-user protocol specification
From: |
John Levon |
Subject: |
Re: [PATCH v8] introduce vfio-user protocol specification |
Date: |
Wed, 19 May 2021 22:38:48 +0000 |
On Wed, May 19, 2021 at 03:08:17PM -0600, Alex Williamson wrote:
> > +VFIO_USER_DMA_MAP
> > +-----------------
> > +
> > +Message Format
> > +^^^^^^^^^^^^^^
> > +
> > ++--------------+------------------------+
> > +| Name | Value |
> > ++==============+========================+
> > +| Message ID | <ID> |
> > ++--------------+------------------------+
> > +| Command | 2 |
> > ++--------------+------------------------+
> > +| Message size | 16 + table size |
> > ++--------------+------------------------+
> > +| Flags | Reply bit set in reply |
> > ++--------------+------------------------+
> > +| Error | 0/errno |
> > ++--------------+------------------------+
> > +| Table | array of table entries |
> > ++--------------+------------------------+
> > +
> > +This command message is sent by the client to the server to inform it of
> > the
> > +memory regions the server can access. It must be sent before the server can
> > +perform any DMA to the client. It is normally sent directly after the
> > version
> > +handshake is completed, but may also occur when memory is added to the
> > client,
> > +or if the client uses a vIOMMU. If the client does not expect the server to
> > +perform DMA then it does not need to send to the server VFIO_USER_DMA_MAP
> > +commands. If the server does not need to perform DMA then it can ignore
> > such
> > +commands but it must still reply to them. The table is an array of the
> > +following structure:
> > +
> > +Table entry format
> > +^^^^^^^^^^^^^^^^^^
> > +
> > ++-------------+--------+-------------+
> > +| Name | Offset | Size |
> > ++=============+========+=============+
> > +| Address | 0 | 8 |
> > ++-------------+--------+-------------+
> > +| Size | 8 | 8 |
> > ++-------------+--------+-------------+
> > +| Offset | 16 | 8 |
> > ++-------------+--------+-------------+
> > +| Protections | 24 | 4 |
> > ++-------------+--------+-------------+
> > +| Flags | 28 | 4 |
> > ++-------------+--------+-------------+
> > +| | +-----+------------+ |
> > +| | | Bit | Definition | |
> > +| | +=====+============+ |
> > +| | | 0 | Mappable | |
> > +| | +-----+------------+ |
> > ++-------------+--------+-------------+
> > +
> > +* *Address* is the base DMA address of the region.
> > +* *Size* is the size of the region.
> > +* *Offset* is the file offset of the region with respect to the associated
> > file
> > + descriptor.
>
> It might help to explicitly state this value is ignored by the server
> for non-mappable DMA, otherwise a server might assume a specific value
> is required (ex. zero) for such cases.
Generally we say that unused inputs must be zero, but yes, this should be
clarified, thanks.
> > +* *Protections* are the region's protection attributes as encoded in
> > + ``<sys/mman.h>``.
> > +* *Flags* contains the following region attributes:
> > +
> > + * *Mappable* indicates that the region can be mapped via the mmap()
> > system
> > + call using the file descriptor provided in the message meta-data.
> > +
> > +This structure is 32 bytes in size, so the message size is:
> > +16 + (# of table entries * 32).
> > +
> > +If a DMA region being added can be directly mapped by the server, an array
> > of
> > +file descriptors must be sent as part of the message meta-data. Each
> > mappable
> > +region entry must have a corresponding file descriptor. On AF_UNIX
> > sockets, the
>
> Is this saying that if the client provides table entries where indexes
> 1, 3, and 5 are indicated as mappable, then there must be an ancillary
> file descriptor array with 3 entries, where fd[0] maps to entry[1],
> fd[1]:entry[3], and fd[2]:entry[5], even if fd[0-2] are all the
> same file descriptor?
Yes. Though we are planning to change these calls to only support single regions
which would make this moot.
> > +file descriptors must be passed as SCM_RIGHTS type ancillary data.
> > Otherwise,
> > +if a DMA region cannot be directly mapped by the server, it can be
> > accessed by
> > +the server using VFIO_USER_DMA_READ and VFIO_USER_DMA_WRITE messages,
> > explained
> > +in `Read and Write Operations`_. A command to map over an existing region
> > must
> > +be failed by the server with ``EEXIST`` set in error field in the reply.
> > +
> > +Adding multiple DMA regions can partially fail. The response does not
> > indicate
> > +which regions were added and which were not, therefore it is a client
> > +implementation detail how to recover from the failure.
> > +
> > +.. Note::
> > + The server can optionally remove succesfully added DMA regions making
> > this
> > + operation atomic.
> > + The client can recover by attempting to unmap one by one all the DMA
> > regions
> > + in the VFIO_USER_DMA_MAP command, ignoring failures for regions that do
> > not
> > + exist.
>
> What's the benefit of specifying this server behavior as optional? I'm
> afraid this unspecified error recovery behavior might actually deter
> clients from performing batch mappings. Servers also have little
> incentive to do their own cleanup if the client has no way to detect
> that behavior.
This may well be moot too.
> > +VFIO_USER_DMA_UNMAP
> > +-------------------
> > +
> > +This command message is sent by the client to the server to inform it that
> > a
> > +DMA region, previously made available via a VFIO_USER_DMA_MAP command
> > message,
> > +is no longer available for DMA. It typically occurs when memory is
> > subtracted
> > +from the client or if the client uses a vIOMMU. If the client does not
> > expect
> > +the server to perform DMA then it does not need to send to the server
> > +VFIO_USER_DMA_UNMAP commands. If the server does not need to perform DMA
> > then
> > +it can ignore such commands but it must still reply to them. The table is
> > an
>
> I'm confused why expectation of DMA plays a factor here. For example,
> if QEMU unplugs a DIMM and the server has an mmap of the file descriptor
> related to that DIMM, does it get to retain the mmap if it doesn't
> currently have any DMA queued targeting that address range? Can QEMU
> skip sending an unmap if the PCI bus master bit is disabled on the
> device preventing further DMA? How can the associated file descriptor
> get released? This doesn't feel strongly specified.
I thought we'd removed those sentences actually, as they're just confusing. In
reality, everything is going to both send and handle map/unmap messages.
> Are there any assumptions about address and size of the unmap command
> relative to the original map command or is the client freely allowed to
> bisect, overlap, or overextend previous mappings?
Good question. Filed https://github.com/nutanix/libvfio-user/issues/504 to
track this.
I actually don't know what clients would like to be able to do in this respect.
> > +* *Offset* is the file offset of the region with respect to the associated
> > file
> > + descriptor.
> > +* *Protections* are the region's protection attributes as encoded in
> > + ``<sys/mman.h>``.
> > +* *Flags* contains the following region attributes:
> > +
> > + * *VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP* indicates that a dirty page
> > bitmap
> > + must be populated before unmapping the DMA region. The client must
> > provide
> > + a ``struct vfio_bitmap`` in the VFIO bitmaps field for each region,
> > with
> > + the ``vfio_bitmap.pgsize`` and ``vfio_bitmap.size`` fields initialized.
> > +
> > +* *VFIO Bitmaps* contains one ``struct vfio_bitmap`` per region (explained
> > + below) if ``VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP`` is set in Flags.
>
> How will this be extended when new flags are added to get new data or
> new data formats? Note for instance that the kernel struct
> vfio_iommu_type1_dma_unmap specifies the data[] as opaque in general and
> only specifies it as struct vfio_bitmap for the case where
> GET_DIRTY_BITMAP is specified.
We're planning to do something like that for the new (actually more like vfio
again) map/unmap format.
> > +Upon receiving a VFIO_USER_DMA_UNMAP command, if the file descriptor is
> > mapped
> > +then the server must release all references to that DMA region before
> > replying,
> > +which includes potentially in flight DMA transactions. Removing a portion
> > of a
> > +DMA region is possible.
>
> Ah, maybe this answers my question about unmap vs map, but it also seems
> to contradict the description allowing the server to ignore unmap
> requests if no DMA is expected when we state here that the server MUST
> release references.
The text is confusing (which is why I've removed it again). What it's really
trying to say is:
If there is a server implementation (such as the gpio-pci-idio-16 sample) that
never needs to access guest memory at all, then the server can choose to ignore
DMA_MAP/UNMAP - so it would never keep any references around in the first place.
It's not a useful thing to mention in the spec IMHO.
> Is this also a good place to point out that the max messages size of
> 4096 is extremely limiting for returning a dirty bitmap for most use
> cases? Some discussion of the error codes for such a case might be
> relevant here too.
It's a silly low default. The only implementation so far reports 65536 for what
it's worth.
We are also prototyping implementation changes such that this limit can be
removed altogether; hopefully that will come in a future spec update.
> > +VFIO_USER_VM_INTERRUPT
> > +
> > +Interrupt info format
> > +^^^^^^^^^^^^^^^^^^^^^
> > +
> > ++-----------+--------+------+
> > +| Name | Offset | Size |
> > ++===========+========+======+
> > +| Sub-index | 16 | 4 |
> > ++-----------+--------+------+
> > +
> > +* *Sub-index* is relative to the IRQ index, e.g., the vector number used
> > in PCI
> > + MSI/X type interrupts.
>
> Sorry if I'm blind, but where is the index itself provided?
You were confused because the message makes no sense as it's defined. It's been
removed.
Thanks for taking a look, much appreciated!
regards
john
- Re: [PATCH v8] introduce vfio-user protocol specification, (continued)
Re: [PATCH v8] introduce vfio-user protocol specification, Alex Williamson, 2021/05/19
- Re: [PATCH v8] introduce vfio-user protocol specification,
John Levon <=