Re: [PATCH v2 1/1] docs/devel: Add VFIO device migration documentation

[Cornelia Huck]

On Tue, 16 Mar 2021 21:48:38 +0530
"Tarun Gupta (SW-GPU)" <targupta@nvidia.com> wrote:

> On 3/15/2021 10:52 PM, Cornelia Huck wrote:
> > 
> > On Thu, 11 Mar 2021 00:50:09 +0530
> > Tarun Gupta <targupta@nvidia.com> wrote:
> >   
> >> Document interfaces used for VFIO device migration. Added flow of state 
> >> changes
> >> during live migration with VFIO device. Tested by building docs with the 
> >> new
> >> vfio-migration.rst file.
> >>
> >> v2:
> >> - Included the new vfio-migration.rst file in index.rst
> >> - Updated dirty page tracking section, also added details about
> >>    'pre-copy-dirty-page-tracking' opt-out option.
> >> - Incorporated comments around wording of doc.
> >>
> >> Signed-off-by: Tarun Gupta <targupta@nvidia.com>
> >> Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
> >> ---
> >>   MAINTAINERS                   |   1 +
> >>   docs/devel/index.rst          |   1 +
> >>   docs/devel/vfio-migration.rst | 135 ++++++++++++++++++++++++++++++++++
> >>   3 files changed, 137 insertions(+)
> >>   create mode 100644 docs/devel/vfio-migration.rst
> >>  
> > 
> > (...)
> >   
> >> diff --git a/docs/devel/vfio-migration.rst b/docs/devel/vfio-migration.rst
> >> new file mode 100644
> >> index 0000000000..6196fb132c
> >> --- /dev/null
> >> +++ b/docs/devel/vfio-migration.rst
> >> @@ -0,0 +1,135 @@
> >> +=====================
> >> +VFIO device Migration
> >> +=====================  
> > 
> > Maybe add an introductory sentence or two describing the general
> > approach? I.e. we have a general framework, and specific support for
> > devices needs to be hooked up.  
> 
> Ummm, the below paragraph does describe the approach we're using for the 
> migration framework involving pre-copy and stop-and-copy phase.
> Can you help elaborate more on the general approach you'd like to have?

The document dives right in with how vfio devices are using an
iterative approach etc. A quick overview of the general setup (before
you are getting to the different phases) might be helpful, i.e. who
does what. Not sure what we expect a reader of this document to know
already.

> 
> >   
> >> +
> >> +VFIO devices use an iterative approach for migration because certain VFIO
> >> +devices (e.g. GPU) have large amount of data to be transfered. The 
> >> iterative
> >> +pre-copy phase of migration allows for the guest to continue whilst the 
> >> VFIO
> >> +device state is transferred to the destination, this helps to reduce the 
> >> total
> >> +downtime of the VM. VFIO devices can choose to skip the pre-copy phase of
> >> +migration by returning pending_bytes as zero during the pre-copy phase.  
> > 
> > What about something like:
> > 
> > "Migration of VFIO devices consists of two phases: the optional
> > pre-copy phase, and the stop-and-copy phase. The pre-copy phase is
> > iterative and allows to accommodate VFIO devices that have a large
> > amount of data that needs to be transferred. The iterative pre-copy
> > phase..."
> >   
> 
> Thanks, this looks better. I'll update it in next version incorporating 
> the other comments too below.