Re: [Qemu-devel] [PATCH 0/12 RFC v2] Localhost migration

[Lei Li]

On 07/26/2013 05:41 PM, Paolo Bonzini wrote:
> Il 25/07/2013 22:18, Lei Li ha scritto:
> > Hi,
> >
> > This patch series tries to add localhost migration support to
> > Qemu.
> >
> > When doing localhost migration, the host memory will balloon
> > up during the period, might consume double memories for some time.
> > So we want to add a new live migration mechanism localhost
> > migration.
> >
> > Following I copied from last version that Anthony added for the
> > benefit of the other reviewers:
> >
> > The goal here is to allow "live upgrade" of a running QEMU instance.
> > The work flow would look like this:
> >
> > 1) Guests are running QEMU release 1.6.1
> > 2) Admin installs QEMU release 1.6.2 via RPM or deb
> > 3) Admin does localhost migration with page flipping to use new version
> >     of QEMU.
> >
> > Page flipping is used in order to avoid requiring that there is enough
> > free memory to fit an additional copy of the largest guest which is the
> > requirement today with localhost migration.
> >
> > You can also read from the link below:
> > http://lists.gnu.org/archive/html/qemu-devel/2013-06/msg02577.html
> >
> > The plan is:
> >
> > 1) Add new command to do localhost migration.
> >
> >     The qmp interface introduced like:
> >
> >     { 'command': 'localhost-migrate', 'data': {'uri': 'str'} }
> >
> > 2) Use different mechanism than current live migration.
> >
> >     The very basic work flow like:
> >
> >         qemu on the source (the source and destination are both on localhost)
> >                |
> >                V
> >             Stop VM
> >                |
> >                V
> >            Create threads
> >                |
> >                V
> >         Page flipping through vmspice
> >                |
> >                V
> >         MADV_DONTNEED the ram pages which are already flipped
> >                |
> >                V
> >         Migration completes
> >
> >     As stopping VM first, we expect/resume the page flipping through vmspice
> >     is fast enough to meet *live migration (low downtime).
> >
> > Notes:
> > Currently the work flow is not exactly the same as description
> > above. For the first step, the work flow we implemented is:
> > stop VM and copy ram pages via unix domain socket, MADV_DONTNEED
> > ram pages that already copied. After that, will replace to vmsplice
> > mechanism instead of copying pages.
> >
> > Now it's still a draft version, and as it is implemented separately
> > to the current migration code for a easy start, the next step will
> > be trying to integrate it into the current migration implementation
> > closely. To make sure we are on the right direction that should be
> > headed, please let me know your suggestions on this.
> >
> > For the interface, as Anthony has suggested using a flag or a
> > capability, which one would you prefer or any ideas?
> Using a capability on the source makes sense; I don't think anything
> special is needed on the destination.  The destination just sees a
> special packet telling it that a pipe is available via SCM_RIGHTS; then
> it fetches the file descriptor and uses it for the new protocol.
>
> It looks like this could reuse a lot of the RAM copying hooks that we
> introduced for RDMA.  You shouldn't need to change anything in savevm.c
> or arch_init.c

Hi Paolo,

Thanks for your suggestions!

And sorry for the late reply, I am looking into RDMA implementation and
trying to figure out the way to integrate with its approach as you suggested.

For the interface on the source, I am OK with using a capability.
For the destination, currently we are copying ram pages via unix
domain socket as first step, will replace to pipe later. How about
add a new URI prefix as Michael R.Hines suggested?



> Paolo
>
> > Your comments are very welcome!
> >
> > TODO:
> > - Integrate to the current implement of migration closely?
> > - Introduce a mechanism to exchange a PIPE via SCM_RIGHTS.
> > - benchmark/evaluation.
> >
> > Lei Li (12):
> >    migration: export MIG_STATE_xxx flags
> >    savevm: export qemu_save_device_state()
> >    rename is_active to is_block_active
> >    arch_init: introduce ram_page_save()
> >    arch_init: introduce ram_save_local()
> >    arch_init: add save_local_setup to savevm_ram_handlers
> >    savevm: introduce qemu_savevm_local()
> >    savevm: adjust is_ram check in register_savevm_live()
> >    migration-local: implementation of outgoing part
> >    migration-local: implementation of incoming part
> >    migration-local: add option to command line for local incoming
> >    hmp:add hmp_localhost_migration interface
> >
> >   Makefile.objs                 |    1 +
> >   arch_init.c                   |   110 +++++++++++++++++++
> >   block-migration.c             |    2 +-
> >   hmp-commands.hx               |   17 ++++
> >   hmp.c                         |   13 +++
> >   hmp.h                         |    1 +
> >   include/migration/migration.h |   32 +++++++
> >   include/sysemu/sysemu.h       |    1 +
> >   migration-local.c             |  228 +++++++++++++++++++++++++++++++++++++++++
> >   migration-unix.c              |   60 ++++++++++++
> >   migration.c                   |    8 --
> >   qapi-schema.json              |   14 +++
> >   qemu-options.hx               |    9 ++
> >   qmp-commands.hx               |   22 +++++
> >   savevm.c                      |   100 +++++++++++++++--
> >   vl.c                          |   14 +++
> >   16 files changed, 613 insertions(+), 19 deletions(-)
> >   create mode 100644 migration-local.c


--
Lei