Re: [PATCH V7 00/24] Live update: cpr-transfer

[Fabiano Rosas]

Steve Sistare <steven.sistare@oracle.com> writes:

> What?
>
> This patch series adds the live migration cpr-transfer mode, which
> allows the user to transfer a guest to a new QEMU instance on the same
> host with minimal guest pause time, by preserving guest RAM in place,
> albeit with new virtual addresses in new QEMU, and by preserving device
> file descriptors.
>
> The new user-visible interfaces are:
>   * cpr-transfer (MigMode migration parameter)
>   * cpr (MigrationChannelType)
>   * incoming MigrationChannel (command-line argument)
>   * aux-ram-share (machine option)
>
> The user sets the mode parameter before invoking the migrate command.
> In this mode, the user starts new QEMU on the same host as old QEMU, with
> the same arguments as old QEMU, plus two -incoming options; one for the main
> channel, and one for the CPR channel.  The user issues the migrate command to
> old QEMU, which stops the VM, saves state to the migration channels, and
> enters the postmigrate state.  Execution resumes in new QEMU.
>
> Memory-backend objects must have the share=on attribute, but 
> memory-backend-epc
> is not supported.  The VM must be started with the '-machine aux-ram-share=on'
> option, which allows auxilliary guest memory to be transferred in place to the
> new process.
>
> This mode requires a second migration channel of type "cpr", in the channel
> arguments on the outgoing side, and in a second -incoming command-line
> parameter on the incoming side.  This CPR channel must support file descriptor
> transfer with SCM_RIGHTS, i.e. it must be a UNIX domain socket.
>
> Why?
>
> This mode has less impact on the guest than any other method of updating
> in place.  The pause time is much lower, because devices need not be torn
> down and recreated, DMA does not need to be drained and quiesced, and minimal
> state is copied to new QEMU.  Further, there are no constraints on the guest.
> By contrast, cpr-reboot mode requires the guest to support S3 suspend-to-ram,
> and suspending plus resuming vfio devices adds multiple seconds to the
> guest pause time.
>
> These benefits all derive from the core design principle of this mode,
> which is preserving open descriptors.  This approach is very general and
> can be used to support a wide variety of devices that do not have hardware
> support for live migration, including but not limited to: vfio, chardev,
> vhost, vdpa, and iommufd.  Some devices need new kernel software interfaces
> to allow a descriptor to be used in a process that did not originally open it.
>
> How?
>
> All memory that is mapped by the guest is preserved in place.  Indeed,
> it must be, because it may be the target of DMA requests, which are not
> quiesced during cpr-transfer.  All such memory must be mmap'able in new QEMU.
> This is easy for named memory-backend objects, as long as they are mapped
> shared, because they are visible in the file system in both old and new QEMU.
> Anonymous memory must be allocated using memfd_create rather than MAP_ANON,
> so the memfd's can be sent to new QEMU.  Pages that were locked in memory
> for DMA in old QEMU remain locked in new QEMU, because the descriptor of
> the device that locked them remains open.
>
> cpr-transfer preserves descriptors by sending them to new QEMU via the CPR
> channel, which must support SCM_RIGHTS, and by sending the unique name of
> each descriptor to new QEMU via CPR state.
>
> For device descriptors, new QEMU reuses the descriptor when creating the
> device, rather than opening it again.  For memfd descriptors, new QEMU
> mmap's the preserved memfd when a ramblock is created.
>
> CPR state cannot be sent over the normal migration channel, because devices
> and backends are created prior to reading the channel, so this mode sends
> CPR state over a second "cpr" migration channel.  New QEMU reads the second
> channel prior to creating devices or backends.
>
> Example:
>
> In this example, we simply restart the same version of QEMU, but in
> a real scenario one would use a new QEMU binary path in terminal 2.
>
>   Terminal 1: start old QEMU
>   # qemu-kvm -qmp stdio -object
>   memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on
>   -m 4G -machine aux-ram-share=on ...
>
>   Terminal 2: start new QEMU
>   # qemu-kvm -monitor stdio ... -incoming tcp:0:44444
>     -incoming '{"channel-type": "cpr",
>                 "addr": { "transport": "socket", "type": "unix",
>                           "path": "cpr.sock"}}'
>
>   Terminal 1:
>   {"execute":"qmp_capabilities"}
>
>   {"execute": "query-status"}
>   {"return": {"status": "running",
>               "running": true}}
>
>   {"execute":"migrate-set-parameters",
>    "arguments":{"mode":"cpr-transfer"}}
>
>   {"execute": "migrate", "arguments": { "channels": [
>     {"channel-type": "main",
>      "addr": { "transport": "socket", "type": "inet",
>                "host": "0", "port": "44444" }},
>     {"channel-type": "cpr",
>      "addr": { "transport": "socket", "type": "unix",
>                "path": "cpr.sock" }}]}}
>
>   {"execute": "query-status"}
>   {"return": {"status": "postmigrate",
>               "running": false}}
>
>   Terminal 2:
>   QEMU 10.0.50 monitor - type 'help' for more information
>   (qemu) info status
>   VM status: running
>
> This patch series implements a minimal version of cpr-transfer.  Additional
> series are ready to be posted to deliver the complete vision described
> above, including
>   * vfio
>   * chardev
>   * vhost and tap
>   * blockers
>   * cpr-exec mode
>   * iommufd
>
> Changes in V2:
>   * cpr-transfer is the first new mode proposed, and cpr-exec is deferred
>   * anon-alloc does not apply to memory-backend-object
>   * replaced hack with proper synchronization between source and target
>   * defined QEMU_CPR_FILE_MAGIC
>   * addressed misc review comments
>
> Changes in V3:
>   * added cpr-transfer to migration-test
>   * documented cpr-transfer in CPR.rst
>   * fix size_t trace format for 32-bit build
>   * drop explicit fd value in VMSTATE_FD
>   * defer cpr_walk_fd() and cpr_resave_fd() to later series
>   * drop "migration: save cpr mode".
>     delete mode from cpr state, and use cpr_uri to infer transfer mode.
>   * drop "migration: stop vm earlier for cpr"
>   * dropped cpr helpers, to be re-added later when needed
>   * fixed an unreported bug for cpr-transfer and migrate cancel
>   * documented cpr-transfer restrictions in qapi
>   * added trace for cpr_state_save and cpr_state_load
>   * added ftruncate to "preserve ram blocks"
>
> Changes in V4:
>   * cleaned up qtest deferred connection code
>   * renamed pass_fd -> can_pass_fd
>   * squashed patch "split qmp_migrate"
>   * deleted cpr-uri and its patches
>   * added cpr channel and its patches
>   * added patch "hostmem-shm: preserve for cpr"
>   * added patch "fd-based shared memory"
>   * added patch "factor out allocation of anonymous shared memory"
>   * added RAM_PRIVATE and its patch
>   * added aux-ram-share and its patch
>
> Changes in V5:
>   * added patch 'enhance migrate_uri_parse'
>   * supported dotted keys for -incoming channel,
>     and rewrote incoming_option_parse
>   * moved migrate_fd_cancel -> vm_resume to "stop vm earlier for cpr"
>     in a future series.
>   * updated command-line definition for aux-ram-share
>   * added patch "resizable qemu_ram_alloc_from_fd"
>   * rewrote patch "fd-based shared memory"
>   * fixed error message in qemu_shm_alloc
>   * added patch 'tests/qtest: optimize migrate_set_ports'
>   * added patch 'tests/qtest: enhance migration channels'
>   * added patch 'tests/qtest: assert qmp_ready'
>   * modified patch 'migration-test: cpr-transfer'
>   * polished the documentation in CPR.rst, qapi, and the
>     cpr-transfer mode commit message
>   * updated to master, and resolved massive context diffs for migration tests
>
> Changes in V6:
>   * added RB's and Acks.
>   * in patch "assert qmp_ready", deleted qmp_ready and checked qmp_fd instead.
>     renamed patch to ""assert qmp connected"
>   * factored out fix into new patch
>     "fix qemu_ram_alloc_from_fd size calculation"
>   * deleted a redundant call to migrate_hup_delete
>   * added commit message to "migration: cpr-transfer documentation"
>   * polished the text of cpr-transfer mode in qapi
>
> Changes in V7:
>   * fixed cpr-transfer test failure for s390
>   * fixed machine_get_aux_ram_share compilation error for Windows
>   * fixed size_t print format compilation error for misc architectures
>   * fixed memory leaks in cpr_transfer_output, cpr_transfer_input, and
>     qemu_file_get_fd
>
> The first 10 patches below are foundational and are needed for both 
> cpr-transfer
> mode and the proposed cpr-exec mode.  The next 6 patches are specific to
> cpr-transfer and implement the mechanisms for sharing state across a socket
> using SCM_RIGHTS.  The last 8 patches supply tests and documentation.
>
> Steve Sistare (24):
>   backends/hostmem-shm: factor out allocation of "anonymous shared
>     memory with an fd"
>   physmem: fix qemu_ram_alloc_from_fd size calculation
>   physmem: qemu_ram_alloc_from_fd extensions
>   physmem: fd-based shared memory
>   memory: add RAM_PRIVATE
>   machine: aux-ram-share option
>   migration: cpr-state
>   physmem: preserve ram blocks for cpr
>   hostmem-memfd: preserve for cpr
>   hostmem-shm: preserve for cpr
>   migration: enhance migrate_uri_parse
>   migration: incoming channel
>   migration: SCM_RIGHTS for QEMUFile
>   migration: VMSTATE_FD
>   migration: cpr-transfer save and load
>   migration: cpr-transfer mode
>   migration-test: memory_backend
>   tests/qtest: optimize migrate_set_ports
>   tests/qtest: defer connection
>   migration-test: defer connection
>   tests/qtest: enhance migration channels
>   tests/qtest: assert qmp connected
>   migration-test: cpr-transfer
>   migration: cpr-transfer documentation
>
>  backends/hostmem-epc.c                 |   2 +-
>  backends/hostmem-file.c                |   2 +-
>  backends/hostmem-memfd.c               |  14 ++-
>  backends/hostmem-ram.c                 |   2 +-
>  backends/hostmem-shm.c                 |  51 ++------
>  docs/devel/migration/CPR.rst           | 182 ++++++++++++++++++++++++++-
>  hw/core/machine.c                      |  22 ++++
>  include/exec/memory.h                  |  10 ++
>  include/exec/ram_addr.h                |  13 +-
>  include/hw/boards.h                    |   1 +
>  include/migration/cpr.h                |  33 +++++
>  include/migration/misc.h               |   7 ++
>  include/migration/vmstate.h            |   9 ++
>  include/qemu/osdep.h                   |   1 +
>  meson.build                            |   8 +-
>  migration/cpr-transfer.c               |  71 +++++++++++
>  migration/cpr.c                        | 224 
> +++++++++++++++++++++++++++++++++
>  migration/meson.build                  |   2 +
>  migration/migration.c                  | 139 +++++++++++++++++++-
>  migration/migration.h                  |   4 +-
>  migration/options.c                    |   8 +-
>  migration/qemu-file.c                  |  84 ++++++++++++-
>  migration/qemu-file.h                  |   2 +
>  migration/ram.c                        |   2 +
>  migration/trace-events                 |  11 ++
>  migration/vmstate-types.c              |  24 ++++
>  qapi/migration.json                    |  44 ++++++-
>  qemu-options.hx                        |  34 +++++
>  stubs/vmstate.c                        |   7 ++
>  system/memory.c                        |   4 +-
>  system/physmem.c                       | 150 ++++++++++++++++++----
>  system/trace-events                    |   1 +
>  system/vl.c                            |  43 ++++++-
>  tests/qtest/libqtest.c                 |  86 ++++++++-----
>  tests/qtest/libqtest.h                 |  19 ++-
>  tests/qtest/migration/cpr-tests.c      |  62 +++++++++
>  tests/qtest/migration/framework.c      |  74 +++++++++--
>  tests/qtest/migration/framework.h      |  11 ++
>  tests/qtest/migration/migration-qmp.c  |  53 ++++++--
>  tests/qtest/migration/migration-qmp.h  |  10 +-
>  tests/qtest/migration/migration-util.c |  23 ++--
>  tests/qtest/migration/misc-tests.c     |   9 +-
>  tests/qtest/migration/precopy-tests.c  |   6 +-
>  tests/qtest/virtio-net-failover.c      |   8 +-
>  util/memfd.c                           |  16 ++-
>  util/oslib-posix.c                     |  52 ++++++++
>  util/oslib-win32.c                     |   6 +
>  47 files changed, 1472 insertions(+), 174 deletions(-)
>  create mode 100644 include/migration/cpr.h
>  create mode 100644 migration/cpr-transfer.c
>  create mode 100644 migration/cpr.c
>
> base-commit: e8aa7fdcddfc8589bdc7c973a052e76e8f999455

I'd like to merge this series by the end of the week if possible. Please
take a look at some comments from Markus that were left behind in v5.