Steven Sistare <steven.sistare@oracle.com> writes:
On 1/17/2025 9:42 AM, Fabiano Rosas wrote:
Steve Sistare <steven.sistare@oracle.com> writes:
Add documentation for the cpr-transfer migration mode.
Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
---
docs/devel/migration/CPR.rst | 182
++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 180 insertions(+), 2 deletions(-)
diff --git a/docs/devel/migration/CPR.rst b/docs/devel/migration/CPR.rst
index 63c3647..d6021d5 100644
--- a/docs/devel/migration/CPR.rst
+++ b/docs/devel/migration/CPR.rst
@@ -5,7 +5,7 @@ CPR is the umbrella name for a set of migration modes in which
the
VM is migrated to a new QEMU instance on the same host. It is
intended for use when the goal is to update host software components
that run the VM, such as QEMU or even the host kernel. At this time,
-cpr-reboot is the only available mode.
+the cpr-reboot and cpr-transfer modes are available.
Because QEMU is restarted on the same host, with access to the same
local devices, CPR is allowed in certain cases where normal migration
@@ -53,7 +53,7 @@ RAM is copied to the migration URI.
Outgoing:
* Set the migration mode parameter to ``cpr-reboot``.
* Set the ``x-ignore-shared`` capability if desired.
- * Issue the ``migrate`` command. It is recommended the the URI be a
+ * Issue the ``migrate`` command. It is recommended the URI be a
``file`` type, but one can use other types such as ``exec``,
provided the command captures all the data from the outgoing side,
and provides all the data to the incoming side.
@@ -145,3 +145,181 @@ Caveats
cpr-reboot mode may not be used with postcopy, background-snapshot,
or COLO.
+
+cpr-transfer mode
+-----------------
+
+This mode allows the user to transfer a guest to a new QEMU instance
+on the same host with minimal guest pause time, by preserving guest
+RAM in place, albeit with new virtual addresses in new QEMU. Devices
+and their pinned memory pages will also be preserved in a future QEMU
+release.
+
+The user starts new QEMU on the same host as old QEMU, with command-
+line arguments to create the same machine, plus the ``-incoming``
+option for the main migration channel, like normal live migration.
+In addition, the user adds a second -incoming option with channel
+type ``cpr``. This CPR channel must support file descriptor transfer
+with SCM_RIGHTS, i.e. it must be a UNIX domain socket.
+
+To initiate CPR, the user issues a migrate command to old QEMU,
+adding a second migration channel of type ``cpr`` in the channels
+argument. Old QEMU stops the VM, saves state to the migration
+channels, and enters the postmigrate state. Execution resumes in
+new QEMU.
+
+New QEMU reads the CPR channel before opening a monitor, hence
+the CPR channel cannot be specified in the list of channels for a
+migrate-incoming command. It may only be specified on the command
+line.
+
+Usage
+^^^^^
+
+Memory backend objects must have the ``share=on`` attribute.
+
+The VM must be started with the ``-machine aux-ram-share=on``
+option. This causes implicit RAM blocks (those not described by
+a memory-backend object) to be allocated by mmap'ing a memfd.
+Examples include VGA and ROM.
+
+Outgoing:
+ * Set the migration mode parameter to ``cpr-transfer``.
+ * Issue the ``migrate`` command, containing a main channel and
+ a cpr channel.
+
+Incoming:
+ * Start new QEMU with two ``-incoming`` options.
+ * If the VM was running when the outgoing ``migrate`` command was
+ issued, then QEMU automatically resumes VM execution.
+
+Caveats
+^^^^^^^
+
+cpr-transfer mode may not be used with postcopy, background-snapshot,
+or COLO.
+
+memory-backend-epc is not supported.
+
+The main incoming migration channel address cannot be a file type.
+
+If the main incoming channel address is an inet socket, then the port
+cannot be 0 (meaning dynamically choose a port).
+
+When using ``-incoming defer``, you must issue the migrate command to
+old QEMU before issuing any monitor commands to new QEMU, because new
+QEMU blocks waiting to read from the cpr channel before starting its
+monitor, and old QEMU does not write to the channel until the migrate
+command is issued. However, new QEMU does not open and read the
+main migration channel until you issue the migrate incoming command.
+
+Example 1: incoming channel
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In these examples, we simply restart the same version of QEMU, but
+in a real scenario one would start new QEMU on the incoming side.
+Note that new QEMU does not print the monitor prompt until old QEMU
+has issued the migrate command. The outgoing side uses QMP because
+HMP cannot specify a CPR channel. Some QMP responses are omitted for
+brevity.
+
+::
+
+ Outgoing: Incoming:
+
+ # qemu-kvm -qmp stdio
+ -object memory-backend-file,id=ram0,size=4G,
+ mem-path=/dev/shm/ram0,share=on -m 4G
+ -machine aux-ram-share=on
+ ...
+ # qemu-kvm -monitor stdio
+ -incoming tcp:0:44444
+ -incoming '{"channel-type": "cpr",
+ "addr": { "transport": "socket",
+ "type": "unix", "path": "cpr.sock"}}'
+ ...
I'm attempting this and not having much success. Surely I'm missing
something:
$ qemu-system-x86_64 -cpu host -smp 16 -machine pc,accel=kvm \
-drive id=drive0,if=none,format=qcow2,file=img.qcow2 \
-device virtio-blk-pci,id=image1,drive=drive0,bootindex=0 \
-qmp unix:./dst-qmp.sock,server,wait=off \
-nographic -serial mon:stdio \
-object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on
\
-m 4G -machine aux-ram-share=on \
-incoming tcp:0:44444 \
-incoming '{"channel-type": "cpr", "addr": { "transport": "socket", "type": "unix",
"path": "cpr.sock"}}' \
-trace loadvm_* -trace cpr_* -trace migration_* -trace migrate_* -trace
qemu_loadvm_*
cpr_transfer_input cpr.sock
cpr_state_load cpr-transfer mode
cpr_find_fd pc.bios, id 0 returns 15
cpr_find_fd pc.rom, id 0 returns 14
cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 13
cpr_find_fd 0000:00:02.0/vga.rom, id 0 returns 12
cpr_find_fd 0000:00:03.0/e1000.rom, id 0 returns 11
cpr_find_fd /rom@etc/acpi/tables, id 0 returns 10
cpr_find_fd /rom@etc/table-loader, id 0 returns 8
cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 3
migrate_set_state new state setup
migration_socket_incoming_accepted
migration_set_incoming_channel ioc=0x564dc31e7000 ioctype=qio-channel-socket
migrate_set_state new state active
loadvm_state_setup
qemu_loadvm_state_section 1
qemu_loadvm_state_section_startfull 2(ram) 0 4
qemu_loadvm_state_section 3
qemu_loadvm_state_section_partend 2
qemu_loadvm_state_section 4
qemu_loadvm_state_section_startfull 0(timer) 0 2
qemu_loadvm_state_section 4
qemu_loadvm_state_section_startfull 1(slirp) 0 4
qemu_loadvm_state_section 4
qemu_loadvm_state_section_startfull 4(cpu_common) 0 1
qemu_loadvm_state_section 4
qemu_loadvm_state_section_startfull 5(cpu) 0 12
qemu_loadvm_state_section 4
qemu_loadvm_state_section_startfull 6(kvm-tpr-opt) 0 1
qemu-system-x86_64: error while loading state for instance 0x0 of device
'kvm-tpr-opt'
qemu_loadvm_state_post_main -1
migrate_set_state new state failed
migrate_error error=load of migration failed: Operation not permitted
loadvm_state_cleanup
qemu-system-x86_64: load of migration failed: Operation not permitted
Check for a mismatch between the qemu args on the source vs dest.
Maybe -cpu.
No.. they're the same:
qemu-system-x86_64 -display none -cpu host -smp 4 -machine pc,accel=kvm
-object
memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m
4G -machine aux-ram-share=on -qmp stdio
qemu-system-x86_64 -display none -cpu host -smp 4 -machine pc,accel=kvm
-object
memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m
4G -machine aux-ram-share=on -incoming tcp:0:44444 -incoming
'{"channel-type": "cpr", "addr": { "transport": "socket", "type":
"unix", "path": "cpr.sock"}}' -monitor stdio
Here's the whole log, see if you spot something:
$ (sleep 5; echo "{ 'execute': 'qmp_capabilities' }
{ 'execute': 'migrate-set-parameters','arguments':{ 'mode':
'cpr-transfer' } }
{ 'execute': 'migrate', 'arguments': \
{ 'channels': [ \
{'channel-type': 'main', 'addr': { 'transport': 'socket',
'type': 'inet', \
'host': '0', 'port': '44444' }}, \
{'channel-type': 'cpr', 'addr': { 'transport': 'socket',
'type': 'unix', \
'path': 'cpr.sock' }} \
]} \
}") | /home/fabiano/qemu-system-x86_64 -display none
-cpu host -smp 4 -machine pc,accel=kvm -qmp stdio
-object
memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on
-m 4G -machine aux-ram-share=on -trace cpr_* -trace
migration_* -trace migrate_* -trace qemu_savevm_*
-trace savevm_*
{"QMP": {"version": {"qemu": {"micro": 50, "minor": 2, "major": 9}, "package":
"v9.2.0-987-gfd4129a8b9"}, "capabilities": ["oob"]}}
cpr_find_fd pc.bios, id 0 returns -1
cpr_save_fd pc.bios, id 0, fd 22
cpr_find_fd pc.rom, id 0 returns -1
cpr_save_fd pc.rom, id 0, fd 23
cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns -1
cpr_save_fd 0000:00:02.0/vga.vram, id 0, fd 24
cpr_find_fd 0000:00:02.0/vga.rom, id 0 returns -1
cpr_save_fd 0000:00:02.0/vga.rom, id 0, fd 26
cpr_find_fd 0000:00:03.0/e1000.rom, id 0 returns -1
cpr_save_fd 0000:00:03.0/e1000.rom, id 0, fd 27
cpr_find_fd /rom@etc/acpi/tables, id 0 returns -1
cpr_save_fd /rom@etc/acpi/tables, id 0, fd 28
cpr_find_fd /rom@etc/table-loader, id 0 returns -1
cpr_save_fd /rom@etc/table-loader, id 0, fd 29
cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns -1
cpr_save_fd /rom@etc/acpi/rsdp, id 0, fd 30
migration_block_activation active-skipped
{"return": {}}
{"return": {}}
migrate_set_state new state setup
cpr_state_save cpr-transfer mode
cpr_transfer_output cpr.sock
{"return": {}}
migration_socket_outgoing_connected hostname=0
migration_set_outgoing_channel ioc=0x55748ba10270 ioctype=qio-channel-socket
hostname=0 err=(nil)
{"timestamp": {"seconds": 1737127025, "microseconds": 606998}, "event": "STOP"}
migration_completion_vm_stop ret 0
migration_transferred_bytes qemu_file 224 multifd 0 RDMA 0
savevm_state_header
savevm_state_setup
migration_bitmap_sync_start
migration_bitmap_sync_end dirty_pages 0
migrate_set_state new state active
migration_thread_setup_complete
migration_transferred_bytes qemu_file 506 multifd 0 RDMA 0
migrate_pending_estimate estimate pending size 0 (pre = 0 post=0)
migration_thread_low_pending 0
migrate_set_state new state device
migration_block_activation inactive
migration_precopy_complete
savevm_section_start ram, section_id 2
migration_bitmap_sync_start
migration_bitmap_sync_end dirty_pages 0
savevm_section_end ram, section_id 2 -> 0
savevm_section_start timer, section_id 0
savevm_section_end timer, section_id 0 -> 0
savevm_section_start slirp, section_id 1
savevm_section_end slirp, section_id 1 -> 0
savevm_section_start cpu_common, section_id 4
savevm_section_end cpu_common, section_id 4 -> 0
savevm_section_start cpu, section_id 5
savevm_section_end cpu, section_id 5 -> 0
savevm_section_start kvm-tpr-opt, section_id 6
savevm_section_end kvm-tpr-opt, section_id 6 -> 0
savevm_section_start apic, section_id 7
savevm_section_end apic, section_id 7 -> 0
savevm_section_start cpu_common, section_id 8
savevm_section_end cpu_common, section_id 8 -> 0
savevm_section_start cpu, section_id 9
savevm_section_end cpu, section_id 9 -> 0
savevm_section_start apic, section_id 10
savevm_section_end apic, section_id 10 -> 0
savevm_section_start cpu_common, section_id 11
savevm_section_end cpu_common, section_id 11 -> 0
savevm_section_start cpu, section_id 12
savevm_section_end cpu, section_id 12 -> 0
savevm_section_start apic, section_id 13
savevm_section_end apic, section_id 13 -> 0
savevm_section_start cpu_common, section_id 14
savevm_section_end cpu_common, section_id 14 -> 0
savevm_section_start cpu, section_id 15
savevm_section_end cpu, section_id 15 -> 0
savevm_section_start apic, section_id 16
savevm_section_end apic, section_id 16 -> 0
savevm_section_start kvmclock, section_id 17
savevm_section_end kvmclock, section_id 17 -> 0
savevm_section_start 0000:00:00.0/I440FX, section_id 18
savevm_section_end 0000:00:00.0/I440FX, section_id 18 -> 0
savevm_section_start PCIHost, section_id 19
savevm_section_end PCIHost, section_id 19 -> 0
savevm_section_start PCIBUS, section_id 20
savevm_section_end PCIBUS, section_id 20 -> 0
savevm_section_start fw_cfg, section_id 21
savevm_section_end fw_cfg, section_id 21 -> 0
savevm_section_start dma, section_id 22
savevm_section_end dma, section_id 22 -> 0
savevm_section_start dma, section_id 23
savevm_section_end dma, section_id 23 -> 0
savevm_section_start mc146818rtc, section_id 24
savevm_section_end mc146818rtc, section_id 24 -> 0
savevm_section_start 0000:00:01.1/ide, section_id 25
savevm_section_end 0000:00:01.1/ide, section_id 25 -> 0
savevm_section_start i2c_bus, section_id 26
savevm_section_end i2c_bus, section_id 26 -> 0
savevm_section_start 0000:00:01.3/piix4_pm, section_id 27
savevm_section_end 0000:00:01.3/piix4_pm, section_id 27 -> 0
savevm_section_start 0000:00:01.0/PIIX3, section_id 28
savevm_section_end 0000:00:01.0/PIIX3, section_id 28 -> 0
savevm_section_start i8259, section_id 29
savevm_section_end i8259, section_id 29 -> 0
savevm_section_start i8259, section_id 30
savevm_section_end i8259, section_id 30 -> 0
savevm_section_start ioapic, section_id 31
savevm_section_end ioapic, section_id 31 -> 0
savevm_section_start 0000:00:02.0/vga, section_id 32
savevm_section_end 0000:00:02.0/vga, section_id 32 -> 0
savevm_section_start hpet, section_id 33
savevm_section_end hpet, section_id 33 -> 0
savevm_section_start i8254, section_id 34
savevm_section_end i8254, section_id 34 -> 0
savevm_section_start pcspk, section_id 35
savevm_section_end pcspk, section_id 35 -> 0
savevm_section_start serial, section_id 36
savevm_section_end serial, section_id 36 -> 0
savevm_section_start parallel_isa, section_id 37
savevm_section_end parallel_isa, section_id 37 -> 0
savevm_section_start fdc, section_id 38
savevm_section_end fdc, section_id 38 -> 0
savevm_section_start ps2kbd, section_id 39
savevm_section_end ps2kbd, section_id 39 -> 0
savevm_section_start ps2mouse, section_id 40
savevm_section_end ps2mouse, section_id 40 -> 0
savevm_section_start pckbd, section_id 41
savevm_section_end pckbd, section_id 41 -> 0
savevm_section_start vmmouse, section_id 42
savevm_section_end vmmouse, section_id 42 -> 0
savevm_section_start port92, section_id 43
savevm_section_end port92, section_id 43 -> 0
savevm_section_start 0000:00:03.0/e1000, section_id 44
savevm_section_end 0000:00:03.0/e1000, section_id 44 -> 0
savevm_section_skip smbus-eeprom, section_id 45
savevm_section_skip smbus-eeprom, section_id 46
savevm_section_skip smbus-eeprom, section_id 47
savevm_section_skip smbus-eeprom, section_id 48
savevm_section_skip smbus-eeprom, section_id 49
savevm_section_skip smbus-eeprom, section_id 50
savevm_section_skip smbus-eeprom, section_id 51
savevm_section_skip smbus-eeprom, section_id 52
savevm_section_start acpi_build, section_id 53
savevm_section_end acpi_build, section_id 53 -> 0
savevm_section_start globalstate, section_id 54
migrate_global_state_pre_save saved state: running
savevm_section_end globalstate, section_id 54 -> 0
migrate_error error=Unable to write to socket: Connection reset by peer
migrate_set_state new state failed
migration_thread_after_loop
migration_block_activation active
{"timestamp": {"seconds": 1737127025, "microseconds": 625404}, "event":
"RESUME"}
migrate_fd_cleanup
savevm_state_cleanup
qemu-system-x86_64: Unable to write to socket: Connection reset by peer
$ /home/fabiano/qemu-system-x86_64 -display none -cpu host -smp 4
-machine pc,accel=kvm -object
memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m
4G -machine aux-ram-share=on -incoming tcp:0:44444 -incoming
'{"channel-type": "cpr", "addr": { "transport": "socket", "type":
"unix", "path": "cpr.sock"}}' -trace loadvm_* -trace cpr_* -trace
migration_* -trace migrate_* -monitor stdio
cpr_transfer_input cpr.sock
cpr_state_load cpr-transfer mode
QEMU 9.2.50 monitor - type 'help' for more information
cpr_find_fd pc.bios, id 0 returns 15
cpr_find_fd pc.rom, id 0 returns 14
cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 13
cpr_find_fd 0000:00:02.0/vga.rom, id 0 returns 12
cpr_find_fd 0000:00:03.0/e1000.rom, id 0 returns 11
cpr_find_fd /rom@etc/acpi/tables, id 0 returns 10
cpr_find_fd /rom@etc/table-loader, id 0 returns 8
cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 3
migrate_set_state new state setup
(qemu) migration_socket_incoming_accepted
migration_set_incoming_channel ioc=0x5565cccb8e70 ioctype=qio-channel-socket
migrate_set_state new state active
loadvm_state_setup
qemu-system-x86_64: error while loading state for instance 0x0 of device
'kvm-tpr-opt'
migrate_set_state new state failed
migrate_error error=load of migration failed: Operation not permitted
loadvm_state_cleanup
qemu-system-x86_64: load of migration failed: Operation not permitted