Re: [PATCH V7 24/24] migration: cpr-transfer documentation

[Fabiano Rosas]

Steven Sistare <steven.sistare@oracle.com> writes:

> On 1/17/2025 9:42 AM, Fabiano Rosas wrote:
>> Steve Sistare <steven.sistare@oracle.com> writes:
>> 
>>> Add documentation for the cpr-transfer migration mode.
>>>
>>> Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
>>> Reviewed-by: Peter Xu <peterx@redhat.com>
>>> ---
>>>   docs/devel/migration/CPR.rst | 182 
>>> ++++++++++++++++++++++++++++++++++++++++++-
>>>   1 file changed, 180 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/docs/devel/migration/CPR.rst b/docs/devel/migration/CPR.rst
>>> index 63c3647..d6021d5 100644
>>> --- a/docs/devel/migration/CPR.rst
>>> +++ b/docs/devel/migration/CPR.rst
>>> @@ -5,7 +5,7 @@ CPR is the umbrella name for a set of migration modes in 
>>> which the
>>>   VM is migrated to a new QEMU instance on the same host.  It is
>>>   intended for use when the goal is to update host software components
>>>   that run the VM, such as QEMU or even the host kernel.  At this time,
>>> -cpr-reboot is the only available mode.
>>> +the cpr-reboot and cpr-transfer modes are available.
>>>   
>>>   Because QEMU is restarted on the same host, with access to the same
>>>   local devices, CPR is allowed in certain cases where normal migration
>>> @@ -53,7 +53,7 @@ RAM is copied to the migration URI.
>>>   Outgoing:
>>>     * Set the migration mode parameter to ``cpr-reboot``.
>>>     * Set the ``x-ignore-shared`` capability if desired.
>>> -  * Issue the ``migrate`` command.  It is recommended the the URI be a
>>> +  * Issue the ``migrate`` command.  It is recommended the URI be a
>>>       ``file`` type, but one can use other types such as ``exec``,
>>>       provided the command captures all the data from the outgoing side,
>>>       and provides all the data to the incoming side.
>>> @@ -145,3 +145,181 @@ Caveats
>>>   
>>>   cpr-reboot mode may not be used with postcopy, background-snapshot,
>>>   or COLO.
>>> +
>>> +cpr-transfer mode
>>> +-----------------
>>> +
>>> +This mode allows the user to transfer a guest to a new QEMU instance
>>> +on the same host with minimal guest pause time, by preserving guest
>>> +RAM in place, albeit with new virtual addresses in new QEMU.  Devices
>>> +and their pinned memory pages will also be preserved in a future QEMU
>>> +release.
>>> +
>>> +The user starts new QEMU on the same host as old QEMU, with command-
>>> +line arguments to create the same machine, plus the ``-incoming``
>>> +option for the main migration channel, like normal live migration.
>>> +In addition, the user adds a second -incoming option with channel
>>> +type ``cpr``.  This CPR channel must support file descriptor transfer
>>> +with SCM_RIGHTS, i.e. it must be a UNIX domain socket.
>>> +
>>> +To initiate CPR, the user issues a migrate command to old QEMU,
>>> +adding a second migration channel of type ``cpr`` in the channels
>>> +argument.  Old QEMU stops the VM, saves state to the migration
>>> +channels, and enters the postmigrate state.  Execution resumes in
>>> +new QEMU.
>>> +
>>> +New QEMU reads the CPR channel before opening a monitor, hence
>>> +the CPR channel cannot be specified in the list of channels for a
>>> +migrate-incoming command.  It may only be specified on the command
>>> +line.
>>> +
>>> +Usage
>>> +^^^^^
>>> +
>>> +Memory backend objects must have the ``share=on`` attribute.
>>> +
>>> +The VM must be started with the ``-machine aux-ram-share=on``
>>> +option.  This causes implicit RAM blocks (those not described by
>>> +a memory-backend object) to be allocated by mmap'ing a memfd.
>>> +Examples include VGA and ROM.
>>> +
>>> +Outgoing:
>>> +  * Set the migration mode parameter to ``cpr-transfer``.
>>> +  * Issue the ``migrate`` command, containing a main channel and
>>> +    a cpr channel.
>>> +
>>> +Incoming:
>>> +  * Start new QEMU with two ``-incoming`` options.
>>> +  * If the VM was running when the outgoing ``migrate`` command was
>>> +    issued, then QEMU automatically resumes VM execution.
>>> +
>>> +Caveats
>>> +^^^^^^^
>>> +
>>> +cpr-transfer mode may not be used with postcopy, background-snapshot,
>>> +or COLO.
>>> +
>>> +memory-backend-epc is not supported.
>>> +
>>> +The main incoming migration channel address cannot be a file type.
>>> +
>>> +If the main incoming channel address is an inet socket, then the port
>>> +cannot be 0 (meaning dynamically choose a port).
>>> +
>>> +When using ``-incoming defer``, you must issue the migrate command to
>>> +old QEMU before issuing any monitor commands to new QEMU, because new
>>> +QEMU blocks waiting to read from the cpr channel before starting its
>>> +monitor, and old QEMU does not write to the channel until the migrate
>>> +command is issued.  However, new QEMU does not open and read the
>>> +main migration channel until you issue the migrate incoming command.
>>> +
>>> +Example 1: incoming channel
>>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> +
>>> +In these examples, we simply restart the same version of QEMU, but
>>> +in a real scenario one would start new QEMU on the incoming side.
>>> +Note that new QEMU does not print the monitor prompt until old QEMU
>>> +has issued the migrate command.  The outgoing side uses QMP because
>>> +HMP cannot specify a CPR channel.  Some QMP responses are omitted for
>>> +brevity.
>>> +
>>> +::
>>> +
>>> +  Outgoing:                             Incoming:
>>> +
>>> +  # qemu-kvm -qmp stdio
>>> +  -object memory-backend-file,id=ram0,size=4G,
>>> +  mem-path=/dev/shm/ram0,share=on -m 4G
>>> +  -machine aux-ram-share=on
>>> +  ...
>>> +                                        # qemu-kvm -monitor stdio
>>> +                                        -incoming tcp:0:44444
>>> +                                        -incoming '{"channel-type": "cpr",
>>> +                                          "addr": { "transport": "socket",
>>> +                                          "type": "unix", "path": 
>>> "cpr.sock"}}'
>>> +                                        ...
>> 
>> I'm attempting this and not having much success. Surely I'm missing
>> something:
>> 
>> 
>> $ qemu-system-x86_64 -cpu host -smp 16 -machine pc,accel=kvm \
>>    -drive id=drive0,if=none,format=qcow2,file=img.qcow2 \
>>    -device virtio-blk-pci,id=image1,drive=drive0,bootindex=0 \
>>    -qmp unix:./dst-qmp.sock,server,wait=off \
>>    -nographic -serial mon:stdio \
>>    -object 
>> memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on \
>>    -m 4G -machine aux-ram-share=on \
>> 
>>    -incoming tcp:0:44444 \
>>    -incoming '{"channel-type": "cpr", "addr": { "transport": "socket", 
>> "type": "unix", "path": "cpr.sock"}}' \
>> 
>>    -trace loadvm_* -trace cpr_* -trace migration_* -trace migrate_* -trace 
>> qemu_loadvm_*
>> 
>> cpr_transfer_input cpr.sock
>> cpr_state_load cpr-transfer mode
>> cpr_find_fd pc.bios, id 0 returns 15
>> cpr_find_fd pc.rom, id 0 returns 14
>> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 13
>> cpr_find_fd 0000:00:02.0/vga.rom, id 0 returns 12
>> cpr_find_fd 0000:00:03.0/e1000.rom, id 0 returns 11
>> cpr_find_fd /rom@etc/acpi/tables, id 0 returns 10
>> cpr_find_fd /rom@etc/table-loader, id 0 returns 8
>> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 3
>> migrate_set_state new state setup
>> migration_socket_incoming_accepted
>> migration_set_incoming_channel ioc=0x564dc31e7000 ioctype=qio-channel-socket
>> migrate_set_state new state active
>> loadvm_state_setup
>> qemu_loadvm_state_section 1
>> qemu_loadvm_state_section_startfull 2(ram) 0 4
>> qemu_loadvm_state_section 3
>> qemu_loadvm_state_section_partend 2
>> qemu_loadvm_state_section 4
>> qemu_loadvm_state_section_startfull 0(timer) 0 2
>> qemu_loadvm_state_section 4
>> qemu_loadvm_state_section_startfull 1(slirp) 0 4
>> qemu_loadvm_state_section 4
>> qemu_loadvm_state_section_startfull 4(cpu_common) 0 1
>> qemu_loadvm_state_section 4
>> qemu_loadvm_state_section_startfull 5(cpu) 0 12
>> qemu_loadvm_state_section 4
>> qemu_loadvm_state_section_startfull 6(kvm-tpr-opt) 0 1
>> qemu-system-x86_64: error while loading state for instance 0x0 of device 
>> 'kvm-tpr-opt'
>> qemu_loadvm_state_post_main -1
>> migrate_set_state new state failed
>> migrate_error error=load of migration failed: Operation not permitted
>> loadvm_state_cleanup
>> qemu-system-x86_64: load of migration failed: Operation not permitted
>
> Check for a mismatch between the qemu args on the source vs dest.
> Maybe -cpu.

No.. they're the same:

qemu-system-x86_64 -display none -cpu host -smp 4 -machine pc,accel=kvm
-object
memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m
4G -machine aux-ram-share=on -qmp stdio

qemu-system-x86_64 -display none -cpu host -smp 4 -machine pc,accel=kvm
-object
memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m
4G -machine aux-ram-share=on -incoming tcp:0:44444 -incoming
'{"channel-type": "cpr", "addr": { "transport": "socket", "type":
"unix", "path": "cpr.sock"}}' -monitor stdio


Here's the whole log, see if you spot something:

$ (sleep 5; echo "{ 'execute': 'qmp_capabilities' }
                 { 'execute': 'migrate-set-parameters','arguments':{ 'mode': 
'cpr-transfer' } }
                 { 'execute': 'migrate', 'arguments': \
                   { 'channels': [ \
                     {'channel-type': 'main', 'addr': { 'transport': 'socket', 
'type': 'inet', \
                                      'host': '0', 'port': '44444' }}, \
                     {'channel-type': 'cpr', 'addr': { 'transport': 'socket', 
'type': 'unix', \
                                      'path': 'cpr.sock' }} \
                   ]} \
                 }") | /home/fabiano/qemu-system-x86_64 -display none
                 -cpu host -smp 4 -machine pc,accel=kvm -qmp stdio
                 -object
                 
memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on
                 -m 4G -machine aux-ram-share=on -trace cpr_* -trace
                 migration_* -trace migrate_* -trace qemu_savevm_*
                 -trace savevm_*

{"QMP": {"version": {"qemu": {"micro": 50, "minor": 2, "major": 9}, "package": 
"v9.2.0-987-gfd4129a8b9"}, "capabilities": ["oob"]}}
cpr_find_fd pc.bios, id 0 returns -1
cpr_save_fd pc.bios, id 0, fd 22
cpr_find_fd pc.rom, id 0 returns -1
cpr_save_fd pc.rom, id 0, fd 23
cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns -1
cpr_save_fd 0000:00:02.0/vga.vram, id 0, fd 24
cpr_find_fd 0000:00:02.0/vga.rom, id 0 returns -1
cpr_save_fd 0000:00:02.0/vga.rom, id 0, fd 26
cpr_find_fd 0000:00:03.0/e1000.rom, id 0 returns -1
cpr_save_fd 0000:00:03.0/e1000.rom, id 0, fd 27
cpr_find_fd /rom@etc/acpi/tables, id 0 returns -1
cpr_save_fd /rom@etc/acpi/tables, id 0, fd 28
cpr_find_fd /rom@etc/table-loader, id 0 returns -1
cpr_save_fd /rom@etc/table-loader, id 0, fd 29
cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns -1
cpr_save_fd /rom@etc/acpi/rsdp, id 0, fd 30
migration_block_activation active-skipped
{"return": {}}
{"return": {}}
migrate_set_state new state setup
cpr_state_save cpr-transfer mode
cpr_transfer_output cpr.sock
{"return": {}}
migration_socket_outgoing_connected hostname=0
migration_set_outgoing_channel ioc=0x55748ba10270 ioctype=qio-channel-socket 
hostname=0 err=(nil)
{"timestamp": {"seconds": 1737127025, "microseconds": 606998}, "event": "STOP"}
migration_completion_vm_stop ret 0
migration_transferred_bytes qemu_file 224 multifd 0 RDMA 0
savevm_state_header
savevm_state_setup
migration_bitmap_sync_start
migration_bitmap_sync_end dirty_pages 0
migrate_set_state new state active
migration_thread_setup_complete
migration_transferred_bytes qemu_file 506 multifd 0 RDMA 0
migrate_pending_estimate estimate pending size 0 (pre = 0 post=0)
migration_thread_low_pending 0
migrate_set_state new state device
migration_block_activation inactive
migration_precopy_complete
savevm_section_start ram, section_id 2
migration_bitmap_sync_start
migration_bitmap_sync_end dirty_pages 0
savevm_section_end ram, section_id 2 -> 0
savevm_section_start timer, section_id 0
savevm_section_end timer, section_id 0 -> 0
savevm_section_start slirp, section_id 1
savevm_section_end slirp, section_id 1 -> 0
savevm_section_start cpu_common, section_id 4
savevm_section_end cpu_common, section_id 4 -> 0
savevm_section_start cpu, section_id 5
savevm_section_end cpu, section_id 5 -> 0
savevm_section_start kvm-tpr-opt, section_id 6
savevm_section_end kvm-tpr-opt, section_id 6 -> 0
savevm_section_start apic, section_id 7
savevm_section_end apic, section_id 7 -> 0
savevm_section_start cpu_common, section_id 8
savevm_section_end cpu_common, section_id 8 -> 0
savevm_section_start cpu, section_id 9
savevm_section_end cpu, section_id 9 -> 0
savevm_section_start apic, section_id 10
savevm_section_end apic, section_id 10 -> 0
savevm_section_start cpu_common, section_id 11
savevm_section_end cpu_common, section_id 11 -> 0
savevm_section_start cpu, section_id 12
savevm_section_end cpu, section_id 12 -> 0
savevm_section_start apic, section_id 13
savevm_section_end apic, section_id 13 -> 0
savevm_section_start cpu_common, section_id 14
savevm_section_end cpu_common, section_id 14 -> 0
savevm_section_start cpu, section_id 15
savevm_section_end cpu, section_id 15 -> 0
savevm_section_start apic, section_id 16
savevm_section_end apic, section_id 16 -> 0
savevm_section_start kvmclock, section_id 17
savevm_section_end kvmclock, section_id 17 -> 0
savevm_section_start 0000:00:00.0/I440FX, section_id 18
savevm_section_end 0000:00:00.0/I440FX, section_id 18 -> 0
savevm_section_start PCIHost, section_id 19
savevm_section_end PCIHost, section_id 19 -> 0
savevm_section_start PCIBUS, section_id 20
savevm_section_end PCIBUS, section_id 20 -> 0
savevm_section_start fw_cfg, section_id 21
savevm_section_end fw_cfg, section_id 21 -> 0
savevm_section_start dma, section_id 22
savevm_section_end dma, section_id 22 -> 0
savevm_section_start dma, section_id 23
savevm_section_end dma, section_id 23 -> 0
savevm_section_start mc146818rtc, section_id 24
savevm_section_end mc146818rtc, section_id 24 -> 0
savevm_section_start 0000:00:01.1/ide, section_id 25
savevm_section_end 0000:00:01.1/ide, section_id 25 -> 0
savevm_section_start i2c_bus, section_id 26
savevm_section_end i2c_bus, section_id 26 -> 0
savevm_section_start 0000:00:01.3/piix4_pm, section_id 27
savevm_section_end 0000:00:01.3/piix4_pm, section_id 27 -> 0
savevm_section_start 0000:00:01.0/PIIX3, section_id 28
savevm_section_end 0000:00:01.0/PIIX3, section_id 28 -> 0
savevm_section_start i8259, section_id 29
savevm_section_end i8259, section_id 29 -> 0
savevm_section_start i8259, section_id 30
savevm_section_end i8259, section_id 30 -> 0
savevm_section_start ioapic, section_id 31
savevm_section_end ioapic, section_id 31 -> 0
savevm_section_start 0000:00:02.0/vga, section_id 32
savevm_section_end 0000:00:02.0/vga, section_id 32 -> 0
savevm_section_start hpet, section_id 33
savevm_section_end hpet, section_id 33 -> 0
savevm_section_start i8254, section_id 34
savevm_section_end i8254, section_id 34 -> 0
savevm_section_start pcspk, section_id 35
savevm_section_end pcspk, section_id 35 -> 0
savevm_section_start serial, section_id 36
savevm_section_end serial, section_id 36 -> 0
savevm_section_start parallel_isa, section_id 37
savevm_section_end parallel_isa, section_id 37 -> 0
savevm_section_start fdc, section_id 38
savevm_section_end fdc, section_id 38 -> 0
savevm_section_start ps2kbd, section_id 39
savevm_section_end ps2kbd, section_id 39 -> 0
savevm_section_start ps2mouse, section_id 40
savevm_section_end ps2mouse, section_id 40 -> 0
savevm_section_start pckbd, section_id 41
savevm_section_end pckbd, section_id 41 -> 0
savevm_section_start vmmouse, section_id 42
savevm_section_end vmmouse, section_id 42 -> 0
savevm_section_start port92, section_id 43
savevm_section_end port92, section_id 43 -> 0
savevm_section_start 0000:00:03.0/e1000, section_id 44
savevm_section_end 0000:00:03.0/e1000, section_id 44 -> 0
savevm_section_skip smbus-eeprom, section_id 45
savevm_section_skip smbus-eeprom, section_id 46
savevm_section_skip smbus-eeprom, section_id 47
savevm_section_skip smbus-eeprom, section_id 48
savevm_section_skip smbus-eeprom, section_id 49
savevm_section_skip smbus-eeprom, section_id 50
savevm_section_skip smbus-eeprom, section_id 51
savevm_section_skip smbus-eeprom, section_id 52
savevm_section_start acpi_build, section_id 53
savevm_section_end acpi_build, section_id 53 -> 0
savevm_section_start globalstate, section_id 54
migrate_global_state_pre_save saved state: running
savevm_section_end globalstate, section_id 54 -> 0
migrate_error error=Unable to write to socket: Connection reset by peer
migrate_set_state new state failed
migration_thread_after_loop
migration_block_activation active
{"timestamp": {"seconds": 1737127025, "microseconds": 625404}, "event": 
"RESUME"}
migrate_fd_cleanup
savevm_state_cleanup
qemu-system-x86_64: Unable to write to socket: Connection reset by peer


$ /home/fabiano/qemu-system-x86_64 -display none -cpu host -smp 4
-machine pc,accel=kvm -object
memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on -m
4G -machine aux-ram-share=on -incoming tcp:0:44444 -incoming
'{"channel-type": "cpr", "addr": { "transport": "socket", "type":
"unix", "path": "cpr.sock"}}' -trace loadvm_* -trace cpr_* -trace
migration_* -trace migrate_* -monitor stdio

cpr_transfer_input cpr.sock
cpr_state_load cpr-transfer mode
QEMU 9.2.50 monitor - type 'help' for more information
cpr_find_fd pc.bios, id 0 returns 15
cpr_find_fd pc.rom, id 0 returns 14
cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 13
cpr_find_fd 0000:00:02.0/vga.rom, id 0 returns 12
cpr_find_fd 0000:00:03.0/e1000.rom, id 0 returns 11
cpr_find_fd /rom@etc/acpi/tables, id 0 returns 10
cpr_find_fd /rom@etc/table-loader, id 0 returns 8
cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 3
migrate_set_state new state setup
(qemu) migration_socket_incoming_accepted
migration_set_incoming_channel ioc=0x5565cccb8e70 ioctype=qio-channel-socket
migrate_set_state new state active
loadvm_state_setup
qemu-system-x86_64: error while loading state for instance 0x0 of device 
'kvm-tpr-opt'
migrate_set_state new state failed
migrate_error error=load of migration failed: Operation not permitted
loadvm_state_cleanup
qemu-system-x86_64: load of migration failed: Operation not permitted