Re: [PULL 00/28] Migration pull patches

[Peter Maydell]

On Fri, 10 Jan 2020 at 17:32, Juan Quintela <address@hidden> wrote:
>
> The following changes since commit f38a71b01f839c7b65ea73ddd507903cb9489ed6:
>
>   Merge remote-tracking branch 
> 'remotes/stsquad/tags/pull-testing-and-semihosting-090120-2' into staging 
> (2020-01-10 13:19:34 +0000)
>
> are available in the Git repository at:
>
>   https://github.com/juanquintela/qemu.git tags/migration-pull-pull-request
>
> for you to fetch changes up to cc708d2411d3ed2ab4a428c996b778c7c7a47a04:
>
>   apic: Use 32bit APIC ID for migration instance ID (2020-01-10 18:19:18 
> +0100)
>
> ----------------------------------------------------------------
> Migration pull request
>
> - several multifd mixes (jiahui, me)
> - rate limit host pages (david)
> - remove unneeded labels (daniel)
> - several multifd fixes (wei)
> - improve handler insert (scott)
> - qlist migration (eric)
> - power fixes (laurent)
> - migration improvemests (yury)
> - lots of fixes (wei)

Hi. This causes a new compile warning for the netbsd VM:

In file included from
/home/qemu/qemu-test.tqjNTZ/src/include/hw/qdev-core.h:4:0,
                 from
/home/qemu/qemu-test.tqjNTZ/src/tests/../migration/migration.h:18,
                 from /home/qemu/qemu-test.tqjNTZ/src/tests/test-vmstate.c:27:
/home/qemu/qemu-test.tqjNTZ/src/tests/test-vmstate.c: In function
'manipulate_container':
/home/qemu/qemu-test.tqjNTZ/src/include/qemu/queue.h:130:34: warning:
'prev' may be used uninitialized in this function
[-Wmaybe-uninitialized]
         (listelm)->field.le_prev = &(elm)->field.le_next;               \
                                  ^
/home/qemu/qemu-test.tqjNTZ/src/tests/test-vmstate.c:1337:24: note:
'prev' was declared here
      TestQListElement *prev, *iter = QLIST_FIRST(&c->list);
                        ^


I also saw this on aarch32 host (more precisely, on the
aarch32-environment-in-aarch64-chroot setup I use for aarch32 build
and test):

malloc_consolidate(): invalid chunk size
Broken pipe
qemu-system-i386: check_section_footer: Read section footer failed: -5
qemu-system-i386: load of migration failed: Invalid argument
/home/peter.maydell/qemu/tests/libqtest.c:140: kill_qemu() tried to
terminate QEMU process but encountered exit status 1 (expected 0)
Aborted
ERROR - too few tests run (expected 14, got 13)

The memory corruption is reproducible running just the
/x86_64/migration/multifd/tcp subtest:

(armhf)pmaydell@mustang-maydell:~/qemu/build/all-a32$
QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64
tests/migration-test -p /x86_64/migration/multifd/tcp
/x86_64/migration/multifd/tcp: qemu-system-x86_64: -accel kvm: invalid
accelerator kvm
qemu-system-x86_64: falling back to tcg
qemu-system-x86_64: -accel kvm: invalid accelerator kvm
qemu-system-x86_64: falling back to tcg
qemu-system-x86_64: multifd_send_sync_main: multifd_send_pages fail
qemu-system-x86_64: failed to save SaveStateEntry with id(name): 3(ram)
double free or corruption (!prev)
Broken pipe
qemu-system-x86_64: Unknown combination of migration flags: 0
qemu-system-x86_64: error while loading state section id 3(ram)
qemu-system-x86_64: load of migration failed: Invalid argument
/home/peter.maydell/qemu/tests/libqtest.c:140: kill_qemu() tried to
terminate QEMU process but encountered exit status 1 (expected 0)
Aborted

Here's what a valgrind run in that aarch32 setup produces:

(armhf)pmaydell@mustang-maydell:~/qemu/build/all-a32$
QTEST_QEMU_BINARY='valgrind --smc-check=all-non-file
x86_64-softmmu/qemu-system-x86_64' tests/migration-test -p
/x86_64/migration/multifd/tcp
/x86_64/migration/multifd/tcp: ==12102== Memcheck, a memory error detector
==12102== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==12102== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==12102== Command: x86_64-softmmu/qemu-system-x86_64 -qtest
unix:/tmp/qtest-12100.sock -qtest-log /dev/null -chardev
socket,path=/tmp/qtest-12100.qmp,id=char0 -mon
chardev=char0,mode=control -display none -accel kvm -accel tcg -name
source,debug-threads=on -m 150M -serial
file:/tmp/migration-test-UlotFX/src_serial -drive
file=/tmp/migration-test-UlotFX/bootsect,format=raw -accel qtest
==12102==
qemu-system-x86_64: -accel kvm: invalid accelerator kvm
qemu-system-x86_64: falling back to tcg
==12108== Memcheck, a memory error detector
==12108== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==12108== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==12108== Command: x86_64-softmmu/qemu-system-x86_64 -qtest
unix:/tmp/qtest-12100.sock -qtest-log /dev/null -chardev
socket,path=/tmp/qtest-12100.qmp,id=char0 -mon
chardev=char0,mode=control -display none -accel kvm -accel tcg -name
target,debug-threads=on -m 150M -serial
file:/tmp/migration-test-UlotFX/dest_serial -incoming defer -drive
file=/tmp/migration-test-UlotFX/bootsect,format=raw -accel qtest
==12108==
qemu-system-x86_64: -accel kvm: invalid accelerator kvm
qemu-system-x86_64: falling back to tcg
==12102== Thread 22 multifdsend_15:
==12102== Syscall param sendmsg(msg.msg_iov[0]) points to uninitialised byte(s)
==12102==    at 0x53C7F06: __libc_do_syscall (libc-do-syscall.S:47)
==12102==    by 0x53C6FCB: sendmsg (sendmsg.c:28)
==12102==    by 0x51B9A9: qio_channel_socket_writev (channel-socket.c:561)
==12102==    by 0x519FCD: qio_channel_writev (channel.c:207)
==12102==    by 0x519FCD: qio_channel_writev_all (channel.c:171)
==12102==    by 0x51A047: qio_channel_write_all (channel.c:257)
==12102==    by 0x25CB17: multifd_send_initial_packet (ram.c:714)
==12102==    by 0x25CB17: multifd_send_thread (ram.c:1136)
==12102==    by 0x557551: qemu_thread_start (qemu-thread-posix.c:519)
==12102==    by 0x53BE613: start_thread (pthread_create.c:463)
==12102==    by 0x54767FB: ??? (clone.S:73)
==12102==  Address 0x262103fd is on thread 22's stack
==12102==  in frame #5, created by multifd_send_thread (ram.c:1127)
==12102==
==12102== Thread 6 multifdsend_1:
==12102== Invalid write of size 4
==12102==    at 0x25CC08: multifd_send_fill_packet (ram.c:806)
==12102==    by 0x25CC08: multifd_send_thread (ram.c:1157)
==12102==    by 0x557551: qemu_thread_start (qemu-thread-posix.c:519)
==12102==    by 0x53BE613: start_thread (pthread_create.c:463)
==12102==    by 0x54767FB: ??? (clone.S:73)
==12102==  Address 0x1d89c470 is 0 bytes after a block of size 832 alloc'd
==12102==    at 0x4841BC4: calloc (vg_replace_malloc.c:711)
==12102==    by 0x49EE269: g_malloc0 (in
/usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0.5600.4)
==12102==
==12102== Invalid write of size 4
==12102==    at 0x25CC0E: multifd_send_fill_packet (ram.c:806)
==12102==    by 0x25CC0E: multifd_send_thread (ram.c:1157)
==12102==    by 0x557551: qemu_thread_start (qemu-thread-posix.c:519)
==12102==    by 0x53BE613: start_thread (pthread_create.c:463)
==12102==    by 0x54767FB: ??? (clone.S:73)
==12102==  Address 0x1d89c474 is 4 bytes after a block of size 832 alloc'd
==12102==    at 0x4841BC4: calloc (vg_replace_malloc.c:711)
==12102==    by 0x49EE269: g_malloc0 (in
/usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0.5600.4)
==12102==
==12102== Invalid read of size 4
==12102==    at 0x519812: qio_channel_writev_full (channel.c:86)
==12102==    by 0x519FCD: qio_channel_writev (channel.c:207)
==12102==    by 0x519FCD: qio_channel_writev_all (channel.c:171)
==12102==    by 0x51A047: qio_channel_write_all (channel.c:257)
==12102==    by 0x25CC6D: multifd_send_thread (ram.c:1168)
==12102==    by 0x557551: qemu_thread_start (qemu-thread-posix.c:519)
==12102==    by 0x53BE613: start_thread (pthread_create.c:463)
==12102==    by 0x54767FB: ??? (clone.S:73)
==12102==  Address 0x30 is not stack'd, malloc'd or (recently) free'd
==12102==
==12102==
==12102== Process terminating with default action of signal 11 (SIGSEGV)
==12102==  Access not within mapped region at address 0x30
==12102==    at 0x519812: qio_channel_writev_full (channel.c:86)
==12102==    by 0x519FCD: qio_channel_writev (channel.c:207)
==12102==    by 0x519FCD: qio_channel_writev_all (channel.c:171)
==12102==    by 0x51A047: qio_channel_write_all (channel.c:257)
==12102==    by 0x25CC6D: multifd_send_thread (ram.c:1168)
==12102==    by 0x557551: qemu_thread_start (qemu-thread-posix.c:519)
==12102==    by 0x53BE613: start_thread (pthread_create.c:463)
==12102==    by 0x54767FB: ??? (clone.S:73)
==12102==  If you believe this happened as a result of a stack
==12102==  overflow in your program's main thread (unlikely but
==12102==  possible), you can try to increase the size of the
==12102==  main thread stack using the --main-stacksize= flag.
==12102==  The main thread stack size used in this run was 8388608.
==12102==
==12102== HEAP SUMMARY:
==12102==     in use at exit: 7,159,914 bytes in 28,035 blocks
==12102==   total heap usage: 370,889 allocs, 342,854 frees,
34,875,720 bytes allocated
==12102==
==12102== LEAK SUMMARY:
==12102==    definitely lost: 56 bytes in 1 blocks
==12102==    indirectly lost: 64 bytes in 2 blocks
==12102==      possibly lost: 5,916 bytes in 58 blocks
==12102==    still reachable: 7,153,878 bytes in 27,974 blocks
==12102==                       of which reachable via heuristic:
==12102==                         newarray           : 832 bytes in 16 blocks
==12102==         suppressed: 0 bytes in 0 blocks
==12102== Rerun with --leak-check=full to see details of leaked memory
==12102==
==12102== For counts of detected and suppressed errors, rerun with: -v
==12102== Use --track-origins=yes to see where uninitialised values come from
==12102== ERROR SUMMARY: 80 errors from 4 contexts (suppressed: 6 from 3)
Broken pipe
qemu-system-x86_64: load of migration failed: Input/output error
==12108==
==12108== HEAP SUMMARY:
==12108==     in use at exit: 6,321,388 bytes in 21,290 blocks
==12108==   total heap usage: 59,082 allocs, 37,792 frees, 23,874,965
bytes allocated
==12108==
==12108== LEAK SUMMARY:
==12108==    definitely lost: 0 bytes in 0 blocks
==12108==    indirectly lost: 0 bytes in 0 blocks
==12108==      possibly lost: 5,440 bytes in 37 blocks
==12108==    still reachable: 6,315,948 bytes in 21,253 blocks
==12108==                       of which reachable via heuristic:
==12108==                         newarray           : 832 bytes in 16 blocks
==12108==         suppressed: 0 bytes in 0 blocks
==12108== Rerun with --leak-check=full to see details of leaked memory
==12108==
==12108== For counts of detected and suppressed errors, rerun with: -v
==12108== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 6 from 3)
/home/peter.maydell/qemu/tests/libqtest.c:140: kill_qemu() tried to
terminate QEMU process but encountered exit status 1 (expected 0)
Aborted


thanks
-- PMM