Re: [PULL 00/28] Migration pull patches

[Daniel P . Berrangé]

On Mon, Jan 13, 2020 at 01:05:22PM +0000, Peter Maydell wrote:
> On Fri, 10 Jan 2020 at 17:32, Juan Quintela <address@hidden> wrote:
> >
> > The following changes since commit f38a71b01f839c7b65ea73ddd507903cb9489ed6:
> >
> >   Merge remote-tracking branch 
> > 'remotes/stsquad/tags/pull-testing-and-semihosting-090120-2' into staging 
> > (2020-01-10 13:19:34 +0000)
> >
> > are available in the Git repository at:
> >
> >   https://github.com/juanquintela/qemu.git tags/migration-pull-pull-request
> >
> > for you to fetch changes up to cc708d2411d3ed2ab4a428c996b778c7c7a47a04:
> >
> >   apic: Use 32bit APIC ID for migration instance ID (2020-01-10 18:19:18 
> > +0100)
> >

[snip]

> I also saw this on aarch32 host (more precisely, on the
> aarch32-environment-in-aarch64-chroot setup I use for aarch32 build
> and test):
> 
> malloc_consolidate(): invalid chunk size
> Broken pipe
> qemu-system-i386: check_section_footer: Read section footer failed: -5
> qemu-system-i386: load of migration failed: Invalid argument
> /home/peter.maydell/qemu/tests/libqtest.c:140: kill_qemu() tried to
> terminate QEMU process but encountered exit status 1 (expected 0)
> Aborted
> ERROR - too few tests run (expected 14, got 13)
> 
> The memory corruption is reproducible running just the
> /x86_64/migration/multifd/tcp subtest:
> 
> (armhf)pmaydell@mustang-maydell:~/qemu/build/all-a32$
> QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64
> tests/migration-test -p /x86_64/migration/multifd/tcp
> /x86_64/migration/multifd/tcp: qemu-system-x86_64: -accel kvm: invalid
> accelerator kvm
> qemu-system-x86_64: falling back to tcg
> qemu-system-x86_64: -accel kvm: invalid accelerator kvm
> qemu-system-x86_64: falling back to tcg
> qemu-system-x86_64: multifd_send_sync_main: multifd_send_pages fail
> qemu-system-x86_64: failed to save SaveStateEntry with id(name): 3(ram)
> double free or corruption (!prev)
> Broken pipe
> qemu-system-x86_64: Unknown combination of migration flags: 0
> qemu-system-x86_64: error while loading state section id 3(ram)
> qemu-system-x86_64: load of migration failed: Invalid argument
> /home/peter.maydell/qemu/tests/libqtest.c:140: kill_qemu() tried to
> terminate QEMU process but encountered exit status 1 (expected 0)
> Aborted
> 
> Here's what a valgrind run in that aarch32 setup produces:
> 
> (armhf)pmaydell@mustang-maydell:~/qemu/build/all-a32$
> QTEST_QEMU_BINARY='valgrind --smc-check=all-non-file
> x86_64-softmmu/qemu-system-x86_64' tests/migration-test -p
> /x86_64/migration/multifd/tcp
> /x86_64/migration/multifd/tcp: ==12102== Memcheck, a memory error detector
> ==12102== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> ==12102== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
> ==12102== Command: x86_64-softmmu/qemu-system-x86_64 -qtest
> unix:/tmp/qtest-12100.sock -qtest-log /dev/null -chardev
> socket,path=/tmp/qtest-12100.qmp,id=char0 -mon
> chardev=char0,mode=control -display none -accel kvm -accel tcg -name
> source,debug-threads=on -m 150M -serial
> file:/tmp/migration-test-UlotFX/src_serial -drive
> file=/tmp/migration-test-UlotFX/bootsect,format=raw -accel qtest
> ==12102==
> qemu-system-x86_64: -accel kvm: invalid accelerator kvm
> qemu-system-x86_64: falling back to tcg
> ==12108== Memcheck, a memory error detector
> ==12108== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> ==12108== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
> ==12108== Command: x86_64-softmmu/qemu-system-x86_64 -qtest
> unix:/tmp/qtest-12100.sock -qtest-log /dev/null -chardev
> socket,path=/tmp/qtest-12100.qmp,id=char0 -mon
> chardev=char0,mode=control -display none -accel kvm -accel tcg -name
> target,debug-threads=on -m 150M -serial
> file:/tmp/migration-test-UlotFX/dest_serial -incoming defer -drive
> file=/tmp/migration-test-UlotFX/bootsect,format=raw -accel qtest
> ==12108==
> qemu-system-x86_64: -accel kvm: invalid accelerator kvm
> qemu-system-x86_64: falling back to tcg
> ==12102== Thread 22 multifdsend_15:
> ==12102== Syscall param sendmsg(msg.msg_iov[0]) points to uninitialised 
> byte(s)
> ==12102==    at 0x53C7F06: __libc_do_syscall (libc-do-syscall.S:47)
> ==12102==    by 0x53C6FCB: sendmsg (sendmsg.c:28)
> ==12102==    by 0x51B9A9: qio_channel_socket_writev (channel-socket.c:561)
> ==12102==    by 0x519FCD: qio_channel_writev (channel.c:207)
> ==12102==    by 0x519FCD: qio_channel_writev_all (channel.c:171)
> ==12102==    by 0x51A047: qio_channel_write_all (channel.c:257)
> ==12102==    by 0x25CB17: multifd_send_initial_packet (ram.c:714)
> ==12102==    by 0x25CB17: multifd_send_thread (ram.c:1136)
> ==12102==    by 0x557551: qemu_thread_start (qemu-thread-posix.c:519)
> ==12102==    by 0x53BE613: start_thread (pthread_create.c:463)
> ==12102==    by 0x54767FB: ??? (clone.S:73)
> ==12102==  Address 0x262103fd is on thread 22's stack
> ==12102==  in frame #5, created by multifd_send_thread (ram.c:1127)

Missing initialization of     MultiFDInit_t msg; to all zeros

> ==12102==
> ==12102== Thread 6 multifdsend_1:
> ==12102== Invalid write of size 4
> ==12102==    at 0x25CC08: multifd_send_fill_packet (ram.c:806)
> ==12102==    by 0x25CC08: multifd_send_thread (ram.c:1157)
> ==12102==    by 0x557551: qemu_thread_start (qemu-thread-posix.c:519)
> ==12102==    by 0x53BE613: start_thread (pthread_create.c:463)
> ==12102==    by 0x54767FB: ??? (clone.S:73)
> ==12102==  Address 0x1d89c470 is 0 bytes after a block of size 832 alloc'd
> ==12102==    at 0x4841BC4: calloc (vg_replace_malloc.c:711)
> ==12102==    by 0x49EE269: g_malloc0 (in
> /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0.5600.4)

This is the same issue that was reported last time this mulitfd unit
test was proposed for merge. Back then I pointed out the likely cause.
We were allocating  ram_addr_t sized quantity for an array which is
uint64_t, and ram_addr_t is probably 32-bit on this particular build.

  https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg03428.html

That suggested fix doesn't seem to have been included


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|