Re: [PULL 00/17] Migration patches for 2024-12-17

[Fabiano Rosas]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> On Fri, 3 Jan 2025 at 13:32, Fabiano Rosas <farosas@suse.de> wrote:
>>
>> Thomas Huth <thuth@redhat.com> writes:
>>
>> > On 20/12/2024 17.28, Peter Xu wrote:
>> >> On Thu, Dec 19, 2024 at 03:53:22PM -0300, Fabiano Rosas wrote:
>> >>> Stefan Hajnoczi <stefanha@redhat.com> writes:
>> >>>
>> >>>> Hi Fabiano,
>> >>>> Please take a look at this CI failure:
>> >>>>
>> >>>>>>> MALLOC_PERTURB_=61 QTEST_QEMU_BINARY=./qemu-system-s390x 
>> >>>>>>> UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1
>> >>>>>>>  QTEST_QEMU_IMG=./qemu-img MESON_TEST_ITERATION=1 
>> >>>>>>> MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1
>> >>>>>>>  ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 
>> >>>>>>> PYTHON=/home/gitlab-runner/builds/4S3awx_3/0/qemu-project/qemu/build/pyvenv/bin/python3
>> >>>>>>>  
>> >>>>>>> QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
>> >>>>>>>  
>> >>>>>>> G_TEST_DBUS_DAEMON=/home/gitlab-runner/builds/4S3awx_3/0/qemu-project/qemu/tests/dbus-vmstate-daemon.sh
>> >>>>>>>  
>> >>>>>>> /home/gitlab-runner/builds/4S3awx_3/0/qemu-project/qemu/build/tests/qtest/migration-test
>> >>>>>>>  --tap -k
>> >>>> ――――――――――――――――――――――――――――――――――――― ✀  
>> >>>> ―――――――――――――――――――――――――――――――――――――
>> >>>> stderr:
>> >>>> Traceback (most recent call last):
>> >>>>    File 
>> >>>> "/home/gitlab-runner/builds/4S3awx_3/0/qemu-project/qemu/build/scripts/analyze-migration.py",
>> >>>>  line 688, in <module>
>> >>>>      dump.read(dump_memory = args.memory)
>> >>>>    File 
>> >>>> "/home/gitlab-runner/builds/4S3awx_3/0/qemu-project/qemu/build/scripts/analyze-migration.py",
>> >>>>  line 625, in read
>> >>>>      section.read()
>> >>>>    File 
>> >>>> "/home/gitlab-runner/builds/4S3awx_3/0/qemu-project/qemu/build/scripts/analyze-migration.py",
>> >>>>  line 461, in read
>> >>>>      field['data'] = reader(field, self.file)
>> >>>>    File 
>> >>>> "/home/gitlab-runner/builds/4S3awx_3/0/qemu-project/qemu/build/scripts/analyze-migration.py",
>> >>>>  line 434, in __init__
>> >>>>      for field in self.desc['struct']['fields']:
>> >>>> KeyError: 'fields'
>> >>>
>> >>> This is the command line that runs only this specific test:
>> >>>
>> >>> PYTHON=/usr/bin/python3.11 QTEST_QEMU_BINARY=./qemu-system-s390x
>> >>> ./tests/qtest/migration-test -p /s390x/migration/analyze-script
>> >>>
>> >>> I cannot reproduce in migration-next nor in the detached HEAD that the
>> >>> pipeline ran in (had to download the tarball from gitlab).
>> >>>
>> >>> The only s390 patch in this PR is one that I can test just fine with
>> >>> TCG, so there shouldn't be any difference from KVM (i.e. there should be
>> >>> no state being migrated with KVM that is not already migrated with TCG).
>> >>>
>> >>>> warning: fd: migration to a file is deprecated. Use file: instead.
>> >>>> warning: fd: migration to a file is deprecated. Use file: instead.
>> >>>
>> >>> This is harmless.
>> >>>
>> >>>> **
>> >>>> ERROR:../tests/qtest/migration-test.c:36:main: assertion failed (ret == 
>> >>>> 0): (1 == 0)
>> >>>> (test program exited with status code -6)
>> >>>
>> >>> This is the assert at the end of the tests, irrelevant.
>> >>>
>> >>>>
>> >>>> https://gitlab.com/qemu-project/qemu/-/jobs/8681858344#L8190
>> >>>>
>> >>>> If you find this pull request caused the failure, please send a new
>> >>>> revision. Otherwise please let me know so we can continue to
>> >>>> investigate.
>> >>>
>> >>> I don't have an s390x host at hand so the only thing I can to is to drop
>> >>> that patch and hope that resolves the problem. @Peter, @Thomas, any
>> >>> other ideas? Can you verify this on your end?
>> >>
>> >> Cannot reproduce either here, x86_64 host only.  The report was from s390
>> >> host, though.  I'm not familiar with the s390 patch, I wonder if any of 
>> >> you
>> >> could use plain brain power to figure more things out.
>> >>
>> >> We could wait for 1-2 more days to see whether Thomas can figure it out,
>> >> hopefully easily reproduceable on s390.. or we can also leave that for
>> >> later.  And if the current issue on such fix is s390-host-only, might be
>> >> easier to be picked up by s390 tree, perhaps?
>> >
>> > I tested migration-20241217-pull-request on a s390x (RHEL) host, but I
>> > cannot reproduce the issue there - make check-qtest works without any
>> > problems. Is it maybe related to that specific Ubuntu installation?
>> >
>>
>> Since we cannot reproduce outside of the staging CI, could we run that
>> job again with a diagnostic patch? Here's the rebased PR with the patch:
>>
>> https://gitlab.com/farosas/qemu/-/commits/migration-next
>>
>> (fork CI run: https://gitlab.com/farosas/qemu/-/pipelines/1610691202)
>>
>> Or should I just send a v2 of this PR with the debug patch?
>
> Here is the staging CI pipeline for your migration-next tree:
> https://gitlab.com/qemu-project/qemu/-/pipelines/1610836485

Great, thanks! Let's find out what is going on...

>
> Stefan