[Qemu-commits] [qemu/qemu] cc5ab8: migration: Unify reset of last_rb on destination n...

[Peter Maydell]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: cc5ab87200257199eba91aba9baf141ae0e91d0c
      
https://github.com/qemu/qemu/commit/cc5ab87200257199eba91aba9baf141ae0e91d0c
  Author: Peter Xu <peterx@redhat.com>
  Date:   2020-11-02 (Mon, 02 Nov 2020)

  Changed paths:
    M migration/postcopy-ram.c
    M migration/savevm.c

  Log Message:
  -----------
  migration: Unify reset of last_rb on destination node when recover

When postcopy recover happens, we need to reset last_rb after each return of
postcopy_pause_fault_thread() because that means we just got the postcopy
migration continued.

Unify this reset to the place right before we want to kick the fault thread
again, when we get the command MIG_CMD_POSTCOPY_RESUME from source.

This is actually more than that - because the main thread on destination will
now be able to call migrate_send_rp_req_pages_pending() too, so the fault
thread is not the only user of last_rb now.  Move the reset earlier will allow
the first call to migrate_send_rp_req_pages_pending() to use the reset value
even if called from the main thread.

(NOTE: this is not a real fix to 0c26781c09 mentioned below, however it is just
 a mark that when picking up 0c26781c09 we'd better have this one too; the real
 fix will come later)

Fixes: 0c26781c09 ("migration: Sync requested pages after postcopy recovery")
Tested-by: Christian Schoenebeck <qemu_oss@crudebyte.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Message-Id: <20201102153010.11979-2-peterx@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: 5e773431134a6c3f65c0088d578384577c4a3f2c
      
https://github.com/qemu/qemu/commit/5e773431134a6c3f65c0088d578384577c4a3f2c
  Author: Peter Xu <peterx@redhat.com>
  Date:   2020-11-02 (Mon, 02 Nov 2020)

  Changed paths:
    M migration/savevm.c

  Log Message:
  -----------
  migration: Postpone the kick of the fault thread after recover

The new migrate_send_rp_req_pages_pending() call should greatly improve
destination responsiveness because it will resync faulted address after
postcopy recovery.  However it is also the 1st place to initiate the page
request from the main thread.

One thing is overlooked on that migrate_send_rp_message_req_pages() is not
designed to be thread-safe.  So if we wake the fault thread before syncing all
the faulted pages in the main thread, it means they can race.

Postpone the wake up operation after the sync of faulted addresses.

Fixes: 0c26781c09 ("migration: Sync requested pages after postcopy recovery")
Tested-by: Christian Schoenebeck <qemu_oss@crudebyte.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20201102153010.11979-3-peterx@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: dcaac9f124833e62e720d0733aa53453f438b7d7
      
https://github.com/qemu/qemu/commit/dcaac9f124833e62e720d0733aa53453f438b7d7
  Author: Dr. David Alan Gilbert <dgilbert@redhat.com>
  Date:   2020-11-02 (Mon, 02 Nov 2020)

  Changed paths:
    M tools/virtiofsd/passthrough_seccomp.c

  Log Message:
  -----------
  virtiofsd: Seccomp: Add 'send' for syslog

On ppc, and some other archs, it looks like syslog ends up using 'send'
rather than 'sendto'.

Reference: https://github.com/kata-containers/kata-containers/issues/1050

Reported-by: amulmek1@in.ibm.com
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Message-Id: <20201102150750.34565-1-dgilbert@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: 2693026042d2f6f5749babb8827b8098ed5d7eb4
      
https://github.com/qemu/qemu/commit/2693026042d2f6f5749babb8827b8098ed5d7eb4
  Author: Philippe Mathieu-Daudé <philmd@redhat.com>
  Date:   2020-11-02 (Mon, 02 Nov 2020)

  Changed paths:
    M tools/virtiofsd/fuse_virtio.c

  Log Message:
  -----------
  tools/virtiofsd: Check vu_init() return value (CID 1435958)

Since commit 6f5fd837889, vu_init() can fail if malloc() returns NULL.

This fixes the following Coverity warning:

  CID 1435958 (#1 of 1): Unchecked return value (CHECKED_RETURN)

Fixes: 6f5fd837889 ("libvhost-user: support many virtqueues")
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201102092339.2034297-1-philmd@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: 0429eaf518be1d4742356056e6c886b7f9bc9712
      
https://github.com/qemu/qemu/commit/0429eaf518be1d4742356056e6c886b7f9bc9712
  Author: Jiachen Zhang <zhangjiachen.jaycee@bytedance.com>
  Date:   2020-11-02 (Mon, 02 Nov 2020)

  Changed paths:
    M tools/virtiofsd/helper.c

  Log Message:
  -----------
  virtiofsd: Fix the help message of posix lock

The commit 88fc107956a5812649e5918e0c092d3f78bb28ad disabled remote
posix locks by default. But the --help message still says it is enabled
by default. So fix it to output no_posix_lock.

Signed-off-by: Jiachen Zhang <zhangjiachen.jaycee@bytedance.com>
Message-Id: <20201027081558.29904-1-zhangjiachen.jaycee@bytedance.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: 9c6ac0436356132da97c4bbb7f6c90b062438176
      
https://github.com/qemu/qemu/commit/9c6ac0436356132da97c4bbb7f6c90b062438176
  Author: Max Reitz <mreitz@redhat.com>
  Date:   2020-11-02 (Mon, 02 Nov 2020)

  Changed paths:
    M tools/virtiofsd/fuse_common.h
    M tools/virtiofsd/fuse_lowlevel.c

  Log Message:
  -----------
  virtiofsd: Check FUSE_SUBMOUNTS

FUSE_SUBMOUNTS is a pure indicator by the kernel to signal that it
supports submounts.  It does not check its state in the init reply, so
there is nothing for fuse_lowlevel.c to do but to check its existence
and copy it into fuse_conn_info.capable.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20201102161859.156603-2-mreitz@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: 93e79851abb78fe36882232b84edc795d1d942fd
      
https://github.com/qemu/qemu/commit/93e79851abb78fe36882232b84edc795d1d942fd
  Author: Max Reitz <mreitz@redhat.com>
  Date:   2020-11-02 (Mon, 02 Nov 2020)

  Changed paths:
    M tools/virtiofsd/fuse_lowlevel.c
    M tools/virtiofsd/fuse_lowlevel.h

  Log Message:
  -----------
  virtiofsd: Add attr_flags to fuse_entry_param

fuse_entry_param is converted to fuse_attr on the line (by
fill_entry()), so it should have a member that mirrors fuse_attr.flags.

fill_entry() should then copy this fuse_entry_param.attr_flags to
fuse_attr.flags.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20201102161859.156603-3-mreitz@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: 84e319a575d04e9d53c24d801572a3ead538f9e4
      
https://github.com/qemu/qemu/commit/84e319a575d04e9d53c24d801572a3ead538f9e4
  Author: Max Reitz <mreitz@redhat.com>
  Date:   2020-11-02 (Mon, 02 Nov 2020)

  Changed paths:
    M meson.build

  Log Message:
  -----------
  meson.build: Check for statx()

Check whether the glibc provides statx() and if so, define CONFIG_STATX.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20201102161859.156603-4-mreitz@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: d672fce6baa0a02dfa64d717fb072c06e301d695
      
https://github.com/qemu/qemu/commit/d672fce6baa0a02dfa64d717fb072c06e301d695
  Author: Max Reitz <mreitz@redhat.com>
  Date:   2020-11-02 (Mon, 02 Nov 2020)

  Changed paths:
    M tools/virtiofsd/passthrough_ll.c
    M tools/virtiofsd/passthrough_seccomp.c

  Log Message:
  -----------
  virtiofsd: Add mount ID to the lo_inode key

Using st_dev is not sufficient to uniquely identify a mount: You can
mount the same device twice, but those are still separate trees, and
e.g. by mounting something else inside one of them, they may differ.

Using statx(), we can get a mount ID that uniquely identifies a mount.
If that is available, add it to the lo_inode key.

Most of this patch is taken from Miklos's mail here:
https://marc.info/?l=fuse-devel&m=160062521827983
(virtiofsd-use-mount-id.patch attachment)

Suggested-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20201102161859.156603-5-mreitz@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: 9d82f6a3e68c2986633ad11687da2193e9acb0cb
      
https://github.com/qemu/qemu/commit/9d82f6a3e68c2986633ad11687da2193e9acb0cb
  Author: Max Reitz <mreitz@redhat.com>
  Date:   2020-11-02 (Mon, 02 Nov 2020)

  Changed paths:
    M tools/virtiofsd/helper.c
    M tools/virtiofsd/passthrough_ll.c

  Log Message:
  -----------
  virtiofsd: Announce sub-mount points

Whenever we encounter a directory with an st_dev or mount ID that
differs from that of its parent, we set the FUSE_ATTR_SUBMOUNT flag so
the guest can create a submount for it.

We only need to do so in lo_do_lookup().  The following functions return
a fuse_attr object:
- lo_create(), though fuse_reply_create(): Calls lo_do_lookup().
- lo_lookup(), though fuse_reply_entry(): Calls lo_do_lookup().
- lo_mknod_symlink(), through fuse_reply_entry(): Calls lo_do_lookup().
- lo_link(), through fuse_reply_entry(): Creating a link cannot create a
  submount, so there is no need to check for it.
- lo_getattr(), through fuse_reply_attr(): Announcing submounts when the
  node is first detected (at lookup) is sufficient.  We do not need to
  return the submount attribute later.
- lo_do_readdir(), through fuse_add_direntry_plus(): Calls
  lo_do_lookup().

Make announcing submounts optional, so submounts are only announced to
the guest with the announce_submounts option.  Some users may prefer the
current behavior, so that the guest learns nothing about the host mount
structure.

(announce_submounts is force-disabled when the guest does not present
the FUSE_SUBMOUNTS capability, or when there is no statx().)

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20201102161859.156603-6-mreitz@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: 0a7798037acb65cf4e9d64372690a47018e60d02
      
https://github.com/qemu/qemu/commit/0a7798037acb65cf4e9d64372690a47018e60d02
  Author: Max Reitz <mreitz@redhat.com>
  Date:   2020-11-02 (Mon, 02 Nov 2020)

  Changed paths:
    M tests/acceptance/boot_linux.py

  Log Message:
  -----------
  tests/acceptance/boot_linux: Accept SSH pubkey

Let download_cloudinit() take an optional pubkey, which subclasses of
BootLinux can pass through setUp().

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20201102161859.156603-7-mreitz@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: af1bb3fe7f146fafdaadb479975ca2b53b49df40
      
https://github.com/qemu/qemu/commit/af1bb3fe7f146fafdaadb479975ca2b53b49df40
  Author: Max Reitz <mreitz@redhat.com>
  Date:   2020-11-02 (Mon, 02 Nov 2020)

  Changed paths:
    A tests/acceptance/virtiofs_submounts.py
    A tests/acceptance/virtiofs_submounts.py.data/cleanup.sh
    A tests/acceptance/virtiofs_submounts.py.data/guest-cleanup.sh
    A tests/acceptance/virtiofs_submounts.py.data/guest.sh
    A tests/acceptance/virtiofs_submounts.py.data/host.sh

  Log Message:
  -----------
  tests/acceptance: Add virtiofs_submounts.py

This test invokes several shell scripts to create a random directory
tree full of submounts, and then check in the VM whether every submount
has its own ID and the structure looks as expected.

(Note that the test scripts must be non-executable, so Avocado will not
try to execute them as if they were tests on their own, too.)

Because at this commit's date it is unlikely that the Linux kernel on
the image provided by boot_linux.py supports submounts in virtio-fs, the
test will be cancelled if no custom Linux binary is provided through the
vmlinuz parameter.  (The on-image kernel can be used by providing an
empty string via vmlinuz=.)

So, invoking the test can be done as follows:
$ avocado run \
    tests/acceptance/virtiofs_submounts.py \
    -p vmlinuz=/path/to/linux/build/arch/x86/boot/bzImage

This test requires root privileges (through passwordless sudo -n),
because at this point, virtiofsd requires them.  (If you have a
timestamp_timeout period for sudoers (e.g. the default of 5 min), you
can provide this by executing something like "sudo true" before invoking
Avocado.)

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20201102161859.156603-8-mreitz@redhat.com>
Tested-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: 8545ae485b1e8e43cc0137310c4c68dbece59990
      
https://github.com/qemu/qemu/commit/8545ae485b1e8e43cc0137310c4c68dbece59990
  Author: Peter Maydell <peter.maydell@linaro.org>
  Date:   2020-11-02 (Mon, 02 Nov 2020)

  Changed paths:
    M meson.build
    M migration/postcopy-ram.c
    M migration/savevm.c
    M tests/acceptance/boot_linux.py
    A tests/acceptance/virtiofs_submounts.py
    A tests/acceptance/virtiofs_submounts.py.data/cleanup.sh
    A tests/acceptance/virtiofs_submounts.py.data/guest-cleanup.sh
    A tests/acceptance/virtiofs_submounts.py.data/guest.sh
    A tests/acceptance/virtiofs_submounts.py.data/host.sh
    M tools/virtiofsd/fuse_common.h
    M tools/virtiofsd/fuse_lowlevel.c
    M tools/virtiofsd/fuse_lowlevel.h
    M tools/virtiofsd/fuse_virtio.c
    M tools/virtiofsd/helper.c
    M tools/virtiofsd/passthrough_ll.c
    M tools/virtiofsd/passthrough_seccomp.c

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20201102a' 
into staging

Migration and virtiofs fixes 2020-11-02

Fixes for postcopy migration test hang
A seccomp crash for virtiofsd on some !x86
Help message and minor CID fix

And another crack at Max's set.

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

# gpg: Signature made Mon 02 Nov 2020 19:54:59 GMT
# gpg:                using RSA key 45F5C71B4A0CB7FB977A9FA90516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>" 
[full]
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A  9FA9 0516 331E BC5B FDE7

* remotes/dgilbert/tags/pull-migration-20201102a:
  tests/acceptance: Add virtiofs_submounts.py
  tests/acceptance/boot_linux: Accept SSH pubkey
  virtiofsd: Announce sub-mount points
  virtiofsd: Add mount ID to the lo_inode key
  meson.build: Check for statx()
  virtiofsd: Add attr_flags to fuse_entry_param
  virtiofsd: Check FUSE_SUBMOUNTS
  virtiofsd: Fix the help message of posix lock
  tools/virtiofsd: Check vu_init() return value (CID 1435958)
  virtiofsd: Seccomp: Add 'send' for syslog
  migration: Postpone the kick of the fault thread after recover
  migration: Unify reset of last_rb on destination node when recover

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>


Compare: https://github.com/qemu/qemu/compare/8680d6e36468...8545ae485b1e