[Qemu-commits] [qemu/qemu] 7ff4f1: tests: fix encoding of IP addresses in x509 certs

[Richard Henderson]

  Branch: refs/heads/staging
  Home:   https://github.com/qemu/qemu
  Commit: 7ff4f1d516eac7e5dce965070ffecc02be92b4f7
      
https://github.com/qemu/qemu/commit/7ff4f1d516eac7e5dce965070ffecc02be92b4f7
  Author: Daniel P. Berrangé <berrange@redhat.com>
  Date:   2022-04-28 (Thu, 28 Apr 2022)

  Changed paths:
    M tests/unit/crypto-tls-x509-helpers.c
    M tests/unit/test-crypto-tlssession.c

  Log Message:
  -----------
  tests: fix encoding of IP addresses in x509 certs

We need to encode just the address bytes, not the whole struct sockaddr
data. Add a test case to validate that we're matching on SAN IP
addresses correctly.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220426160048.812266-2-berrange@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: 707e3d2b2d5ddf38d1eb77cb67dc40dd2077b5dd
      
https://github.com/qemu/qemu/commit/707e3d2b2d5ddf38d1eb77cb67dc40dd2077b5dd
  Author: Daniel P. Berrangé <berrange@redhat.com>
  Date:   2022-04-28 (Thu, 28 Apr 2022)

  Changed paths:
    M tests/qtest/migration-test.c

  Log Message:
  -----------
  tests: convert XBZRLE migration test to use common helper

Most of the XBZRLE migration test logic is common with the rest of the
precopy tests, so it can use the helper with just one small tweak.

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220426160048.812266-6-berrange@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
  dgilbert: Manual merge since I skipped some of the other patches


  Commit: 6e578dcbfc8db5c50f542b74f3d3711a208c8889
      
https://github.com/qemu/qemu/commit/6e578dcbfc8db5c50f542b74f3d3711a208c8889
  Author: Daniel P. Berrangé <berrange@redhat.com>
  Date:   2022-04-28 (Thu, 28 Apr 2022)

  Changed paths:
    M tests/qtest/migration-test.c

  Log Message:
  -----------
  tests: convert multifd migration tests to use common helper

Most of the multifd migration test logic is common with the rest of the
precopy tests, so it can use the helper without difficulty. The only
exception of the multifd cancellation test which tries to run multiple
migrations in a row.

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220426160048.812266-7-berrange@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: ba703f9d64091c63f874803b0be1931e1a0daf35
      
https://github.com/qemu/qemu/commit/ba703f9d64091c63f874803b0be1931e1a0daf35
  Author: Daniel P. Berrangé <berrange@redhat.com>
  Date:   2022-04-28 (Thu, 28 Apr 2022)

  Changed paths:
    M tests/qtest/migration-helpers.c
    M tests/qtest/migration-helpers.h
    M tests/qtest/migration-test.c

  Log Message:
  -----------
  tests: ensure migration status isn't reported as failed

Various methods in the migration test call 'query_migrate' to fetch the
current status and then access a particular field. Almost all of these
cases expect the migration to be in a non-failed state. In the case of
'wait_for_migration_pass' in particular, if the status is 'failed' then
it will get into an infinite loop. By validating that the status is
not 'failed' the test suite will assert rather than hang when getting
into an unexpected state.

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220426160048.812266-10-berrange@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: 0974a3420c5611adaf3f4557f492ab873286aa33
      
https://github.com/qemu/qemu/commit/0974a3420c5611adaf3f4557f492ab873286aa33
  Author: Leonardo Bras <leobras@redhat.com>
  Date:   2022-04-28 (Thu, 28 Apr 2022)

  Changed paths:
    M chardev/char-io.c
    M hw/remote/mpqemu-link.c
    M include/io/channel.h
    M io/channel-buffer.c
    M io/channel-command.c
    M io/channel-file.c
    M io/channel-socket.c
    M io/channel-tls.c
    M io/channel-websock.c
    M io/channel.c
    M migration/rdma.c
    M scsi/pr-manager-helper.c
    M tests/unit/test-io-channel-socket.c

  Log Message:
  -----------
  QIOChannel: Add flags on io_writev and introduce io_flush callback

Add flags to io_writev and introduce io_flush as optional callback to
QIOChannelClass, allowing the implementation of zero copy writes by
subclasses.

How to use them:
- Write data using qio_channel_writev*(...,QIO_CHANNEL_WRITE_FLAG_ZERO_COPY),
- Wait write completion with qio_channel_flush().

Notes:
As some zero copy write implementations work asynchronously, it's
recommended to keep the write buffer untouched until the return of
qio_channel_flush(), to avoid the risk of sending an updated buffer
instead of the buffer state during write.

As io_flush callback is optional, if a subclass does not implement it, then:
- io_flush will return 0 without changing anything.

Also, some functions like qio_channel_writev_full_all() were adapted to
receive a flag parameter. That allows shared code between zero copy and
non-zero copy writev, and also an easier implementation on new flags.

Signed-off-by: Leonardo Bras <leobras@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Message-Id: <20220426230654.637939-2-leobras@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: e294321b70971e21155746b1c0f3ae51c2b599f1
      
https://github.com/qemu/qemu/commit/e294321b70971e21155746b1c0f3ae51c2b599f1
  Author: Leonardo Bras <leobras@redhat.com>
  Date:   2022-04-28 (Thu, 28 Apr 2022)

  Changed paths:
    M include/io/channel-socket.h
    M io/channel-socket.c

  Log Message:
  -----------
  QIOChannelSocket: Implement io_writev zero copy flag & io_flush for 
CONFIG_LINUX

For CONFIG_LINUX, implement the new zero copy flag and the optional callback
io_flush on QIOChannelSocket, but enables it only when MSG_ZEROCOPY
feature is available in the host kernel, which is checked on
qio_channel_socket_connect_sync()

qio_channel_socket_flush() was implemented by counting how many times
sendmsg(...,MSG_ZEROCOPY) was successfully called, and then reading the
socket's error queue, in order to find how many of them finished sending.
Flush will loop until those counters are the same, or until some error occurs.

Notes on using writev() with QIO_CHANNEL_WRITE_FLAG_ZERO_COPY:
1: Buffer
- As MSG_ZEROCOPY tells the kernel to use the same user buffer to avoid copying,
some caution is necessary to avoid overwriting any buffer before it's sent.
If something like this happen, a newer version of the buffer may be sent 
instead.
- If this is a problem, it's recommended to call qio_channel_flush() before 
freeing
or re-using the buffer.

2: Locked memory
- When using MSG_ZERCOCOPY, the buffer memory will be locked after queued, and
unlocked after it's sent.
- Depending on the size of each buffer, and how often it's sent, it may require
a larger amount of locked memory than usually available to non-root user.
- If the required amount of locked memory is not available, writev_zero_copy
will return an error, which can abort an operation like migration,
- Because of this, when an user code wants to add zero copy as a feature, it
requires a mechanism to disable it, so it can still be accessible to less
privileged users.

Signed-off-by: Leonardo Bras <leobras@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Message-Id: <20220426230654.637939-3-leobras@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: f4d7a2530bd29d152e689c5968c9e18df6741732
      
https://github.com/qemu/qemu/commit/f4d7a2530bd29d152e689c5968c9e18df6741732
  Author: Leonardo Bras <leobras@redhat.com>
  Date:   2022-04-28 (Thu, 28 Apr 2022)

  Changed paths:
    M migration/migration.c
    M migration/migration.h
    M migration/socket.c
    M monitor/hmp-cmds.c
    M qapi/migration.json

  Log Message:
  -----------
  migration: Add zero-copy-send parameter for QMP/HMP for Linux

Add property that allows zero-copy migration of memory pages
on the sending side, and also includes a helper function
migrate_use_zero_copy_send() to check if it's enabled.

No code is introduced to actually do the migration, but it allow
future implementations to enable/disable this feature.

On non-Linux builds this parameter is compiled-out.

Signed-off-by: Leonardo Bras <leobras@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Message-Id: <20220426230654.637939-4-leobras@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: a0e236a86dd5c076e224861ba0bacae17ffd0d11
      
https://github.com/qemu/qemu/commit/a0e236a86dd5c076e224861ba0bacae17ffd0d11
  Author: Leonardo Bras <leobras@redhat.com>
  Date:   2022-04-28 (Thu, 28 Apr 2022)

  Changed paths:
    M migration/channel.c
    M migration/migration.c
    M migration/migration.h
    M migration/multifd.c

  Log Message:
  -----------
  migration: Add migrate_use_tls() helper

A lot of places check parameters.tls_creds in order to evaluate if TLS is
in use, and sometimes call migrate_get_current() just for that test.

Add new helper function migrate_use_tls() in order to simplify testing
for TLS usage.

Signed-off-by: Leonardo Bras <leobras@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220426230654.637939-5-leobras@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: d583a7ca5c41e5249e5fc011956d6f6b8137d4b4
      
https://github.com/qemu/qemu/commit/d583a7ca5c41e5249e5fc011956d6f6b8137d4b4
  Author: Leonardo Bras <leobras@redhat.com>
  Date:   2022-04-28 (Thu, 28 Apr 2022)

  Changed paths:
    M migration/multifd.c
    M migration/multifd.h
    M migration/ram.c

  Log Message:
  -----------
  multifd: multifd_send_sync_main now returns negative on error

Even though multifd_send_sync_main() currently emits error_reports, it's
callers don't really check it before continuing.

Change multifd_send_sync_main() to return -1 on error and 0 on success.
Also change all it's callers to make use of this change and possibly fail
earlier.

(This change is important to next patch on  multifd zero copy
implementation, to make it sure an error in zero-copy flush does not go
unnoticed.

Signed-off-by: Leonardo Bras <leobras@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Message-Id: <20220426230654.637939-6-leobras@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: 1b1bc090ab4daa40f3263b8c1202473aa8adee5e
      
https://github.com/qemu/qemu/commit/1b1bc090ab4daa40f3263b8c1202473aa8adee5e
  Author: Leonardo Bras <leobras@redhat.com>
  Date:   2022-04-28 (Thu, 28 Apr 2022)

  Changed paths:
    M migration/multifd.c

  Log Message:
  -----------
  multifd: Send header packet without flags if zero-copy-send is enabled

Since d48c3a0445 ("multifd: Use a single writev on the send side"),
sending the header packet and the memory pages happens in the same
writev, which can potentially make the migration faster.

Using channel-socket as example, this works well with the default copying
mechanism of sendmsg(), but with zero-copy-send=true, it will cause
the migration to often break.

This happens because the header packet buffer gets reused quite often,
and there is a high chance that by the time the MSG_ZEROCOPY mechanism get
to send the buffer, it has already changed, sending the wrong data and
causing the migration to abort.

It means that, as it is, the buffer for the header packet is not suitable
for sending with MSG_ZEROCOPY.

In order to enable zero copy for multifd, send the header packet on an
individual write(), without any flags, and the remanining pages with a
writev(), as it was happening before. This only changes how a migration
with zero-copy-send=true works, not changing any current behavior for
migrations with zero-copy-send=false.

Signed-off-by: Leonardo Bras <leobras@redhat.com>
Message-Id: <20220426230654.637939-7-leobras@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
  dgilbert: Removed blank line


  Commit: 62c49432662815dc520a41cd9f2adbd7ec1e22f4
      
https://github.com/qemu/qemu/commit/62c49432662815dc520a41cd9f2adbd7ec1e22f4
  Author: Leonardo Bras <leobras@redhat.com>
  Date:   2022-04-28 (Thu, 28 Apr 2022)

  Changed paths:
    M migration/migration.c
    M migration/multifd.c
    M migration/multifd.h
    M migration/socket.c

  Log Message:
  -----------
  multifd: Implement zero copy write in multifd migration (multifd-zero-copy)

Implement zero copy send on nocomp_send_write(), by making use of QIOChannel
writev + flags & flush interface.

Change multifd_send_sync_main() so flush_zero_copy() can be called
after each iteration in order to make sure all dirty pages are sent before
a new iteration is started. It will also flush at the beginning and at the
end of migration.

Also make it return -1 if flush_zero_copy() fails, in order to cancel
the migration process, and avoid resuming the guest in the target host
without receiving all current RAM.

This will work fine on RAM migration because the RAM pages are not usually 
freed,
and there is no problem on changing the pages content between 
writev_zero_copy() and
the actual sending of the buffer, because this change will dirty the page and
cause it to be re-sent on a next iteration anyway.

A lot of locked memory may be needed in order to use multifd migration
with zero-copy enabled, so disabling the feature should be necessary for
low-privileged users trying to perform multifd migrations.

Signed-off-by: Leonardo Bras <leobras@redhat.com>
Message-Id: <20220426230654.637939-8-leobras@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>


  Commit: ba6154cd0ce0a644288044299100fc439013fb7c
      
https://github.com/qemu/qemu/commit/ba6154cd0ce0a644288044299100fc439013fb7c
  Author: Richard Henderson <richard.henderson@linaro.org>
  Date:   2022-04-28 (Thu, 28 Apr 2022)

  Changed paths:
    M chardev/char-io.c
    M hw/remote/mpqemu-link.c
    M include/io/channel-socket.h
    M include/io/channel.h
    M io/channel-buffer.c
    M io/channel-command.c
    M io/channel-file.c
    M io/channel-socket.c
    M io/channel-tls.c
    M io/channel-websock.c
    M io/channel.c
    M migration/channel.c
    M migration/migration.c
    M migration/migration.h
    M migration/multifd.c
    M migration/multifd.h
    M migration/ram.c
    M migration/rdma.c
    M migration/socket.c
    M monitor/hmp-cmds.c
    M qapi/migration.json
    M scsi/pr-manager-helper.c
    M tests/qtest/migration-helpers.c
    M tests/qtest/migration-helpers.h
    M tests/qtest/migration-test.c
    M tests/unit/crypto-tls-x509-helpers.c
    M tests/unit/test-crypto-tlssession.c
    M tests/unit/test-io-channel-socket.c

  Log Message:
  -----------
  Merge tag 'pull-migration-20220428a' of https://gitlab.com/dagrh/qemu into 
staging

Migration pull 2022-04-28

Zero copy send features from Leo
Test fixes from Dan

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCAAdFiEERfXHG0oMt/uXep+pBRYzHrxb/ecFAmJqprIACgkQBRYzHrxb
# /edG6hAAli6/wpDeRbIilhrg86sA1KKdlVVu29p6Xbl9auEw1Dy2tUG44zI58bm4
# RPQpnhDcSB1eQw3QdItfnMrbOQcZDIDskllGobcMGQHfhepJ0jf+pu3meABMTBv1
# cqexfkcl+iPrj687Nm+4jc/RIqDRjzqidhtulYrPWlJMD79zapiqxZ7h6FXA6RBQ
# x5peEusb8FgDDMLHa8v4ZCGAgqKS0zmzG44z8LMLLqXjwcRiD4/6w0tK0eBXhiJi
# Mkh7mEN3GarroKDki6mZGpatBmaleZ81PW1LRNvgTFIgKEvN7JvIYtKSTnH4hglG
# 3J94CQMED8945hfH5sKM95id2d5nyRuA9y9xkPYP7EKx/qZwMZgP1u8XOs53/u6J
# CDqmCO3hyGd7tw9JCrb8i9udnMn03/kBIo1/bEx/pHzw+yf93NwXSstwFQxerLsF
# ClAOM29Cc+hluljuKr6gatT6VApzrIu12KfPOrSwbvAfU2rJaJS8O2NmS3yuGyvp
# wflt0aDv/1aXsnj+pHAbBy35PvYRly6GQ5esta7vRtQOvF4Plw1+QcQBaxuGUlaV
# OlRHF1soBc+tVv9eI7aOYvMOLW/ayw3E7MerHq82UZbGUirzTXOT9+XePa/bZbpZ
# L2VsZavfv6G+o/GsPRRwHINITWCagtQyKeiHh4ufpqqvh9kDiFg=
# =DW1r
# -----END PGP SIGNATURE-----
# gpg: Signature made Thu 28 Apr 2022 07:37:38 AM PDT
# gpg:                using RSA key 45F5C71B4A0CB7FB977A9FA90516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>" 
[full]

* tag 'pull-migration-20220428a' of https://gitlab.com/dagrh/qemu:
  multifd: Implement zero copy write in multifd migration (multifd-zero-copy)
  multifd: Send header packet without flags if zero-copy-send is enabled
  multifd: multifd_send_sync_main now returns negative on error
  migration: Add migrate_use_tls() helper
  migration: Add zero-copy-send parameter for QMP/HMP for Linux
  QIOChannelSocket: Implement io_writev zero copy flag & io_flush for 
CONFIG_LINUX
  QIOChannel: Add flags on io_writev and introduce io_flush callback
  tests: ensure migration status isn't reported as failed
  tests: convert multifd migration tests to use common helper
  tests: convert XBZRLE migration test to use common helper
  tests: fix encoding of IP addresses in x509 certs

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>


Compare: https://github.com/qemu/qemu/compare/f22833602095...ba6154cd0ce0