[Qemu-commits] [qemu/qemu] 38908b: migration: refactor init_dirty_bitmap_migration

[Peter Maydell]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 38908bbc65bcabf110dcc87a84c518e3e6d9a183
      
https://github.com/qemu/qemu/commit/38908bbc65bcabf110dcc87a84c518e3e6d9a183
  Author: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
  Date:   2020-05-28 (Thu, 28 May 2020)

  Changed paths:
    M migration/block-dirty-bitmap.c

  Log Message:
  -----------
  migration: refactor init_dirty_bitmap_migration

Split out handling one bs, it is needed for the following commit, which
will handle BlockBackends separately.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200521220648.3255-2-vsementsov@virtuozzo.com>
[eblake: shorter subject line]
Signed-off-by: Eric Blake <eblake@redhat.com>


  Commit: 7ae89a0de945616fcde5947cec5aecb8dce07d81
      
https://github.com/qemu/qemu/commit/7ae89a0de945616fcde5947cec5aecb8dce07d81
  Author: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
  Date:   2020-05-28 (Thu, 28 May 2020)

  Changed paths:
    M block/dirty-bitmap.c
    M include/block/dirty-bitmap.h

  Log Message:
  -----------
  block/dirty-bitmap: add bdrv_has_named_bitmaps helper

To be used for bitmap migration in further commit.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200521220648.3255-3-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>


  Commit: 107cfb72541c70030caa9a21578edd73127cf2b4
      
https://github.com/qemu/qemu/commit/107cfb72541c70030caa9a21578edd73127cf2b4
  Author: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
  Date:   2020-05-28 (Thu, 28 May 2020)

  Changed paths:
    M migration/block-dirty-bitmap.c

  Log Message:
  -----------
  migration: fix bitmaps pre-blockdev migration with mirror job

Important thing for bitmap migration is to select destination block
node to obtain the migrated bitmap.

Prepatch, on source we use bdrv_get_device_or_node_name() to identify
the node, and on target we do bdrv_lookup_bs.
bdrv_get_device_or_node_name() returns blk name only for direct
children of blk. So, bitmaps of direct children of blks are migrated by
blk name and others - by node name.

Old libvirt is unprepared to bitmap migration by node-name,
node-names are mostly auto-generated. So actually only migration by blk
name works for it.

Newer libvirt will use new interface (which will be added soon) to
specify node-mapping for bitmaps migration explicitly. Still, let's
improve the current behavior a bit.

Now, consider classic libvirt migrations assisted by mirror block job:
mirror block job inserts filter, so our source is not a direct child of
blk, and bitmaps are migrated by node-names. And this just doesn't work
with auto-generated node names.

Let's fix it by using blk-name even if some implicit filters are
inserted.

Note2: we, of course, can't skip filters and use blk name to migrate
bitmaps in filtered node by blk name for this blk if these filters have
named bitmaps which should be migrated.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1652424
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200521220648.3255-4-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[eblake: comment typo fix, shorter subject line]
Signed-off-by: Eric Blake <eblake@redhat.com>


  Commit: ae00aa2398476824f0eca80461da215e7cdc1c3b
      
https://github.com/qemu/qemu/commit/ae00aa2398476824f0eca80461da215e7cdc1c3b
  Author: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
  Date:   2020-05-28 (Thu, 28 May 2020)

  Changed paths:
    M tests/qemu-iotests/194
    M tests/qemu-iotests/194.out

  Log Message:
  -----------
  iotests: 194: test also migration of dirty bitmap

Test that dirty bitmap migration works when we deal with mirror.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200521220648.3255-5-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>


  Commit: 82640edb88faacad90ff19f6fced37f9c8025e48
      
https://github.com/qemu/qemu/commit/82640edb88faacad90ff19f6fced37f9c8025e48
  Author: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
  Date:   2020-05-28 (Thu, 28 May 2020)

  Changed paths:
    M migration/block-dirty-bitmap.c

  Log Message:
  -----------
  migration: add_bitmaps_to_list: check disk name once

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200521220648.3255-6-vsementsov@virtuozzo.com>
[eblake: shorter subject line]
Signed-off-by: Eric Blake <eblake@redhat.com>


  Commit: 4ff5cc121b089d8b16fa3da1e44feababcd1f815
      
https://github.com/qemu/qemu/commit/4ff5cc121b089d8b16fa3da1e44feababcd1f815
  Author: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
  Date:   2020-05-28 (Thu, 28 May 2020)

  Changed paths:
    M migration/block-dirty-bitmap.c

  Log Message:
  -----------
  migration: forbid bitmap migration by generated node-name

It actually never worked with libvirt, as auto-generated names are
different on source and destination.

It's unsafe and useless to migrate by auto-generated node-names, so
let's forbid it.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200521220648.3255-7-vsementsov@virtuozzo.com>
[eblake: shorter subject line]
Signed-off-by: Eric Blake <eblake@redhat.com>


  Commit: ca01b7a641527052e3e8961845b40b81706ce5f9
      
https://github.com/qemu/qemu/commit/ca01b7a641527052e3e8961845b40b81706ce5f9
  Author: Eric Blake <eblake@redhat.com>
  Date:   2020-05-28 (Thu, 28 May 2020)

  Changed paths:
    M tests/qemu-iotests/178.out.qcow2
    M tests/qemu-iotests/178.out.raw

  Log Message:
  -----------
  iotests: Fix test 178

A recent change to qemu-img changed expected error message output, but
178 takes long enough to execute that it does not get run by 'make
check' or './check -g quick'.

Fixes: 43d589b074
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200521192137.1120211-2-eblake@redhat.com>


  Commit: 5d72c68b49769c927e90b78af6d90f6a384b26ac
      
https://github.com/qemu/qemu/commit/5d72c68b49769c927e90b78af6d90f6a384b26ac
  Author: Eric Blake <eblake@redhat.com>
  Date:   2020-05-28 (Thu, 28 May 2020)

  Changed paths:
    M block/crypto.c
    M block/qcow2-bitmap.c
    M block/qcow2.c
    M block/qcow2.h
    M block/raw-format.c
    M docs/tools/qemu-img.rst
    M qapi/block-core.json
    M qemu-img.c
    M tests/qemu-iotests/178.out.qcow2
    M tests/qemu-iotests/190
    M tests/qemu-iotests/190.out

  Log Message:
  -----------
  qcow2: Expose bitmaps' size during measure

It's useful to know how much space can be occupied by qcow2 persistent
bitmaps, even though such metadata is unrelated to the guest-visible
data.  Report this value as an additional QMP field, present when
measuring an existing image and output format that both support
bitmaps.  Update iotest 178 and 190 to updated output, as well as new
coverage in 190 demonstrating non-zero values made possible with the
recently-added qemu-img bitmap command (see 3b51ab4b).

The new 'bitmaps size:' field is displayed automatically as part of
'qemu-img measure' any time it is present in QMP (that is, any time
both the source image being measured and destination format support
bitmaps, even if the measurement is 0 because there are no bitmaps
present).  If the field is absent, it means that no bitmaps can be
copied (source, destination, or both lack bitmaps, including when
measuring based on size rather than on a source image).  This behavior
is compatible with an upcoming patch adding 'qemu-img convert
--bitmaps': that command will fail in the same situations where this
patch omits the field.

The addition of a new field demonstrates why we should always
zero-initialize qapi C structs; while the qcow2 driver still fully
populates all fields, the raw and crypto drivers had to be tweaked to
avoid uninitialized data.

Consideration was also given towards having a 'qemu-img measure
--bitmaps' which errors out when bitmaps are not possible, and
otherwise sums the bitmaps into the existing allocation totals rather
than displaying as a separate field, as a potential convenience
factor.  But this was ultimately decided to be more complexity than
necessary when the QMP interface was sufficient enough with bitmaps
remaining a separate field.

See also: https://bugzilla.redhat.com/1779904

Reported-by: Nir Soffer <nsoffer@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200521192137.1120211-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>


  Commit: 6c729dd832207d7347ecb074912f538e2942f269
      
https://github.com/qemu/qemu/commit/6c729dd832207d7347ecb074912f538e2942f269
  Author: Eric Blake <eblake@redhat.com>
  Date:   2020-05-28 (Thu, 28 May 2020)

  Changed paths:
    M qemu-img.c

  Log Message:
  -----------
  qemu-img: Factor out code for merging bitmaps

The next patch will add another client that wants to merge dirty
bitmaps; it will be easier to refactor the code to construct the QAPI
struct correctly into a helper function.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200521192137.1120211-4-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>


  Commit: 15e39ad95078d528dfb9a75417453cab60332b77
      
https://github.com/qemu/qemu/commit/15e39ad95078d528dfb9a75417453cab60332b77
  Author: Eric Blake <eblake@redhat.com>
  Date:   2020-05-28 (Thu, 28 May 2020)

  Changed paths:
    M docs/tools/qemu-img.rst
    M qemu-img-cmds.hx
    M qemu-img.c

  Log Message:
  -----------
  qemu-img: Add convert --bitmaps option

Make it easier to copy all the persistent bitmaps of (the top layer
of) a source image along with its guest-visible contents, by adding a
boolean flag for use with qemu-img convert.  This is basically
shorthand, as the same effect could be accomplished with a series of
'qemu-img bitmap --add' and 'qemu-img bitmap --merge -b source'
commands, or by their corresponding QMP commands.

Note that this command will fail in the same scenarios where 'qemu-img
measure' omits a 'bitmaps size:' line, namely, when either the source
or the destination lacks persistent bitmap support altogether.

See also https://bugzilla.redhat.com/show_bug.cgi?id=1779893

While touching this, clean up a couple coding issues spotted in the
same function: an extra blank line, and merging back-to-back 'if
(!skip_create)' blocks.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200521192137.1120211-5-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>


  Commit: cf2d1203dcfc2bf964453d83a2302231ce77f2dc
      
https://github.com/qemu/qemu/commit/cf2d1203dcfc2bf964453d83a2302231ce77f2dc
  Author: Eric Blake <eblake@redhat.com>
  Date:   2020-05-28 (Thu, 28 May 2020)

  Changed paths:
    A tests/qemu-iotests/291
    A tests/qemu-iotests/291.out
    M tests/qemu-iotests/group

  Log Message:
  -----------
  iotests: Add test 291 to for qemu-img bitmap coverage

Add a new test covering the 'qemu-img bitmap' subcommand, as well as
'qemu-img convert --bitmaps', both added in recent patches.

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200521192137.1120211-6-eblake@redhat.com>


  Commit: ce20db593f50752badbc94d6a96e4576aa4a2443
      
https://github.com/qemu/qemu/commit/ce20db593f50752badbc94d6a96e4576aa4a2443
  Author: Peter Maydell <peter.maydell@linaro.org>
  Date:   2020-05-29 (Fri, 29 May 2020)

  Changed paths:
    M block/crypto.c
    M block/dirty-bitmap.c
    M block/qcow2-bitmap.c
    M block/qcow2.c
    M block/qcow2.h
    M block/raw-format.c
    M docs/tools/qemu-img.rst
    M include/block/dirty-bitmap.h
    M migration/block-dirty-bitmap.c
    M qapi/block-core.json
    M qemu-img-cmds.hx
    M qemu-img.c
    M tests/qemu-iotests/178.out.qcow2
    M tests/qemu-iotests/178.out.raw
    M tests/qemu-iotests/190
    M tests/qemu-iotests/190.out
    M tests/qemu-iotests/194
    M tests/qemu-iotests/194.out
    A tests/qemu-iotests/291
    A tests/qemu-iotests/291.out
    M tests/qemu-iotests/group

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/ericb/tags/pull-bitmaps-2020-05-26-v3' 
into staging

bitmaps patches for 2020-05-26

- fix non-blockdev migration of bitmaps when mirror job is in use
- add bitmap sizing to 'qemu-img measure'
- add 'qemu-img convert --bitmaps'

# gpg: Signature made Thu 28 May 2020 19:16:47 BST
# gpg:                using RSA key 71C2CC22B1C4602927D2F3AAA7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg:                 aka "Eric Blake (Free Software Programmer) 
<ebb9@byu.net>" [full]
# gpg:                 aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2  F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-bitmaps-2020-05-26-v3:
  iotests: Add test 291 to for qemu-img bitmap coverage
  qemu-img: Add convert --bitmaps option
  qemu-img: Factor out code for merging bitmaps
  qcow2: Expose bitmaps' size during measure
  iotests: Fix test 178
  migration: forbid bitmap migration by generated node-name
  migration: add_bitmaps_to_list: check disk name once
  iotests: 194: test also migration of dirty bitmap
  migration: fix bitmaps pre-blockdev migration with mirror job
  block/dirty-bitmap: add bdrv_has_named_bitmaps helper
  migration: refactor init_dirty_bitmap_migration

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>


Compare: https://github.com/qemu/qemu/compare/c86274bc2e34...ce20db593f50