[Qemu-commits] [qemu/qemu] 945c1e: blockdev-backup: add bitmap argument

[GitHub]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 945c1ee0cb7d29f2fd0fece2cd2b5329802de5e9
      
https://github.com/qemu/qemu/commit/945c1ee0cb7d29f2fd0fece2cd2b5329802de5e9
  Author: John Snow <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M blockdev.c
    M qapi/block-core.json

  Log Message:
  -----------
  blockdev-backup: add bitmap argument

It is only an oversight that we don't allow incremental backup with
blockdev-backup. Add the bitmap argument which enables this.

Signed-off-by: John Snow <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 06bf50068a7e952afff8c4f6470ec54a712570f7
      
https://github.com/qemu/qemu/commit/06bf50068a7e952afff8c4f6470ec54a712570f7
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M block/dirty-bitmap.c
    M blockdev.c

  Log Message:
  -----------
  dirty-bitmap: switch assert-fails to errors in bdrv_merge_dirty_bitmap

Move checks from qmp_x_block_dirty_bitmap_merge() to
bdrv_merge_dirty_bitmap(), to share them with dirty bitmap merge
transaction action in future commit.

Note: for now, only qmp_x_block_dirty_bitmap_merge() calls
bdrv_merge_dirty_bitmap().

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: John Snow <address@hidden>


  Commit: 56bd662497259400b7c9f155aaebaddde4450028
      
https://github.com/qemu/qemu/commit/56bd662497259400b7c9f155aaebaddde4450028
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M block/dirty-bitmap.c
    M blockdev.c
    M include/block/block_int.h

  Log Message:
  -----------
  dirty-bitmap: rename bdrv_undo_clear_dirty_bitmap

Use more generic names to reuse the function for bitmap merge in the
following commit.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: John Snow <address@hidden>


  Commit: fa000f2f9fd96a75a0a33d50ead247fce11da92a
      
https://github.com/qemu/qemu/commit/fa000f2f9fd96a75a0a33d50ead247fce11da92a
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M block/dirty-bitmap.c
    M blockdev.c
    M include/block/dirty-bitmap.h
    M include/qemu/hbitmap.h
    M util/hbitmap.c

  Log Message:
  -----------
  dirty-bitmap: make it possible to restore bitmap after merge

Add backup parameter to bdrv_merge_dirty_bitmap() to be used then with
bdrv_restore_dirty_bitmap() if it needed to restore the bitmap after
merge operation.

This is needed to implement bitmap merge transaction action in further
commit.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: John Snow <address@hidden>


  Commit: 5c4cf8b294ee65c049d6c40f5f6ff7c1befdb3d9
      
https://github.com/qemu/qemu/commit/5c4cf8b294ee65c049d6c40f5f6ff7c1befdb3d9
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M blockdev.c

  Log Message:
  -----------
  blockdev: rename block-dirty-bitmap-clear transaction handlers

Rename block-dirty-bitmap-clear transaction handlers to reuse them for
x-block-dirty-bitmap-merge transaction in the following patch.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: John Snow <address@hidden>


  Commit: 6fd2e40789ef7389b17c5fff93b0bf82d4352cb3
      
https://github.com/qemu/qemu/commit/6fd2e40789ef7389b17c5fff93b0bf82d4352cb3
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M blockdev.c
    M qapi/transaction.json

  Log Message:
  -----------
  qapi: add transaction support for x-block-dirty-bitmap-merge

New action is like clean action: do the whole thing in .prepare and
undo in .abort. This behavior for bitmap-changing actions is needed
because backup job actions use bitmap in .prepare.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: John Snow <address@hidden>


  Commit: 304cc429a07eb6601020212a478050ebbe87df88
      
https://github.com/qemu/qemu/commit/304cc429a07eb6601020212a478050ebbe87df88
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M tests/qemu-iotests/169

  Log Message:
  -----------
  iotests: 169: drop deprecated 'autoload' parameter

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: John Snow <address@hidden>
Signed-off-by: John Snow <address@hidden>


  Commit: 132adb682098e9af40a2132ec4feec6850fce8cd
      
https://github.com/qemu/qemu/commit/132adb682098e9af40a2132ec4feec6850fce8cd
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M block/qcow2.c

  Log Message:
  -----------
  block/qcow2: improve error message in qcow2_inactivate

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
[Maintainer edit -- touched up error message. --js]
Reviewed-by: John Snow <address@hidden>
Signed-off-by: John Snow <address@hidden>


  Commit: 2ea427effff61efa5d0dc69f9cae126d13879617
      
https://github.com/qemu/qemu/commit/2ea427effff61efa5d0dc69f9cae126d13879617
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M block/qcow2.c
    M block/qcow2.h

  Log Message:
  -----------
  bloc/qcow2: drop dirty_bitmaps_loaded state variable

This variable doesn't work as it should, because it is actually cleared
in qcow2_co_invalidate_cache() by memset(). Drop it, as the following
patch will introduce new behavior.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: John Snow <address@hidden>
Signed-off-by: John Snow <address@hidden>


  Commit: 993edc0ce0c6f44deb8272a7a857e419417f5f84
      
https://github.com/qemu/qemu/commit/993edc0ce0c6f44deb8272a7a857e419417f5f84
  Author: John Snow <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M block/dirty-bitmap.c
    M blockdev.c
    M include/block/dirty-bitmap.h
    M migration/block-dirty-bitmap.c

  Log Message:
  -----------
  block/dirty-bitmaps: add user_locked status checker

Instead of both frozen and qmp_locked checks, wrap it into one check.
frozen implies the bitmap is split in two (for backup), and shouldn't
be modified. qmp_locked implies it's being used by another operation,
like being exported over NBD. In both cases it means we shouldn't allow
the user to modify it in any meaningful way.

Replace any usages where we check both frozen and qmp_locked with the
new check.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Message-id: address@hidden
[w/edits Suggested-By: Vladimir Sementsov-Ogievskiy <address@hidden>]
Signed-off-by: John Snow <address@hidden>


  Commit: 283d7a04f2addcc51468635300208b60c19a0db3
      
https://github.com/qemu/qemu/commit/283d7a04f2addcc51468635300208b60c19a0db3
  Author: John Snow <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M block/dirty-bitmap.c

  Log Message:
  -----------
  block/dirty-bitmaps: fix merge permissions

In prior commits that made merge transactionable, we removed the
assertion that merge cannot operate on disabled bitmaps. In addition,
we want to make sure that we are prohibiting merges to "locked" bitmaps.

Use the new user_locked function to check.

Reported-by: Eric Blake <address@hidden>
Signed-off-by: John Snow <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Reviewed-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 0be37c9e19f541643ef407bdafe0282b667ec23c
      
https://github.com/qemu/qemu/commit/0be37c9e19f541643ef407bdafe0282b667ec23c
  Author: John Snow <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M block/dirty-bitmap.c
    M blockdev.c

  Log Message:
  -----------
  block/dirty-bitmaps: allow clear on disabled bitmaps

Similarly to merge, it's OK to allow clear operations on disabled
bitmaps, as this condition only means that they are not recording
new writes. We are free to clear it if the user requests it.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Reviewed-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: b053bb55738f35832f3d6472b12277a75c32a038
      
https://github.com/qemu/qemu/commit/b053bb55738f35832f3d6472b12277a75c32a038
  Author: John Snow <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M blockdev.c

  Log Message:
  -----------
  block/dirty-bitmaps: prohibit enable/disable on locked/frozen bitmaps

We're not being consistent about this. If it's in use by an operation,
the user should not be able to change the behavior of that bitmap.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Reviewed-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: b27a6b8b329a8dcbab9dc1af45586f7585f3d47b
      
https://github.com/qemu/qemu/commit/b27a6b8b329a8dcbab9dc1af45586f7585f3d47b
  Author: John Snow <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M blockdev.c

  Log Message:
  -----------
  block/backup: prohibit backup from using in use bitmaps

If the bitmap is frozen, we shouldn't touch it.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: d9782022bda7f8eccaf961044e9efe980dc90c04
      
https://github.com/qemu/qemu/commit/d9782022bda7f8eccaf961044e9efe980dc90c04
  Author: John Snow <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M nbd/server.c

  Log Message:
  -----------
  nbd: forbid use of frozen bitmaps

Whether it's "locked" or "frozen", it's in use and should
not be allowed for the purposes of this operation.

Signed-off-by: John Snow <address@hidden>
Reviewed-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: d1dde7149e376d72b422a529ec4bf3ed47f3ba30
      
https://github.com/qemu/qemu/commit/d1dde7149e376d72b422a529ec4bf3ed47f3ba30
  Author: Eric Blake <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M util/hbitmap.c

  Log Message:
  -----------
  bitmap: Update count after a merge

We need an accurate count of the number of bits set in a bitmap
after a merge. In particular, since the merge operation short-circuits
a merge from an empty source, if you have bitmaps A, B, and C where
B started empty, then merge C into B, and B into A, an inaccurate
count meant that A did not get the contents of C.

In the worst case, we may falsely regard the bitmap as empty when
it has had new writes merged into it.

Fixes: be58721db
CC: address@hidden
Signed-off-by: Eric Blake <address@hidden>
Signed-off-by: John Snow <address@hidden>
Reviewed-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Signed-off-by: John Snow <address@hidden>


  Commit: 9c98f145dfb994e1e9d68a4d606ee5693891280d
      
https://github.com/qemu/qemu/commit/9c98f145dfb994e1e9d68a4d606ee5693891280d
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M block.c
    M block/dirty-bitmap.c
    M block/qcow2-bitmap.c
    M block/qcow2.c
    M include/block/dirty-bitmap.h
    M migration/block-dirty-bitmap.c

  Log Message:
  -----------
  dirty-bitmaps: clean-up bitmaps loading and migration logic

This patch aims to bring the following behavior:

1. We don't load bitmaps, when started in inactive mode. It's the case
of incoming migration. In this case we wait for bitmaps migration
through migration channel (if 'dirty-bitmaps' capability is enabled) or
for invalidation (to load bitmaps from the image).

2. We don't remove persistent bitmaps on inactivation. Instead, we only
remove bitmaps after storing. This is the only way to restore bitmaps,
if we decided to resume source after [failed] migration with
'dirty-bitmaps' capability enabled (which means, that bitmaps were not
stored).

3. We load bitmaps on open and any invalidation, it's ok for all cases:
  - normal open
  - migration target invalidation with dirty-bitmaps capability
    (bitmaps are migrating through migration channel, the are not
     stored, so they should have IN_USE flag set and will be skipped
     when loading. However, it would fail if bitmaps are read-only[1])
  - migration target invalidation without dirty-bitmaps capability
    (normal load of the bitmaps, if migrated with shared storage)
  - source invalidation with dirty-bitmaps capability
    (skip because IN_USE)
  - source invalidation without dirty-bitmaps capability
    (bitmaps were dropped, reload them)

[1]: to accurately handle this, migration of read-only bitmaps is
     explicitly forbidden in this patch.

New mechanism for not storing bitmaps when migrate with dirty-bitmaps
capability is introduced: migration filed in BdrvDirtyBitmap.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Signed-off-by: John Snow <address@hidden>


  Commit: b9247fc1a8ffe5c367fa049f295fbb58c8ca9d05
      
https://github.com/qemu/qemu/commit/b9247fc1a8ffe5c367fa049f295fbb58c8ca9d05
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M tests/qemu-iotests/169

  Log Message:
  -----------
  iotests: improve 169

Before previous patch, iotest 169 was actually broken for the case
test_persistent__not_migbitmap__offline_shared, while formally
passing.

After migration log of vm_b had message:

    qemu-system-x86_64: Could not reopen qcow2 layer: Bitmap already
    exists: bitmap0

which means that invalidation failed and bs->drv = NULL.

It was because we've loaded bitmap twice: on open and on invalidation.

Add code to 169, to catch such fails.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Signed-off-by: John Snow <address@hidden>


  Commit: 3e6d88f280a53b5b399e73b1f80efe4c3db306f1
      
https://github.com/qemu/qemu/commit/3e6d88f280a53b5b399e73b1f80efe4c3db306f1
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2018-10-29 (Mon, 29 Oct 2018)

  Changed paths:
    M tests/qemu-iotests/169
    M tests/qemu-iotests/169.out

  Log Message:
  -----------
  iotests: 169: add cases for source vm resuming

Test that we can resume source vm after [failed] migration, and bitmaps
are ok.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Signed-off-by: John Snow <address@hidden>


  Commit: 3f3285491dd52014852a56135c90e428c8b507ea
      
https://github.com/qemu/qemu/commit/3f3285491dd52014852a56135c90e428c8b507ea
  Author: Peter Maydell <address@hidden>
  Date:   2018-10-30 (Tue, 30 Oct 2018)

  Changed paths:
    M block.c
    M block/dirty-bitmap.c
    M block/qcow2-bitmap.c
    M block/qcow2.c
    M block/qcow2.h
    M blockdev.c
    M include/block/block_int.h
    M include/block/dirty-bitmap.h
    M include/qemu/hbitmap.h
    M migration/block-dirty-bitmap.c
    M nbd/server.c
    M qapi/block-core.json
    M qapi/transaction.json
    M tests/qemu-iotests/169
    M tests/qemu-iotests/169.out
    M util/hbitmap.c

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/jnsnow/tags/bitmaps-pull-request' into 
staging

Pull request

# gpg: Signature made Mon 29 Oct 2018 21:24:08 GMT
# gpg:                using RSA key 7DEF8106AAFC390E
# gpg: Good signature from "John Snow (John Huston) <address@hidden>"
# Primary key fingerprint: FAEB 9711 A12C F475 812F  18F2 88A9 064D 1835 61EB
#      Subkey fingerprint: F9B7 ABDB BCAC DF95 BE76  CBD0 7DEF 8106 AAFC 390E

* remotes/jnsnow/tags/bitmaps-pull-request:
  iotests: 169: add cases for source vm resuming
  iotests: improve 169
  dirty-bitmaps: clean-up bitmaps loading and migration logic
  bitmap: Update count after a merge
  nbd: forbid use of frozen bitmaps
  block/backup: prohibit backup from using in use bitmaps
  block/dirty-bitmaps: prohibit enable/disable on locked/frozen bitmaps
  block/dirty-bitmaps: allow clear on disabled bitmaps
  block/dirty-bitmaps: fix merge permissions
  block/dirty-bitmaps: add user_locked status checker
  bloc/qcow2: drop dirty_bitmaps_loaded state variable
  block/qcow2: improve error message in qcow2_inactivate
  iotests: 169: drop deprecated 'autoload' parameter
  qapi: add transaction support for x-block-dirty-bitmap-merge
  blockdev: rename block-dirty-bitmap-clear transaction handlers
  dirty-bitmap: make it possible to restore bitmap after merge
  dirty-bitmap: rename bdrv_undo_clear_dirty_bitmap
  dirty-bitmap: switch assert-fails to errors in bdrv_merge_dirty_bitmap
  blockdev-backup: add bitmap argument

Signed-off-by: Peter Maydell <address@hidden>


Compare: https://github.com/qemu/qemu/compare/e8b38d737144...3f3285491dd5
      **NOTE:** This service has been marked for deprecation: 
https://developer.github.com/changes/2018-04-25-github-services-deprecation/

      Functionality will be removed from GitHub.com on January 31st, 2019.