[Qemu-commits] [qemu/qemu] 50ab0e: replication: interrupt failover if the main device...

[GitHub]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 50ab0e0908d592b8bda56c2d7495e1190d734b0b
      
https://github.com/qemu/qemu/commit/50ab0e0908d592b8bda56c2d7495e1190d734b0b
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M block/replication.c

  Log Message:
  -----------
  replication: interrupt failover if the main device is closed

Without this change, there is a race condition in tests/test-replication.
Depending on how fast the failover job (active commit) runs, there is a
chance of two bad things happening:

1) replication_done can be called after the secondary has been closed
and hence when the BDRVReplicationState is not valid anymore.

2) two copies of the active disk are present during the
/replication/secondary/stop test (that test runs immediately after
/replication/secondary/start, which tests failover).  This causes the
corruption detector to fire.

Reviewed-by: Wen Congyang <address@hidden>
Reviewed-by: Changlong Xie <address@hidden>
Signed-off-by: Paolo Bonzini <address@hidden>
Reviewed-by: Fam Zheng <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: bae8196d9f97916de6323e70e3e374362ee16ec4
      
https://github.com/qemu/qemu/commit/bae8196d9f97916de6323e70e3e374362ee16ec4
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M block/backup.c
    M block/mirror.c
    M blockjob.c
    M include/block/blockjob.h

  Log Message:
  -----------
  blockjob: introduce .drain callback for jobs

This is required to decouple block jobs from running in an
AioContext.  With multiqueue block devices, a BlockDriverState
does not really belong to a single AioContext.

The solution is to first wait until all I/O operations are
complete; then loop in the main thread for the block job to
complete entirely.

Signed-off-by: Paolo Bonzini <address@hidden>
Reviewed-by: Fam Zheng <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: 9a0cec664eefed475a5954006e29f2fd7cd29d2a
      
https://github.com/qemu/qemu/commit/9a0cec664eefed475a5954006e29f2fd7cd29d2a
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M block/mirror.c

  Log Message:
  -----------
  mirror: use bdrv_drained_begin/bdrv_drained_end

Ensure that there are no changes between the last check to
bdrv_get_dirty_count and the switch to the target.

There is already a bdrv_drained_end call, we only need to ensure
that bdrv_drained_begin is not called twice.

Signed-off-by: Paolo Bonzini <address@hidden>
Reviewed-by: Fam Zheng <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: 99723548561978da8ef44cf804fb7912698f5d88
      
https://github.com/qemu/qemu/commit/99723548561978da8ef44cf804fb7912698f5d88
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M block/block-backend.c
    M block/io.c
    M include/block/block_int.h

  Log Message:
  -----------
  block: add BDS field to count in-flight requests

Unlike tracked_requests, this field also counts throttled requests,
and remains non-zero if an AIO operation needs a BH to be "really"
completed.

With this change, it is no longer necessary to have a dummy
BdrvTrackedRequest for requests that are never serialising, and
it is no longer necessary to poll the AioContext once after
bdrv_requests_pending(bs) returns false.

Signed-off-by: Paolo Bonzini <address@hidden>
Reviewed-by: Fam Zheng <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: d42cf28837801cd1f835089fe9db2a42a1af55cd
      
https://github.com/qemu/qemu/commit/d42cf28837801cd1f835089fe9db2a42a1af55cd
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M block/io.c

  Log Message:
  -----------
  block: change drain to look only at one child at a time

bdrv_requests_pending is checking children to also wait until internal
requests (such as metadata writes) have completed.  However, checking
children is in general overkill.  Children requests can be of two kinds:

- requests caused by an operation on bs, e.g. a bdrv_aio_write to bs
causing a write to bs->file->bs.  In this case, the parent's in_flight
count will always be incremented by at least one for every request in
the child.

- asynchronous metadata writes or flushes.  Such writes can be started
even if bs's in_flight count is zero, but not after the .bdrv_drain
callback has been invoked.

This patch therefore changes bdrv_drain to finish I/O in the parent
(after which the parent's in_flight will be locked to zero), call
bdrv_drain (after which the parent will not generate I/O on the child
anymore), and then wait for internal I/O in the children to complete.

Signed-off-by: Paolo Bonzini <address@hidden>
Reviewed-by: Fam Zheng <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: 6653a73d12fbac8c9b5c22bb21aef22759b28c1c
      
https://github.com/qemu/qemu/commit/6653a73d12fbac8c9b5c22bb21aef22759b28c1c
  Author: Fam Zheng <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M block/qed.c

  Log Message:
  -----------
  qed: Implement .bdrv_drain

The "need_check_timer" is used to clear the "NEED_CHECK" flag in the
image header after a grace period once metadata update has finished. To
comply with the bdrv_drain semantics, we should make sure it remains
deleted once .bdrv_drain is called.

The change to qed_need_check_timer_cb is needed because bdrv_qed_drain
is called after s->bs has been drained, and should not operate on it;
instead it should operate on the BdrvChild-ren exclusively.  Doing so
is easy because QED does not have a bdrv_co_flush_to_os callback, hence
all that is needed to flush it is to ensure writes have reached the disk.

Based on commit df9a681dc9a (which however included some unrelated
hunks, possibly due to a merge failure or an overlooked squash).
The patch was reverted because at the time bdrv_qed_drain could call
qed_plug_allocating_write_reqs while an allocating write was queued.
This however is not possible anymore after the previous patch;
.bdrv_drain is only called after all writes have completed at the
QED level, and its purpose is to trigger metadata writes in bs->file.

Signed-off-by: Fam Zheng <address@hidden>
Signed-off-by: Paolo Bonzini <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: 88b062c2036cfd05b5111147736a08ba05ea05a9
      
https://github.com/qemu/qemu/commit/88b062c2036cfd05b5111147736a08ba05ea05a9
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M block/block-backend.c
    M block/io.c
    M block/qed-table.c
    M include/block/block.h

  Log Message:
  -----------
  block: introduce BDRV_POLL_WHILE

We want the BDS event loop to run exclusively in the iothread that
owns the BDS's AioContext.  This macro will provide the synchronization
between the two event loops; for now it just wraps the common idiom
of a while loop around aio_poll.

Signed-off-by: Paolo Bonzini <address@hidden>
Reviewed-by: Fam Zheng <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: aa92d6c4609e174fc6884e4b7b87367fac33cbe9
      
https://github.com/qemu/qemu/commit/aa92d6c4609e174fc6884e4b7b87367fac33cbe9
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M block/nfs.c

  Log Message:
  -----------
  nfs: move nfs_set_events out of the while loops

nfs_set_events only needs to be called once before entering the
while loop; afterwards, nfs_process_read and nfs_process_write
take care of it.

Reviewed-by: Stefan Hajnoczi <address@hidden>
Reviewed-by: Fam Zheng <address@hidden>
Signed-off-by: Paolo Bonzini <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: d746427aafa5abf7d607d974979c1d15c8ab3c8d
      
https://github.com/qemu/qemu/commit/d746427aafa5abf7d607d974979c1d15c8ab3c8d
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M block/nfs.c

  Log Message:
  -----------
  nfs: use BDRV_POLL_WHILE

This will make it possible to use nfs_get_allocated_file_size on
a file that is not in the main AioContext.

Reviewed-by: Stefan Hajnoczi <address@hidden>
Reviewed-by: Fam Zheng <address@hidden>
Signed-off-by: Paolo Bonzini <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: f11672dbc0207cfb0015a2d08e7693b957847cbe
      
https://github.com/qemu/qemu/commit/f11672dbc0207cfb0015a2d08e7693b957847cbe
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M block/sheepdog.c

  Log Message:
  -----------
  sheepdog: use BDRV_POLL_WHILE

This is important when the sheepdog driver works on a BlockDriverState
that is attached to an I/O thread other than the main thread.

Signed-off-by: Paolo Bonzini <address@hidden>
Reviewed-by: Fam Zheng <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: e437016511edb0dfa13cc98587c39590eaa2609a
      
https://github.com/qemu/qemu/commit/e437016511edb0dfa13cc98587c39590eaa2609a
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M include/block/aio.h
    M iothread.c
    M stubs/Makefile.objs
    A stubs/iothread.c

  Log Message:
  -----------
  aio: introduce qemu_get_current_aio_context

This will be used by BDRV_POLL_WHILE (and thus by bdrv_drain)
to choose how to wait for I/O completion.

Reviewed-by: Fam Zheng <address@hidden>
Signed-off-by: Paolo Bonzini <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: d16341fa6998c530db17cbb7d5c781913e674255
      
https://github.com/qemu/qemu/commit/d16341fa6998c530db17cbb7d5c781913e674255
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M iothread.c

  Log Message:
  -----------
  iothread: detach all block devices before stopping them

Soon bdrv_drain will not call aio_poll itself on iothreads.  If block
devices are left hanging off the iothread's AioContext, there will be no
one to do I/O for those poor devices.

Signed-off-by: Paolo Bonzini <address@hidden>
Reviewed-by: Fam Zheng <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: 8dd9006e9b2869b1b4ae3a6f6e9acf249ce5f2f9
      
https://github.com/qemu/qemu/commit/8dd9006e9b2869b1b4ae3a6f6e9acf249ce5f2f9
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M block/replication.c

  Log Message:
  -----------
  replication: pass BlockDriverState to reopen_backing_file

This will be needed in the next patch to retrieve the AioContext.

Reviewed-by: Stefan Hajnoczi <address@hidden>
Reviewed-by: Fam Zheng <address@hidden>
Signed-off-by: Paolo Bonzini <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: 720150f3189648712fc50efd1cca0a39076722a9
      
https://github.com/qemu/qemu/commit/720150f3189648712fc50efd1cca0a39076722a9
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M block.c
    M block/commit.c
    M block/replication.c
    M include/block/block.h
    M qemu-io-cmds.c

  Log Message:
  -----------
  block: prepare bdrv_reopen_multiple to release AioContext

After the next patch bdrv_drain_all will have to be called without holding any
AioContext.  Prepare to do this by adding an AioContext argument to
bdrv_reopen_multiple.

Reviewed-by: Stefan Hajnoczi <address@hidden>
Reviewed-by: Fam Zheng <address@hidden>
Signed-off-by: Paolo Bonzini <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: 15afd94a0470dcace6c73d7187a0602188bfd21c
      
https://github.com/qemu/qemu/commit/15afd94a0470dcace6c73d7187a0602188bfd21c
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M qemu-io-cmds.c

  Log Message:
  -----------
  qemu-io: acquire AioContext

Signed-off-by: Paolo Bonzini <address@hidden>
Reviewed-by: Fam Zheng <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: 9e944cb4744b527dd93aa989023739375a8880fb
      
https://github.com/qemu/qemu/commit/9e944cb4744b527dd93aa989023739375a8880fb
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M qemu-img.c

  Log Message:
  -----------
  qemu-img: call aio_context_acquire/release around block job

This will be needed by bdrv_reopen_multiple, which calls
bdrv_drain_all and thus will *release* the AioContext.

Signed-off-by: Paolo Bonzini <address@hidden>
Reviewed-by: Fam Zheng <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: c9d1a56174339b0afdef63b7d151b38f4bb6dae5
      
https://github.com/qemu/qemu/commit/c9d1a56174339b0afdef63b7d151b38f4bb6dae5
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M async.c
    M block.c
    M block/io.c
    M block/nfs.c
    M block/sheepdog.c
    M hw/scsi/virtio-scsi-dataplane.c
    M include/block/block.h
    M include/block/block_int.h

  Log Message:
  -----------
  block: only call aio_poll on the current thread's AioContext

aio_poll is not thread safe; for example bdrv_drain can hang if
the last in-flight I/O operation is completed in the I/O thread after
the main thread has checked bs->in_flight.

The bug remains latent as long as all of it is called within
aio_context_acquire/aio_context_release, but this will change soon.

To fix this, if bdrv_drain is called from outside the I/O thread,
signal the main AioContext through a dummy bottom half.  The event
loop then only runs in the I/O thread.

Reviewed-by: Fam Zheng <address@hidden>
Signed-off-by: Paolo Bonzini <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: 65c1b5b622d09044af3f872240d65e7343f6f83e
      
https://github.com/qemu/qemu/commit/65c1b5b622d09044af3f872240d65e7343f6f83e
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M async.c
    M docs/multiple-iothreads.txt
    M include/block/aio.h
    M iothread.c
    M tests/test-aio.c

  Log Message:
  -----------
  iothread: release AioContext around aio_poll

This is the first step towards having fine-grained critical sections in
dataplane threads, which will resolve lock ordering problems between
address_space_* functions (which need the BQL when doing MMIO, even
after we complete RCU-based dispatch) and the AioContext.

Because AioContext does not use contention callbacks anymore, the
unit test has to be changed.

Previously applied as a0710f7995f914e3044e5899bd8ff6c43c62f916 and
then reverted.

Reviewed-by: Fam Zheng <address@hidden>
Signed-off-by: Paolo Bonzini <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: feadec63846d569829a4302486025b2915228712
      
https://github.com/qemu/qemu/commit/feadec63846d569829a4302486025b2915228712
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M include/qemu/thread-posix.h
    M include/qemu/thread-win32.h
    M include/qemu/thread.h
    M util/qemu-thread-posix.c
    M util/qemu-thread-win32.c

  Log Message:
  -----------
  qemu-thread: introduce QemuRecMutex

GRecMutex is new in glib 2.32, so we cannot use it.  Introduce
a recursive mutex in qemu-thread instead, which will be used
instead of RFifoLock.

Reviewed-by: Fam Zheng <address@hidden>
Reviewed-by: Stefan Hajnoczi <address@hidden>
Signed-off-by: Paolo Bonzini <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: 3fe71223374e71436d4aced8865e50fd36588ff7
      
https://github.com/qemu/qemu/commit/3fe71223374e71436d4aced8865e50fd36588ff7
  Author: Paolo Bonzini <address@hidden>
  Date:   2016-10-28 (Fri, 28 Oct 2016)

  Changed paths:
    M async.c
    M include/block/aio.h
    R include/qemu/rfifolock.h
    M tests/.gitignore
    M tests/Makefile.include
    R tests/test-rfifolock.c
    M util/Makefile.objs
    R util/rfifolock.c

  Log Message:
  -----------
  aio: convert from RFifoLock to QemuRecMutex

It is simpler and a bit faster, and QEMU does not need the contention
callbacks (and thus the fairness) anymore.

Reviewed-by: Fam Zheng <address@hidden>
Signed-off-by: Paolo Bonzini <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>


  Commit: 5273a45e7521a45b27447fe6e4510ef43ff2fa67
      
https://github.com/qemu/qemu/commit/5273a45e7521a45b27447fe6e4510ef43ff2fa67
  Author: Peter Maydell <address@hidden>
  Date:   2016-10-31 (Mon, 31 Oct 2016)

  Changed paths:
    M async.c
    M block.c
    M block/backup.c
    M block/block-backend.c
    M block/commit.c
    M block/io.c
    M block/mirror.c
    M block/nfs.c
    M block/qed-table.c
    M block/qed.c
    M block/replication.c
    M block/sheepdog.c
    M blockjob.c
    M docs/multiple-iothreads.txt
    M hw/scsi/virtio-scsi-dataplane.c
    M include/block/aio.h
    M include/block/block.h
    M include/block/block_int.h
    M include/block/blockjob.h
    R include/qemu/rfifolock.h
    M include/qemu/thread-posix.h
    M include/qemu/thread-win32.h
    M include/qemu/thread.h
    M iothread.c
    M qemu-img.c
    M qemu-io-cmds.c
    M stubs/Makefile.objs
    A stubs/iothread.c
    M tests/.gitignore
    M tests/Makefile.include
    M tests/test-aio.c
    R tests/test-rfifolock.c
    M util/Makefile.objs
    M util/qemu-thread-posix.c
    M util/qemu-thread-win32.c
    R util/rfifolock.c

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/famz/tags/for-upstream' into staging

# gpg: Signature made Fri 28 Oct 2016 15:47:39 BST
# gpg:                using RSA key 0xCA35624C6A9171C6
# gpg: Good signature from "Fam Zheng <address@hidden>"
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg:          There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 5003 7CB7 9706 0F76 F021  AD56 CA35 624C 6A91 71C6

* remotes/famz/tags/for-upstream:
  aio: convert from RFifoLock to QemuRecMutex
  qemu-thread: introduce QemuRecMutex
  iothread: release AioContext around aio_poll
  block: only call aio_poll on the current thread's AioContext
  qemu-img: call aio_context_acquire/release around block job
  qemu-io: acquire AioContext
  block: prepare bdrv_reopen_multiple to release AioContext
  replication: pass BlockDriverState to reopen_backing_file
  iothread: detach all block devices before stopping them
  aio: introduce qemu_get_current_aio_context
  sheepdog: use BDRV_POLL_WHILE
  nfs: use BDRV_POLL_WHILE
  nfs: move nfs_set_events out of the while loops
  block: introduce BDRV_POLL_WHILE
  qed: Implement .bdrv_drain
  block: change drain to look only at one child at a time
  block: add BDS field to count in-flight requests
  mirror: use bdrv_drained_begin/bdrv_drained_end
  blockjob: introduce .drain callback for jobs
  replication: interrupt failover if the main device is closed

Signed-off-by: Peter Maydell <address@hidden>


Compare: https://github.com/qemu/qemu/compare/2dfe5113b11c...5273a45e7521