qemu-commits
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Qemu-commits] [qemu/qemu] 2d9187: qemu-img: make convert async


From: GitHub
Subject: [Qemu-commits] [qemu/qemu] 2d9187: qemu-img: make convert async
Date: Thu, 02 Mar 2017 00:45:12 -0800

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 2d9187bc65727d9dd63e2c410b5500add3db0b0d
      
https://github.com/qemu/qemu/commit/2d9187bc65727d9dd63e2c410b5500add3db0b0d
  Author: Peter Lieven <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M qemu-img-cmds.hx
    M qemu-img.c
    M qemu-img.texi

  Log Message:
  -----------
  qemu-img: make convert async

the convert process is currently completely implemented with sync operations.
That means it reads one buffer and then writes it. No parallelism and each sync
request takes as long as it takes until it is completed.

This can be a big performance hit when the convert process reads and writes
to devices which do not benefit from kernel readahead or pagecache.
In our environment we heavily have the following two use cases when using
qemu-img convert.

a) reading from NFS and writing to iSCSI for deploying templates
b) reading from iSCSI and writing to NFS for backups

In both processes we use libiscsi and libnfs so we have no kernel cache.

This patch changes the convert process to work with parallel running coroutines
which can significantly improve performance for network storage devices:

qemu-img (master)
 nfs -> iscsi 22.8 secs
 nfs -> ram   11.7 secs
 ram -> iscsi 12.3 secs

qemu-img-async (8 coroutines, in-order write disabled)
 nfs -> iscsi 11.0 secs
 nfs -> ram   10.4 secs
 ram -> iscsi  9.0 secs

This patches introduces 2 new cmdline parameters. The -m parameter to specify
the number of coroutines running in parallel (defaults to 8). And the -W 
parameter to
allow qemu-img to write to the target out of order rather than sequential. This 
improves
performance as the writes do not have to wait for each other to complete.

Signed-off-by: Peter Lieven <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 9e19ad4e49c8dc7f776bf770f52ad6ea1ec28edc
      
https://github.com/qemu/qemu/commit/9e19ad4e49c8dc7f776bf770f52ad6ea1ec28edc
  Author: Markus Armbruster <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M tests/qemu-iotests/049.out
    M util/qemu-option.c

  Log Message:
  -----------
  option: Tweak invalid size error message and unbreak iotest 049

Commit 75cdcd1 neglected to update tests/qemu-iotests/049.out, and
made the error message for negative size worse.  Fix that.

Reported-by: Thomas Huth <address@hidden>
Signed-off-by: Markus Armbruster <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Reviewed-by: Thomas Huth <address@hidden>
Tested-by: Christian Borntraeger <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 7006c9a76119a863e5812de572841618b9540ac2
      
https://github.com/qemu/qemu/commit/7006c9a76119a863e5812de572841618b9540ac2
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M include/block/block.h

  Log Message:
  -----------
  block: Add op blocker permission constants

This patch defines the permission categories that will be used by the
new op blocker system.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 8b2ff5291f9e39fb1c0c6c0c4321daac60aab4db
      
https://github.com/qemu/qemu/commit/8b2ff5291f9e39fb1c0c6c0c4321daac60aab4db
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c
    M block/quorum.c
    M include/block/block.h

  Log Message:
  -----------
  block: Add Error argument to bdrv_attach_child()

It will have to return an error soon, so prepare the callers for it.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: d5e6f437c5508614803d11e59ee16a758dde09ef
      
https://github.com/qemu/qemu/commit/d5e6f437c5508614803d11e59ee16a758dde09ef
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c
    M block/block-backend.c
    M include/block/block_int.h

  Log Message:
  -----------
  block: Let callers request permissions when attaching a child node

When attaching a node as a child to a new parent, the required and
shared permissions for this parent are checked against all other parents
of the node now, and an error is returned if there is a conflict.

This allows error returns to a function that previously always
succeeded, and the same is true for quite a few callers and their
callers. Converting all of them within the same patch would be too much,
so for now everyone tells that they don't need any permissions and allow
everyone else to do anything. This way we can use &error_abort initially
and convert caller by caller to pass actual permission requirements and
implement error handling.

All these places are marked with FIXME comments and it will be the job
of the next patches to clean them up again.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 33a610c398603efafd954c706ba07850835a5098
      
https://github.com/qemu/qemu/commit/33a610c398603efafd954c706ba07850835a5098
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c
    M include/block/block_int.h

  Log Message:
  -----------
  block: Involve block drivers in permission granting

In many cases, the required permissions of one node on its children
depend on what its parents require from it. For example, the raw format
or most filter drivers only need to request consistent reads if that's
something that one of their parents wants.

In order to achieve this, this patch introduces two new BlockDriver
callbacks. The first one lets drivers first check (recursively) whether
the requested permissions can be set; the second one actually sets the
new permission bitmask.

Also add helper functions that drivers can use in their implementation
of the callbacks to update their permissions on a specific child.

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: 6a1b9ee152101868082dbf24ccb683af0431c85b
      
https://github.com/qemu/qemu/commit/6a1b9ee152101868082dbf24ccb683af0431c85b
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c
    M include/block/block_int.h

  Log Message:
  -----------
  block: Default .bdrv_child_perm() for filter drivers

Most filters need permissions related to read and write for their
children, but only if the node has a parent that wants to use the same
operation on the filter. The same is true for resize.

This adds a default implementation that simply forwards all necessary
permissions to all children of the node and leaves the other permissions
unchanged.

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: d7010dfb685506cc4addd102627205f85fcd0fe7
      
https://github.com/qemu/qemu/commit/d7010dfb685506cc4addd102627205f85fcd0fe7
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/blkdebug.c
    M block/blkreplay.c
    M block/blkverify.c
    M block/quorum.c
    M block/raw-format.c
    M block/replication.c

  Log Message:
  -----------
  block: Request child permissions in filter drivers

All callers will have to request permissions for all of their child
nodes. Block drivers that act as simply filters can use the default
implementation of .bdrv_child_perm().

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 6b1a044afb363f03b51c9d0218bef44a34ea98c3
      
https://github.com/qemu/qemu/commit/6b1a044afb363f03b51c9d0218bef44a34ea98c3
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c
    M include/block/block_int.h

  Log Message:
  -----------
  block: Default .bdrv_child_perm() for format drivers

Almost all format drivers have the same characteristics as far as
permissions are concerned: They have one or more children for storing
their own data and, more importantly, metadata (can be written to and
grow even without external write requests, must be protected against
other writers and present consistent data) and optionally a backing file
(this is just data, so like for a filter, it only depends on what the
parent nodes need).

This provides a default implementation that can be shared by most of
our format drivers.

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: 862f215fabf31c80c953155fcb223fea5320bbdf
      
https://github.com/qemu/qemu/commit/862f215fabf31c80c953155fcb223fea5320bbdf
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/bochs.c
    M block/cloop.c
    M block/crypto.c
    M block/dmg.c
    M block/parallels.c
    M block/qcow.c
    M block/qcow2.c
    M block/qed.c
    M block/vdi.c
    M block/vhdx.c
    M block/vmdk.c
    M block/vpc.c

  Log Message:
  -----------
  block: Request child permissions in format drivers

This makes use of the .bdrv_child_perm() implementation for formats that
we just added. All format drivers expose the permissions they actually
need nows, so that they can be set accordingly and updated when parents
are attached or detached.

The only format not included here is raw, which was already converted
with the other filter drivers.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 91ef38257ad225f7fa17a6583fb792c0be9e8acf
      
https://github.com/qemu/qemu/commit/91ef38257ad225f7fa17a6583fb792c0be9e8acf
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c
    M block/vvfat.c
    M include/block/block_int.h

  Log Message:
  -----------
  vvfat: Implement .bdrv_child_perm()

vvfat is the last remaining driver that can have children, but doesn't
implement .bdrv_child_perm() yet. The default handlers aren't suitable
here, so let's implement a very simple driver-specific one that protects
the internal child from being used by other users as good as our
permissions permit.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 78e421c9fbed9d501d7ada84ddc786a92178a71d
      
https://github.com/qemu/qemu/commit/78e421c9fbed9d501d7ada84ddc786a92178a71d
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c

  Log Message:
  -----------
  block: Require .bdrv_child_perm() with child nodes

All block drivers that can have child nodes implement .bdrv_child_perm()
now. Make this officially a requirement by asserting that only drivers
without children can omit .bdrv_child_perm().

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: f68c598be6a48995ca4c7cc42fc1f6e1195ec7aa
      
https://github.com/qemu/qemu/commit/f68c598be6a48995ca4c7cc42fc1f6e1195ec7aa
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c

  Log Message:
  -----------
  block: Request real permissions in bdrv_attach_child()

Now that all block drivers with children tell us what permissions they
need from each of their children, bdrv_attach_child() can use this
information and make the right requirements while trying to attach new
children.

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: 981776b34875ef75b218a338e1831b8fc65ea6bd
      
https://github.com/qemu/qemu/commit/981776b34875ef75b218a338e1831b8fc65ea6bd
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/block-backend.c
    M include/sysemu/block-backend.h

  Log Message:
  -----------
  block: Add permissions to BlockBackend

The BlockBackend can now store the permissions that its user requires.
This is necessary because nodes can be ejected from or inserted into a
BlockBackend and all of these operations must make sure that the user
still gets what it requested initially.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 6d0eb64d5c6d57017c52a4f36ccae1db79215ee1
      
https://github.com/qemu/qemu/commit/6d0eb64d5c6d57017c52a4f36ccae1db79215ee1
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c
    M block/backup.c
    M block/block-backend.c
    M block/commit.c
    M block/mirror.c
    M block/qcow2.c
    M blockdev.c
    M blockjob.c
    M hmp.c
    M hw/block/fdc.c
    M hw/core/qdev-properties-system.c
    M hw/ide/qdev.c
    M hw/scsi/scsi-disk.c
    M include/sysemu/block-backend.h
    M migration/block.c
    M nbd/server.c
    M tests/test-blockjob.c
    M tests/test-throttle.c

  Log Message:
  -----------
  block: Add permissions to blk_new()

We want every user to be specific about the permissions it needs, so
we'll pass the initial permissions as parameters to blk_new(). A user
only needs to call blk_set_perm() if it wants to change the permissions
after the fact.

The permissions are stored in the BlockBackend and applied whenever a
BlockDriverState should be attached in blk_insert_bs().

This does not include actually choosing the right set of permissions
everywhere yet. Instead, the usual FIXME comment is added to each place
and will be addressed in individual patches.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: d7086422b1c1e75e320519cfe26176db6ec97a37
      
https://github.com/qemu/qemu/commit/d7086422b1c1e75e320519cfe26176db6ec97a37
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c
    M block/backup.c
    M block/block-backend.c
    M block/commit.c
    M block/mirror.c
    M block/qcow2.c
    M blockdev.c
    M blockjob.c
    M hmp.c
    M hw/core/qdev-properties-system.c
    M include/sysemu/block-backend.h
    M migration/block.c
    M nbd/server.c
    M tests/test-blockjob.c

  Log Message:
  -----------
  block: Add error parameter to blk_insert_bs()

Now that blk_insert_bs() requests the BlockBackend permissions for the
node it attaches to, it can fail. Instead of aborting, pass the errors
to the callers.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 55880601d82d55cbfa3b5bd9757496b6ebbc527c
      
https://github.com/qemu/qemu/commit/55880601d82d55cbfa3b5bd9757496b6ebbc527c
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/parallels.c
    M block/qcow.c
    M block/qcow2.c
    M block/qed.c
    M block/sheepdog.c
    M block/vdi.c
    M block/vhdx.c
    M block/vmdk.c
    M block/vpc.c
    M include/block/block.h
    M qemu-img.c

  Log Message:
  -----------
  block: Add BDRV_O_RESIZE for blk_new_open()

blk_new_open() is a convenience function that processes flags rather
than QDict options as a simple way to just open an image file.

In order to keep it convenient in the future, it must automatically
request the necessary permissions. This can easily be inferred from the
flags for read and write, but we need another flag that tells us whether
to get the resize permission.

We can't just always request it because that means that no block jobs
can run on the resulting BlockBackend (which is something that e.g.
qemu-img commit wants to do), but we also can't request it never because
most of the .bdrv_create() implementations call blk_truncate().

The solution is to introduce another flag that is passed by all users
that want to resize the image.

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: c62d32f503b37322a3960bad4cd4cdb69947d81e
      
https://github.com/qemu/qemu/commit/c62d32f503b37322a3960bad4cd4cdb69947d81e
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/block-backend.c

  Log Message:
  -----------
  block: Request real permissions in blk_new_open()

We can figure out the necessary permissions from the flags that the
caller passed.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 39829a01ae524788c68dc0794e6912faa898eb75
      
https://github.com/qemu/qemu/commit/39829a01ae524788c68dc0794e6912faa898eb75
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/block-backend.c
    M blockdev.c
    M hw/block/fdc.c
    M hw/ide/core.c
    M hw/scsi/scsi-disk.c
    M hw/sd/sd.c
    M include/block/block_int.h
    M include/sysemu/block-backend.h

  Log Message:
  -----------
  block: Allow error return in BlockDevOps.change_media_cb()

Some devices allow a media change between read-only and read-write
media. They need to adapt the permissions in their .change_media_cb()
implementation, which can fail. So add an Error parameter to the
function.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: a17c17a274f24f0c0259f89d288f29b8ce0511aa
      
https://github.com/qemu/qemu/commit/a17c17a274f24f0c0259f89d288f29b8ce0511aa
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M hw/block/block.c
    M hw/block/fdc.c
    M hw/block/m25p80.c
    M hw/block/nand.c
    M hw/block/nvme.c
    M hw/block/onenand.c
    M hw/block/pflash_cfi01.c
    M hw/block/pflash_cfi02.c
    M hw/block/virtio-blk.c
    M hw/core/qdev-properties-system.c
    M hw/ide/qdev.c
    M hw/nvram/spapr_nvram.c
    M hw/scsi/scsi-disk.c
    M hw/sd/sd.c
    M hw/usb/dev-storage.c
    M include/hw/block/block.h
    M tests/qemu-iotests/051.pc.out

  Log Message:
  -----------
  hw/block: Request permissions

This makes all device emulations with a qdev drive property request
permissions on their BlockBackend. The only thing we block at this point
is resizing images for some devices that can't support it.

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: dabd18f64c8800d441fd9fb232c2102e8409aa2e
      
https://github.com/qemu/qemu/commit/dabd18f64c8800d441fd9fb232c2102e8409aa2e
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M hw/block/block.c
    M include/hw/block/block.h
    M tests/qemu-iotests/172.out

  Log Message:
  -----------
  hw/block: Introduce share-rw qdev property

By default, don't allow another writer for block devices that are
attached to a guest device. For the cases where this setup is intended
(e.g. using a cluster filesystem on the disk), the new option can be
used to allow it.

This change affects only devices using DEFINE_BLOCK_PROPERTIES().
Devices directly using DEFINE_PROP_DRIVE() still accept writers
unconditionally.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: c6cc12bfa7bb9c61f4fa20491258b9bebc5b4771
      
https://github.com/qemu/qemu/commit/c6cc12bfa7bb9c61f4fa20491258b9bebc5b4771
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/backup.c
    M block/commit.c
    M block/mirror.c
    M block/stream.c
    M blockjob.c
    M include/block/blockjob_int.h
    M tests/test-blockjob-txn.c
    M tests/test-blockjob.c

  Log Message:
  -----------
  blockjob: Add permissions to block_job_create()

This functions creates a BlockBackend internally, so the block jobs need
to tell it what they want to do with the BB.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: b5411555877121b74cbe0a12a867d16d861746b1
      
https://github.com/qemu/qemu/commit/b5411555877121b74cbe0a12a867d16d861746b1
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c
    M block/block-backend.c
    M include/block/block_int.h

  Log Message:
  -----------
  block: Add BdrvChildRole.get_parent_desc()

For meaningful error messages in the permission system, we need to get
some human-readable description of the parent of a BdrvChild.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: d083319fe007e100b38995d0ea254845c8efa433
      
https://github.com/qemu/qemu/commit/d083319fe007e100b38995d0ea254845c8efa433
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c

  Log Message:
  -----------
  block: Include details on permission errors in message

Instead of just telling that there was some conflict, we can be specific
and tell which permissions were in conflict and which way the conflict
is.

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: 26de9438c1b6013532fb95de0720e2696588332f
      
https://github.com/qemu/qemu/commit/26de9438c1b6013532fb95de0720e2696588332f
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c
    M include/block/block_int.h

  Log Message:
  -----------
  block: Add BdrvChildRole.stay_at_node

When the parents' child links are updated in bdrv_append() or
bdrv_replace_in_backing_chain(), this should affect all child links of
BlockBackends or other nodes, but not on child links held for other
purposes (like for setting permissions). This patch allows to control
the behaviour per BdrvChildRole.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 76d554e20bd0a965ac22d6155a129be12fac2667
      
https://github.com/qemu/qemu/commit/76d554e20bd0a965ac22d6155a129be12fac2667
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/backup.c
    M block/commit.c
    M block/mirror.c
    M block/stream.c
    M blockjob.c
    M include/block/blockjob.h

  Log Message:
  -----------
  blockjob: Add permissions to block_job_add_bdrv()

Block jobs don't actually do I/O through the the reference they create
with block_job_add_bdrv(), but they might want to use the permisssion
system to express what the block job does to intermediate nodes. This
adds permissions to block_job_add_bdrv() to provide the means to request
permissions.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 8dfba2797761d8a43744e4e6571c8175e448a478
      
https://github.com/qemu/qemu/commit/8dfba2797761d8a43744e4e6571c8175e448a478
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/commit.c

  Log Message:
  -----------
  commit: Use real permissions in commit block job

This is probably one of the most interesting conversions to the new
op blocker system because a commit block job intentionally leaves some
intermediate block nodes in the backing chain that aren't valid on their
own any more; only the whole chain together results in a valid view.

In order to provide the 'consistent read' permission to the parents of
the 'top' node of the commit job, a new filter block driver is inserted
above 'top' which doesn't require 'consistent read' on its backing
chain. Subsequently, the commit job can block 'consistent read' on all
intermediate nodes without causing a conflict.

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: d3f06759222e70979965e3afccfd88eddd3cc454
      
https://github.com/qemu/qemu/commit/d3f06759222e70979965e3afccfd88eddd3cc454
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/commit.c

  Log Message:
  -----------
  commit: Use real permissions for HMP 'commit'

This is a little simpler than the commit block job because it's
synchronous and only commits into the immediate backing file, but
otherwise doing more or less the same.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 4e9e4323d5ec07a07f8db9317e1842a5e00a14e2
      
https://github.com/qemu/qemu/commit/4e9e4323d5ec07a07f8db9317e1842a5e00a14e2
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/backup.c
    M tests/qemu-iotests/055

  Log Message:
  -----------
  backup: Use real permissions in backup block job

The backup block job doesn't have very complicated requirements: It
needs to read from the source and write to the target, but it's fine
with either side being changed. The only restriction is that we can't
resize the image because the job uses a cached value.

qemu-iotests 055 needs to be changed because it used a target which was
already attached to a virtio-blk device. The permission system correctly
forbids this (virtio-blk can't accept another writer with its default
share-rw=off).

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: dd65a52e4aa4a0adfedf0ed9a35da1960f359fe1
      
https://github.com/qemu/qemu/commit/dd65a52e4aa4a0adfedf0ed9a35da1960f359fe1
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c

  Log Message:
  -----------
  block: Fix pending requests check in bdrv_append()

bdrv_append() cares about isolation of the node that it modifies, but
not about activity in some subtree below it. Instead of using the
recursive bdrv_requests_pending(), directly check bs->in_flight, which
considers only the node in question.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: db95dbba3b7b09cd11ffaf3a8453c2500e807f80
      
https://github.com/qemu/qemu/commit/db95dbba3b7b09cd11ffaf3a8453c2500e807f80
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c
    M include/block/block_int.h

  Log Message:
  -----------
  block: BdrvChildRole.attach/detach() callbacks

Backing files are somewhat special compared to other kinds of children
because they are attached and detached using bdrv_set_backing_hd()
rather than the normal set of functions, which does a few more things
like setting backing blockers, toggling the BDRV_O_NO_BACKING flag,
setting parent_bs->backing_file, etc.

These special features are a reason why change_parent_backing_link()
can't handle backing files yet. With abstracting the additional features
into .attach/.detach callbacks, we get a step closer to a function that
can actually deal with this.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 3e44c8e08a4b84ec1f4f1eb249d33005bb9cf572
      
https://github.com/qemu/qemu/commit/3e44c8e08a4b84ec1f4f1eb249d33005bb9cf572
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c

  Log Message:
  -----------
  block: Allow backing file links in change_parent_backing_link()

Now that the backing file child role implements .attach/.detach
callbacks, nothing prevents us from modifying the graph even if that
involves changing backing file links.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: bbc02b90bcba371818dbffec89933072f9406945
      
https://github.com/qemu/qemu/commit/bbc02b90bcba371818dbffec89933072f9406945
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M blockjob.c
    M include/block/blockjob.h

  Log Message:
  -----------
  blockjob: Factor out block_job_remove_all_bdrv()

In some cases, we want to remove op blockers on intermediate nodes
before the whole block job transaction has completed (because they block
restoring the final graph state during completion). Provide a function
for this.

The whole block job lifecycle is a bit messed up and it's hard to
actually do all things in the right order, but I'll leave simplifying
this for another day.

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: 4ef85a9c233936d9ac1400d67b066353a6e9f32f
      
https://github.com/qemu/qemu/commit/4ef85a9c233936d9ac1400d67b066353a6e9f32f
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/mirror.c
    M qemu-img.c
    M tests/qemu-iotests/141
    M tests/qemu-iotests/141.out

  Log Message:
  -----------
  mirror: Use real permissions in mirror/active commit block job

The mirror block job is mainly used for two different scenarios:
Mirroring to an otherwise unused, independent target node, or for active
commit where the target node is part of the backing chain of the source.

Similarly to the commit block job patch, we need to insert a new filter
node to keep the permissions correct during active commit.

Note that one change this implies is that job->blk points to
mirror_top_bs as its root now, and mirror_top_bs (rather than the actual
source node) contains the bs->job pointer. This requires qemu-img commit
to get the job by name now rather than just taking bs->job.

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Acked-by: Max Reitz <address@hidden>


  Commit: a170a91fd3eab6155da39e740381867e80bcc93e
      
https://github.com/qemu/qemu/commit/a170a91fd3eab6155da39e740381867e80bcc93e
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/stream.c

  Log Message:
  -----------
  stream: Use real permissions in streaming block job

The correct permissions are relatively obvious here (and explained in
code comments). For intermediate streaming, we need to reopen the top
node read-write before creating the job now because the permissions
system catches attempts to get the BLK_PERM_WRITE_UNCHANGED permission
on a read-only node.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 6cdbceb12cf955398df48eda94a45ca41e956c78
      
https://github.com/qemu/qemu/commit/6cdbceb12cf955398df48eda94a45ca41e956c78
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/mirror.c
    M blockdev.c
    M include/block/block_int.h
    M qapi/block-core.json

  Log Message:
  -----------
  mirror: Add filter-node-name to blockdev-mirror

Management tools need to be able to know about every node in the graph
and need a way to address them. Changing the graph structure was okay
because libvirt doesn't really manage the node level yet, but future
libvirt versions need to deal with both new and old version of qemu.

This new option to blockdev-mirror allows the client to set a node-name
for the automatically inserted filter driver, and at the same time
serves as a witness for a future libvirt that this version of qemu does
automatically insert a filter driver.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 0db832f42e445398b2815cd740e9cd915e7dd644
      
https://github.com/qemu/qemu/commit/0db832f42e445398b2815cd740e9cd915e7dd644
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/commit.c
    M block/mirror.c
    M block/replication.c
    M blockdev.c
    M include/block/block_int.h
    M qapi/block-core.json
    M qemu-img.c

  Log Message:
  -----------
  commit: Add filter-node-name to block-commit

Management tools need to be able to know about every node in the graph
and need a way to address them. Changing the graph structure was okay
because libvirt doesn't really manage the node level yet, but future
libvirt versions need to deal with both new and old version of qemu.

This new option to blockdev-commit allows the client to set a node-name
for the automatically inserted filter driver, and at the same time
serves as a witness for a future libvirt that this version of qemu does
automatically insert a filter driver.

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: 887354bd13ecb7ff68ec26892806c97512b77877
      
https://github.com/qemu/qemu/commit/887354bd13ecb7ff68ec26892806c97512b77877
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/block-backend.c
    M hmp.c
    M include/qemu-io.h
    M include/sysemu/block-backend.h
    M qemu-io-cmds.c

  Log Message:
  -----------
  hmp: Request permissions in qemu-io

The HMP command 'qemu-io' is a bit tricky because it wants to work on
the original BlockBackend, but additional permissions could be required.
The details are explained in a comment in the code, but in summary, just
request whatever permissions the current qemu-io command needs.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 6f5ef23a3ff09919b73eef8196969685cb2383ee
      
https://github.com/qemu/qemu/commit/6f5ef23a3ff09919b73eef8196969685cb2383ee
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M migration/block.c

  Log Message:
  -----------
  migration/block: Use real permissions

Request BLK_PERM_CONSISTENT_READ for the source of block migration, and
handle potential permission errors as good as we can in this place
(which is not very good, but it matches the other failure cases).

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: 8a7ce4f9338c475df1afc12502af704e4300a3e0
      
https://github.com/qemu/qemu/commit/8a7ce4f9338c475df1afc12502af704e4300a3e0
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M nbd/server.c

  Log Message:
  -----------
  nbd/server: Use real permissions for NBD exports

NBD can't cope with device size changes, so resize must be forbidden,
but otherwise we can tolerate anything. Depending on whether the export
is writable or not, we only require consistent reads and writes.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 2807c0cd439321dbac118b895cdd2b595f14bf4b
      
https://github.com/qemu/qemu/commit/2807c0cd439321dbac118b895cdd2b595f14bf4b
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M tests/test-blockjob.c
    M tests/test-throttle.c

  Log Message:
  -----------
  tests: Remove FIXME comments

Not requesting any permissions is actually correct for these test cases
because no actual I/O or other operation covered by the permission
system is performed.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 85c97ca7a10b93216bc95052e9dabe3a4bb8736a
      
https://github.com/qemu/qemu/commit/85c97ca7a10b93216bc95052e9dabe3a4bb8736a
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/io.c

  Log Message:
  -----------
  block: Pass BdrvChild to bdrv_aligned_preadv/pwritev and copy-on-read

This is where we want to check the permissions, so we need to have the
BdrvChild around where they are stored.

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: afa4b293230c0bad3dfbfa9c0ff3f7bdfab40430
      
https://github.com/qemu/qemu/commit/afa4b293230c0bad3dfbfa9c0ff3f7bdfab40430
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block/io.c

  Log Message:
  -----------
  block: Assertions for write permissions

This adds assertions that ensure that the necessary write permissions
have been granted before someone attempts to write to a node.

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: c8f6d58edb0defbe1e90d44419ec8ec6d711c341
      
https://github.com/qemu/qemu/commit/c8f6d58edb0defbe1e90d44419ec8ec6d711c341
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c
    M block/io.c

  Log Message:
  -----------
  block: Assertions for resize permission

This adds an assertion that ensures that the necessary resize permission
has been granted before bdrv_truncate() is called.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Acked-by: Fam Zheng <address@hidden>


  Commit: 12fa4af61fb2a08b156134c3b6717534c637c995
      
https://github.com/qemu/qemu/commit/12fa4af61fb2a08b156134c3b6717534c637c995
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c
    M block/commit.c
    M block/mirror.c
    M block/stream.c
    M block/vvfat.c
    M include/block/block.h

  Log Message:
  -----------
  block: Add Error parameter to bdrv_set_backing_hd()

Not all callers of bdrv_set_backing_hd() know for sure that attaching
the backing file will be allowed by the permission system. Return the
error from the function rather than aborting.

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: b2c2832c6140cfe3ddc0de2d77eeb0b77dea8fd3
      
https://github.com/qemu/qemu/commit/b2c2832c6140cfe3ddc0de2d77eeb0b77dea8fd3
  Author: Kevin Wolf <address@hidden>
  Date:   2017-02-28 (Tue, 28 Feb 2017)

  Changed paths:
    M block.c
    M block/mirror.c
    M blockdev.c
    M include/block/block.h
    M tests/qemu-iotests/085.out

  Log Message:
  -----------
  block: Add Error parameter to bdrv_append()

Aborting on error in bdrv_append() isn't correct. This patch fixes it
and lets the callers handle failures.

Test case 085 needs a reference output update. This is caused by the
reversed order of bdrv_set_backing_hd() and change_parent_backing_link()
in bdrv_append(): When the backing file of the new node is set, the
parent nodes are still pointing to the old top, so the backing blocker
is now initialised with the node name rather than the BlockBackend name.

Signed-off-by: Kevin Wolf <address@hidden>
Acked-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: b9fe31392bab329219260d846b3002a12667d992
      
https://github.com/qemu/qemu/commit/b9fe31392bab329219260d846b3002a12667d992
  Author: Peter Maydell <address@hidden>
  Date:   2017-03-01 (Wed, 01 Mar 2017)

  Changed paths:
    M block.c
    M block/backup.c
    M block/blkdebug.c
    M block/blkreplay.c
    M block/blkverify.c
    M block/block-backend.c
    M block/bochs.c
    M block/cloop.c
    M block/commit.c
    M block/crypto.c
    M block/dmg.c
    M block/io.c
    M block/mirror.c
    M block/parallels.c
    M block/qcow.c
    M block/qcow2.c
    M block/qed.c
    M block/quorum.c
    M block/raw-format.c
    M block/replication.c
    M block/sheepdog.c
    M block/stream.c
    M block/vdi.c
    M block/vhdx.c
    M block/vmdk.c
    M block/vpc.c
    M block/vvfat.c
    M blockdev.c
    M blockjob.c
    M hmp.c
    M hw/block/block.c
    M hw/block/fdc.c
    M hw/block/m25p80.c
    M hw/block/nand.c
    M hw/block/nvme.c
    M hw/block/onenand.c
    M hw/block/pflash_cfi01.c
    M hw/block/pflash_cfi02.c
    M hw/block/virtio-blk.c
    M hw/core/qdev-properties-system.c
    M hw/ide/core.c
    M hw/ide/qdev.c
    M hw/nvram/spapr_nvram.c
    M hw/scsi/scsi-disk.c
    M hw/sd/sd.c
    M hw/usb/dev-storage.c
    M include/block/block.h
    M include/block/block_int.h
    M include/block/blockjob.h
    M include/block/blockjob_int.h
    M include/hw/block/block.h
    M include/qemu-io.h
    M include/sysemu/block-backend.h
    M migration/block.c
    M nbd/server.c
    M qapi/block-core.json
    M qemu-img-cmds.hx
    M qemu-img.c
    M qemu-img.texi
    M qemu-io-cmds.c
    M tests/qemu-iotests/049.out
    M tests/qemu-iotests/051.pc.out
    M tests/qemu-iotests/055
    M tests/qemu-iotests/085.out
    M tests/qemu-iotests/141
    M tests/qemu-iotests/141.out
    M tests/qemu-iotests/172.out
    M tests/test-blockjob-txn.c
    M tests/test-blockjob.c
    M tests/test-throttle.c
    M util/qemu-option.c

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches

# gpg: Signature made Tue 28 Feb 2017 20:35:32 GMT
# gpg:                using RSA key 0x7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <address@hidden>"
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74  56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream: (46 commits)
  block: Add Error parameter to bdrv_append()
  block: Add Error parameter to bdrv_set_backing_hd()
  block: Assertions for resize permission
  block: Assertions for write permissions
  block: Pass BdrvChild to bdrv_aligned_preadv/pwritev and copy-on-read
  tests: Remove FIXME comments
  nbd/server: Use real permissions for NBD exports
  migration/block: Use real permissions
  hmp: Request permissions in qemu-io
  commit: Add filter-node-name to block-commit
  mirror: Add filter-node-name to blockdev-mirror
  stream: Use real permissions in streaming block job
  mirror: Use real permissions in mirror/active commit block job
  blockjob: Factor out block_job_remove_all_bdrv()
  block: Allow backing file links in change_parent_backing_link()
  block: BdrvChildRole.attach/detach() callbacks
  block: Fix pending requests check in bdrv_append()
  backup: Use real permissions in backup block job
  commit: Use real permissions for HMP 'commit'
  commit: Use real permissions in commit block job
  ...

Signed-off-by: Peter Maydell <address@hidden>


Compare: https://github.com/qemu/qemu/compare/1e0addb682c3...b9fe31392bab

reply via email to

[Prev in Thread] Current Thread [Next in Thread]