[Qemu-commits] [qemu/qemu] 4d7315: qemu-io: add pattern file for write command

[Peter Maydell]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 4d731510d34f280ed45a6de621d016f67a49ea48
      
https://github.com/qemu/qemu/commit/4d731510d34f280ed45a6de621d016f67a49ea48
  Author: Denis Plotnikov <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M qemu-io-cmds.c

  Log Message:
  -----------
  qemu-io: add pattern file for write command

The patch allows to provide a pattern file for write
command. There was no similar ability before.

Signed-off-by: Denis Plotnikov <address@hidden>
Message-id: address@hidden
Reviewed-by: Eric Blake <address@hidden>
[mreitz: Keep optstring in alphabetical order]
Signed-off-by: Max Reitz <address@hidden>


  Commit: b503de619ed462cd433187db60719f98fab470c2
      
https://github.com/qemu/qemu/commit/b503de619ed462cd433187db60719f98fab470c2
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M block.c

  Log Message:
  -----------
  block: fix permission update in bdrv_replace_node

It's wrong to OR shared permissions. It may lead to crash on further
permission updates.
Also, no needs to consider previously calculated permissions, as at
this point we already bind all new parents and bdrv_get_cumulative_perm
result is enough. So fix the bug by just set permissions by
bdrv_get_cumulative_perm result.

Bug was introduced in long ago 234ac1a9025, in 2.9.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: 3a20013fbb26d2a1bd11ef148eefdb1508783787
      
https://github.com/qemu/qemu/commit/3a20013fbb26d2a1bd11ef148eefdb1508783787
  Author: Nir Soffer <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M block/file-posix.c
    M tests/qemu-iotests/059.out
    R tests/qemu-iotests/150.out
    A tests/qemu-iotests/150.out.qcow2
    A tests/qemu-iotests/150.out.raw
    M tests/qemu-iotests/175
    M tests/qemu-iotests/175.out
    M tests/qemu-iotests/178.out.qcow2
    M tests/qemu-iotests/221.out
    M tests/qemu-iotests/253.out

  Log Message:
  -----------
  block: posix: Always allocate the first block

When creating an image with preallocation "off" or "falloc", the first
block of the image is typically not allocated. When using Gluster
storage backed by XFS filesystem, reading this block using direct I/O
succeeds regardless of request length, fooling alignment detection.

In this case we fallback to a safe value (4096) instead of the optimal
value (512), which may lead to unneeded data copying when aligning
requests.  Allocating the first block avoids the fallback.

Since we allocate the first block even with preallocation=off, we no
longer create images with zero disk size:

    $ ./qemu-img create -f raw test.raw 1g
    Formatting 'test.raw', fmt=raw size=1073741824

    $ ls -lhs test.raw
    4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw

And converting the image requires additional cluster:

    $ ./qemu-img measure -f raw -O qcow2 test.raw
    required size: 458752
    fully allocated size: 1074135040

When using format like vmdk with multiple files per image, we allocate
one block per file:

    $ ./qemu-img create -f vmdk -o subformat=twoGbMaxExtentFlat test.vmdk 4g
    Formatting 'test.vmdk', fmt=vmdk size=4294967296 compat6=off 
hwversion=undefined subformat=twoGbMaxExtentFlat

    $ ls -lhs test*.vmdk
    4.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f001.vmdk
    4.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f002.vmdk
    4.0K -rw-r--r--. 1 nsoffer nsoffer  353 Aug 27 03:23 test.vmdk

I did quick performance test for copying disks with qemu-img convert to
new raw target image to Gluster storage with sector size of 512 bytes:

    for i in $(seq 10); do
        rm -f dst.raw
        sleep 10
        time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw
    done

Here is a table comparing the total time spent:

Type    Before(s)   After(s)    Diff(%)
---------------------------------------
real      530.028    469.123      -11.4
user       17.204     10.768      -37.4
sys        17.881      7.011      -60.7

We can see very clear improvement in CPU usage.

Signed-off-by: Nir Soffer <address@hidden>
Message-id: address@hidden
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 7e3dc2ba9a11862d4e1a08325b9165f27a1b1e7c
      
https://github.com/qemu/qemu/commit/7e3dc2ba9a11862d4e1a08325b9165f27a1b1e7c
  Author: Nir Soffer <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M tests/qemu-iotests/175
    M tests/qemu-iotests/175.out

  Log Message:
  -----------
  iotests: Test allocate_first_block() with O_DIRECT

Using block_resize we can test allocate_first_block() with file
descriptor opened with O_DIRECT, ensuring that it works for any size
larger than 4096 bytes.

Testing smaller sizes is tricky as the result depends on the filesystem
used for testing. For example on NFS any size will work since O_DIRECT
does not require any alignment.

Signed-off-by: Nir Soffer <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: f158ffdba62360b221047dee488450f8c49f6a22
      
https://github.com/qemu/qemu/commit/f158ffdba62360b221047dee488450f8c49f6a22
  Author: Max Reitz <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M tests/qemu-iotests/059.out
    M tests/qemu-iotests/common.filter

  Log Message:
  -----------
  iotests: Fix _filter_img_create()

fe646693acc changed qemu-img create's output so that it no longer prints
single quotes around parameter values.  The subformat and adapter_type
filters in _filter_img_create() have never been adapted to that change.

Fixes: fe646693acc13ac48b98435d14149ab04dc597bc
Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: John Snow <address@hidden>
Message-id: address@hidden
Reviewed-by: John Snow <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: cdc0dd2586d726f7b549be08a8e073f803cd5dc9
      
https://github.com/qemu/qemu/commit/cdc0dd2586d726f7b549be08a8e073f803cd5dc9
  Author: Max Reitz <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M block/vmdk.c
    M tests/qemu-iotests/059
    M tests/qemu-iotests/059.out

  Log Message:
  -----------
  vmdk: Use bdrv_dirname() for relative extent paths

This makes iotest 033 pass with e.g. subformat=monolithicFlat.  It also
turns a former error in 059 into success.

Signed-off-by: Max Reitz <address@hidden>
Message-id: address@hidden
Reviewed-by: John Snow <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 12b7cbcabcacb9393b1aeebd2c23d49a25451ad5
      
https://github.com/qemu/qemu/commit/12b7cbcabcacb9393b1aeebd2c23d49a25451ad5
  Author: Max Reitz <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M tests/qemu-iotests/059
    M tests/qemu-iotests/059.out

  Log Message:
  -----------
  iotests: Keep testing broken relative extent paths

We had a test for a case where relative extent paths did not work, but
unfortunately we just fixed the underlying problem, so it works now.
This patch adds a new test case that still fails.

Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: John Snow <address@hidden>
Message-id: address@hidden
Reviewed-by: John Snow <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: bedb8bb41953fd7ee15236cc6005d97063d38efa
      
https://github.com/qemu/qemu/commit/bedb8bb41953fd7ee15236cc6005d97063d38efa
  Author: Max Reitz <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M block/vmdk.c

  Log Message:
  -----------
  vmdk: Reject invalid compressed writes

Compressed writes generally have to write full clusters, not just in
theory but also in practice when it comes to vmdk's streamOptimized
subformat.  It currently is just silently broken for writes with
non-zero in-cluster offsets:

$ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
$ qemu-io -c 'write 4k 4k' -c 'read 4k 4k' foo.vmdk
wrote 4096/4096 bytes at offset 4096
4 KiB, 1 ops; 00.01 sec (443.724 KiB/sec and 110.9309 ops/sec)
read failed: Invalid argument

(The technical reason is that vmdk_write_extent() just writes the
incomplete compressed data actually to offset 4k.  When reading the
data, vmdk_read_extent() looks at offset 0 and finds the compressed data
size to be 0, because that is what it reads from there.  This yields an
error.)

For incomplete writes with zero in-cluster offsets, the error path when
reading the rest of the cluster is a bit different, but the result is
the same:

$ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
$ qemu-io -c 'write 0k 4k' -c 'read 4k 4k' foo.vmdk
wrote 4096/4096 bytes at offset 0
4 KiB, 1 ops; 00.01 sec (362.641 KiB/sec and 90.6603 ops/sec)
read failed: Invalid argument

(Here, vmdk_read_extent() finds the data and then sees that the
uncompressed data is short.)

It is better to reject invalid writes than to make the user believe they
might have succeeded and then fail when trying to read it back.

Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: John Snow <address@hidden>
Message-id: address@hidden
Reviewed-by: John Snow <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 325dd915b231801e48f89b38a4bfb89c8956098c
      
https://github.com/qemu/qemu/commit/325dd915b231801e48f89b38a4bfb89c8956098c
  Author: Max Reitz <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M tests/qemu-iotests/002
    M tests/qemu-iotests/003
    M tests/qemu-iotests/005
    M tests/qemu-iotests/009
    M tests/qemu-iotests/010
    M tests/qemu-iotests/011
    M tests/qemu-iotests/017
    M tests/qemu-iotests/018
    M tests/qemu-iotests/019
    M tests/qemu-iotests/020
    M tests/qemu-iotests/027
    M tests/qemu-iotests/032
    M tests/qemu-iotests/033
    M tests/qemu-iotests/034
    M tests/qemu-iotests/037
    M tests/qemu-iotests/063
    M tests/qemu-iotests/072
    M tests/qemu-iotests/105
    M tests/qemu-iotests/197
    M tests/qemu-iotests/215
    M tests/qemu-iotests/251

  Log Message:
  -----------
  iotests: Disable broken streamOptimized tests

streamOptimized does not support writes that do not span exactly one
cluster.  Furthermore, it cannot rewrite already allocated clusters.
As such, many iotests do not work with it.  Disable them.

Signed-off-by: Max Reitz <address@hidden>
Message-id: address@hidden
Reviewed-by: John Snow <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: c64c3ae35b84b41140f6cd888624bea13ce4fc7e
      
https://github.com/qemu/qemu/commit/c64c3ae35b84b41140f6cd888624bea13ce4fc7e
  Author: Max Reitz <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M tests/qemu-iotests/110

  Log Message:
  -----------
  iotests: Disable 110 for vmdk.twoGbMaxExtentSparse

The error message for the test case where we have a quorum node for
which no directory name can be generated is different: For
twoGbMaxExtentSparse, it complains that it cannot open the extent file.
For other (sub)formats, it just notes that it cannot determine the
backing file path.  Both are fine, but just disable twoGbMaxExtentSparse
for simplicity's sake.

Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: John Snow <address@hidden>
Message-id: address@hidden
Reviewed-by: John Snow <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 39af39c428522e7c6a4015cb103d387b42a198b6
      
https://github.com/qemu/qemu/commit/39af39c428522e7c6a4015cb103d387b42a198b6
  Author: Max Reitz <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M tests/qemu-iotests/126

  Log Message:
  -----------
  iotests: Disable 126 for flat vmdk subformats

iotest 126 requires backing file support, which flat vmdks cannot offer.
Skip this test for such subformats.

Signed-off-by: Max Reitz <address@hidden>
Message-id: address@hidden
Reviewed-by: John Snow <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 236094c73815370f1a57abe4096b8b2dfd9bfd1b
      
https://github.com/qemu/qemu/commit/236094c73815370f1a57abe4096b8b2dfd9bfd1b
  Author: Stefan Hajnoczi <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M block/file-posix.c

  Log Message:
  -----------
  file-posix: fix request_alignment typo

Fixes: a6b257a08e3d72219f03e461a52152672fec0612
       ("file-posix: Handle undetectable alignment")
Signed-off-by: Stefan Hajnoczi <address@hidden>
Message-id: address@hidden
Reviewed-by: Thomas Huth <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 9da126fc2e7c5ab395eac8a5f7c4f119b1434943
      
https://github.com/qemu/qemu/commit/9da126fc2e7c5ab395eac8a5f7c4f119b1434943
  Author: Max Reitz <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M tests/qemu-iotests/check

  Log Message:
  -----------
  iotests: Add -display none to the qemu options

Without this argument, qemu will print an angry message about not being
able to connect to a display server if $DISPLAY is not set.  For me,
that breaks iotests.supported_formats() because it thus only sees
["Could", "not", "connect"] as the supported formats.

Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: Philippe Mathieu-Daudé <address@hidden>
Message-id: address@hidden
Reviewed-by: Thomas Huth <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 21b43d004813ae71d964f74e59ff149bb480db73
      
https://github.com/qemu/qemu/commit/21b43d004813ae71d964f74e59ff149bb480db73
  Author: Thomas Huth <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M tests/qemu-iotests/071
    M tests/qemu-iotests/081
    M tests/qemu-iotests/099
    M tests/qemu-iotests/120
    M tests/qemu-iotests/162
    M tests/qemu-iotests/184
    M tests/qemu-iotests/186
    M tests/qemu-iotests/common.rc

  Log Message:
  -----------
  iotests: Check for enabled drivers before testing them

It is possible to enable only a subset of the block drivers with the
"--block-drv-rw-whitelist" option of the "configure" script. All other
drivers are marked as unusable (or only included as read-only with the
"--block-drv-ro-whitelist" option). If an iotest is now using such a
disabled block driver, it is failing - which is bad, since at least the
tests in the "auto" group should be able to deal with this situation.
Thus let's introduce a "_require_drivers" function that can be used by
the shell tests to check for the availability of certain drivers first,
and marks the test as "not run" if one of the drivers is missing.

This patch mainly targets the test in the "auto" group which should
never fail in such a case, but also improves some of the other tests
along the way. Note that we also assume that the "qcow2" and "file"
drivers are always available - otherwise it does not make sense to
run "make check-block" at all (which only tests with qcow2 by default).

Signed-off-by: Thomas Huth <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: 2cc4d1c5eab1d7b1fd7112c1fafccaf648b92a86
      
https://github.com/qemu/qemu/commit/2cc4d1c5eab1d7b1fd7112c1fafccaf648b92a86
  Author: Thomas Huth <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M tests/check-block.sh

  Log Message:
  -----------
  tests/check-block: Skip iotests when sanitizers are enabled

The sanitizers (especially the address sanitizer from Clang) are
sometimes printing out warnings or false positives - this spoils
the output of the iotests, causing some of the tests to fail.
Thus let's skip the automatic iotests during "make check" when the
user configured QEMU with --enable-sanitizers.

Signed-off-by: Thomas Huth <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: 755c5fe79d88717600356d3edf04835bba43dcb6
      
https://github.com/qemu/qemu/commit/755c5fe79d88717600356d3edf04835bba43dcb6
  Author: Nir Soffer <address@hidden>
  Date:   2019-09-03 (Tue, 03 Sep 2019)

  Changed paths:
    M tests/qemu-iotests/026
    M tests/qemu-iotests/039
    M tests/qemu-iotests/052
    M tests/qemu-iotests/091

  Log Message:
  -----------
  iotests: Unify cache mode quoting

Quoting cache mode is not needed, and most tests use unquoted values.
Unify all test to use the same style.

Message-id: address@hidden
Signed-off-by: Nir Soffer <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 6b422e5f58f50b34b20c50ebe496fcb822f419b5
      
https://github.com/qemu/qemu/commit/6b422e5f58f50b34b20c50ebe496fcb822f419b5
  Author: Peter Maydell <address@hidden>
  Date:   2019-09-04 (Wed, 04 Sep 2019)

  Changed paths:
    M block.c
    M block/file-posix.c
    M block/vmdk.c
    M qemu-io-cmds.c
    M tests/check-block.sh
    M tests/qemu-iotests/002
    M tests/qemu-iotests/003
    M tests/qemu-iotests/005
    M tests/qemu-iotests/009
    M tests/qemu-iotests/010
    M tests/qemu-iotests/011
    M tests/qemu-iotests/017
    M tests/qemu-iotests/018
    M tests/qemu-iotests/019
    M tests/qemu-iotests/020
    M tests/qemu-iotests/026
    M tests/qemu-iotests/027
    M tests/qemu-iotests/032
    M tests/qemu-iotests/033
    M tests/qemu-iotests/034
    M tests/qemu-iotests/037
    M tests/qemu-iotests/039
    M tests/qemu-iotests/052
    M tests/qemu-iotests/059
    M tests/qemu-iotests/059.out
    M tests/qemu-iotests/063
    M tests/qemu-iotests/071
    M tests/qemu-iotests/072
    M tests/qemu-iotests/081
    M tests/qemu-iotests/091
    M tests/qemu-iotests/099
    M tests/qemu-iotests/105
    M tests/qemu-iotests/110
    M tests/qemu-iotests/120
    M tests/qemu-iotests/126
    R tests/qemu-iotests/150.out
    A tests/qemu-iotests/150.out.qcow2
    A tests/qemu-iotests/150.out.raw
    M tests/qemu-iotests/162
    M tests/qemu-iotests/175
    M tests/qemu-iotests/175.out
    M tests/qemu-iotests/178.out.qcow2
    M tests/qemu-iotests/184
    M tests/qemu-iotests/186
    M tests/qemu-iotests/197
    M tests/qemu-iotests/215
    M tests/qemu-iotests/221.out
    M tests/qemu-iotests/251
    M tests/qemu-iotests/253.out
    M tests/qemu-iotests/check
    M tests/qemu-iotests/common.filter
    M tests/qemu-iotests/common.rc

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2019-09-03' 
into staging

Block patches:
- qemu-io now accepts a file to read a write pattern from
- Ensure that raw files have their first block allocated so we can probe
  the O_DIRECT alignment if necessary
- Various fixes

# gpg: Signature made Tue 03 Sep 2019 13:58:57 BST
# gpg:                using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg:                issuer "address@hidden"
# gpg: Good signature from "Max Reitz <address@hidden>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1  1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2019-09-03:
  iotests: Unify cache mode quoting
  tests/check-block: Skip iotests when sanitizers are enabled
  iotests: Check for enabled drivers before testing them
  iotests: Add -display none to the qemu options
  file-posix: fix request_alignment typo
  iotests: Disable 126 for flat vmdk subformats
  iotests: Disable 110 for vmdk.twoGbMaxExtentSparse
  iotests: Disable broken streamOptimized tests
  vmdk: Reject invalid compressed writes
  iotests: Keep testing broken relative extent paths
  vmdk: Use bdrv_dirname() for relative extent paths
  iotests: Fix _filter_img_create()
  iotests: Test allocate_first_block() with O_DIRECT
  block: posix: Always allocate the first block
  block: fix permission update in bdrv_replace_node
  qemu-io: add pattern file for write command

Signed-off-by: Peter Maydell <address@hidden>


Compare: https://github.com/qemu/qemu/compare/d3714799718b...6b422e5f58f5