[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Qemu-commits] [qemu/qemu] 4b196c: iotests: Simplify _filter_img_create(
|
From: |
Peter Maydell |
|
Subject: |
[Qemu-commits] [qemu/qemu] 4b196c: iotests: Simplify _filter_img_create() a bit |
|
Date: |
Tue, 14 Jul 2020 12:30:29 -0700 |
Branch: refs/heads/master
Home: https://github.com/qemu/qemu
Commit: 4b196cd16dcfb17de19a4121f12aa4ef4bf7925f
https://github.com/qemu/qemu/commit/4b196cd16dcfb17de19a4121f12aa4ef4bf7925f
Author: Max Reitz <mreitz@redhat.com>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M tests/qemu-iotests/020
M tests/qemu-iotests/020.out
M tests/qemu-iotests/141
M tests/qemu-iotests/common.filter
Log Message:
-----------
iotests: Simplify _filter_img_create() a bit
Not only is it a bit stupid to try to filter multi-line "Formatting"
output (because we only need it for a single test, which can easily be
amended to no longer need it), it is also problematic when there can be
output after a "Formatting" line that we do not want to filter as if it
were part of it.
So rename _filter_img_create to _do_filter_img_create, let it filter
only a single line, and let _filter_img_create loop over all input
lines, calling _do_filter_img_create only on those that match
/^Formatting/ (basically, what _filter_img_create_in_qmp did already).
(And fix 020 to work with that.)
Reported-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200709110205.310942-1-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit: d0ceea88dea053e0c1c038d42ca98782c2e3872d
https://github.com/qemu/qemu/commit/d0ceea88dea053e0c1c038d42ca98782c2e3872d
Author: Kevin Wolf <kwolf@redhat.com>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M qemu-img.c
Log Message:
-----------
qemu-img map: Don't limit block status request size
Limiting each loop iteration of qemu-img map to 1 GB was arbitrary from
the beginning, though it only cut the maximum in half then because the
interface was a signed 32 bit byte count. These days, bdrv_block_status
supports a 64 bit byte count, so the arbitrary limit is even worse.
On file-posix, bdrv_block_status() eventually maps to SEEK_HOLE and
SEEK_DATA, which don't support a limit, but always do all of the work
necessary to find the start of the next hole/data. Much of this work may
be repeated if we don't use this information fully, but query with an
only slightly larger offset in the next loop iteration. Therefore, if
bdrv_block_status() is called in a loop, it should always pass the
full number of bytes that the whole loop is interested in.
This removes the arbitrary limit and speeds up 'qemu-img map'
significantly on heavily fragmented images.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707144629.51235-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit: 046e07ca556b3eb44ac5c64911b5bda204403113
https://github.com/qemu/qemu/commit/046e07ca556b3eb44ac5c64911b5bda204403113
Author: Kevin Wolf <kwolf@redhat.com>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M tests/qemu-iotests/059
M tests/qemu-iotests/059.out
Log Message:
-----------
iotests/059: Filter out disk size with more standard filter
The actual disk space used by an image can vary between filesystems and
depending on other settings like an extent size hint. Replace the one
call of "$QEMU_IMG info" and the associated one-off sed filter with the
more standard "_img_info" and the standard filter from common.filter.
Apart from turning "vmdk" into "IMGFMT" and changing the placeholder for
cid fields, this only removes the "disk size" line.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit: ffa244c84a1a30dff69ecc80b0137a2b6d428ecb
https://github.com/qemu/qemu/commit/ffa244c84a1a30dff69ecc80b0137a2b6d428ecb
Author: Kevin Wolf <kwolf@redhat.com>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M block/file-posix.c
M include/block/block_int.h
M qapi/block-core.json
M tests/qemu-iotests/082.out
M tests/qemu-iotests/106
M tests/qemu-iotests/175
M tests/qemu-iotests/243
Log Message:
-----------
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240
extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192
-o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting
at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192
-o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting
at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240
extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192
-o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting
at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192
-o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting
at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit: 4e2f4418784da09cb106264340241856cd2846df
https://github.com/qemu/qemu/commit/4e2f4418784da09cb106264340241856cd2846df
Author: Eric Blake <eblake@redhat.com>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M block.c
M tests/qemu-iotests/049.out
M tests/qemu-iotests/054.out
M tests/qemu-iotests/079.out
M tests/qemu-iotests/112.out
M tests/qemu-iotests/259.out
M tests/qemu-iotests/282.out
Log Message:
-----------
qemu-img: Flush stdout before before potential stderr messages
During 'qemu-img create ... 2>&1', if --quiet is not in force, we can
end up with buffered I/O in stdout that was produced before failure,
but which appears in output after failure. This is confusing; the fix
is to flush stdout prior to attempting anything that might produce an
error message. Several iotests demonstrate the resulting ordering
change now that the merged outputs now reflect chronology. (An even
better fix would be to avoid printf from within block.c altogether,
but that's much more invasive...)
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-2-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit: 25956af3fe5dd0385ad8017bc768a6afe41e2a74
https://github.com/qemu/qemu/commit/25956af3fe5dd0385ad8017bc768a6afe41e2a74
Author: Eric Blake <eblake@redhat.com>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M docs/system/deprecated.rst
M qemu-img.c
M tests/qemu-iotests/122
M tests/qemu-iotests/122.out
Log Message:
-----------
block: Finish deprecation of 'qemu-img convert -n -o'
It's been two releases since we started warning; time to make the
combination an error as promised. There was no iotest coverage, so
add some.
While touching the documentation, tweak another section heading for
consistent style.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-3-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit: 80fa43e7df5d68d60a2662036b7d3d95ca8b68f3
https://github.com/qemu/qemu/commit/80fa43e7df5d68d60a2662036b7d3d95ca8b68f3
Author: Eric Blake <eblake@redhat.com>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M block/sheepdog.c
Log Message:
-----------
sheepdog: Add trivial backing_fmt support
Sheepdog already requires that if backing_file is present, that it be
another sheepdog image (see sd_co_create). Meanwhile, we want to move
towards always being explicit about the backing format for other
drivers where it matters. So for convenience, make qemu-img create -F
sheepdog work, while rejecting all other explicit formats (note that
this is only for QemuOpts usage; there is no change to the QAPI to
allow a format through -blockdev).
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-4-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit: d51a814cf41033d2d29b050e04d85155ac941221
https://github.com/qemu/qemu/commit/d51a814cf41033d2d29b050e04d85155ac941221
Author: Eric Blake <eblake@redhat.com>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M block/vmdk.c
Log Message:
-----------
vmdk: Add trivial backing_fmt support
vmdk already requires that if backing_file is present, that it be
another vmdk image (see vmdk_co_do_create). Meanwhile, we want to
move towards always being explicit about the backing format for other
drivers where it matters. So for convenience, make qemu-img create -F
vmdk work, while rejecting all other explicit formats (note that this
is only for QemuOpts usage; there is no change to the QAPI to allow a
format through -blockdev).
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-5-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit: 344acbd62ffdbeb7f803644ad46a8129059f6823
https://github.com/qemu/qemu/commit/344acbd62ffdbeb7f803644ad46a8129059f6823
Author: Eric Blake <eblake@redhat.com>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M block/qcow.c
A tests/qemu-iotests/301
A tests/qemu-iotests/301.out
M tests/qemu-iotests/group
Log Message:
-----------
qcow: Tolerate backing_fmt=
qcow has no space in the metadata to store a backing format, and there
are existing qcow images backed both by raw or by other formats
(usually qcow) images, reliant on probing to tell the difference. On
the bright side, because we probe every time, raw files are marked as
probed and we thus forbid a commit action into the backing file where
guest-controlled contents could change the result of the probe next
time around (the iotest added here proves that).
Still, allowing the user to specify the backing format during
creation, even if we can't record it, is a good thing. This patch
blindly allows any value that resolves to a known driver, even if the
user's request is a mismatch from what probing finds; then the next
patch will further enhance things to verify that the user's request
matches what we actually probe. With this and the next patch in
place, we will finally be ready to deprecate the creation of images
where a backing format was not explicitly specified by the user.
Note that this is only for QemuOpts usage; there is no change to the
QAPI to allow a format through -blockdev.
Add a new iotest 301 just for qcow, to demonstrate the latest
behavior, and to make it easier to show the improvements made in the
next patch.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-6-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit: add8200dd14041d059cc376eff91461fadd93ec5
https://github.com/qemu/qemu/commit/add8200dd14041d059cc376eff91461fadd93ec5
Author: Eric Blake <eblake@redhat.com>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M block.c
M docs/system/deprecated.rst
M tests/qemu-iotests/111.out
M tests/qemu-iotests/301.out
Log Message:
-----------
block: Error if backing file fails during creation without -u
Back in commit 6e6e55f5 (Jul 2017, v2.10), we tweaked the code to warn
if the backing file could not be opened but the user gave a size,
unless the user also passes the -u option to bypass the open of the
backing file. As one common reason for failure to open the backing
file is when there is mismatch in the requested backing format in
relation to what the backing file actually contains, we actually want
to open the backing file and ensure that it has the right format in as
many cases as possible. iotest 301 for qcow demonstrates how
detecting explicit format mismatch is useful to prevent the creation
of an image that would probe differently than the user requested. Now
is the time to finally turn the warning an error, as promised.
Note that the original warning was added prior to our documentation of
an official deprecation policy (eb22aeca, also Jul 2017), and because
the warning didn't mention the word "deprecated", we never actually
remembered to document it as such. But the warning has been around
long enough that I don't see prolonging it another two releases.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-7-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit: bc5ee6da7122f6fe93ed07241a44315a331487e9
https://github.com/qemu/qemu/commit/bc5ee6da7122f6fe93ed07241a44315a331487e9
Author: Eric Blake <eblake@redhat.com>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M block/qcow2.c
M docs/system/deprecated.rst
M docs/tools/qemu-img.rst
M tests/qemu-iotests/061.out
M tests/qemu-iotests/082.out
Log Message:
-----------
qcow2: Deprecate use of qemu-img amend to change backing file
The use of 'qemu-img amend' to change qcow2 backing files is not
tested very well. In particular, our implementation has a bug where
if a new backing file is provided without a format, then the prior
format is blindly reused, even if this results in data corruption, but
this is not caught by iotests.
There are also situations where amending other options needs access to
the original backing file (for example, on a downgrade to a v2 image,
knowing whether a v3 zero cluster must be allocated or may be left
unallocated depends on knowing whether the backing file already reads
as zero), but the command line does not have a nice way to tell us
both the backing file to use for opening the image as well as the
backing file to install after the operation is complete.
Even if we do allow changing the backing file, it is redundant with
the existing ability to change backing files via 'qemu-img rebase -u'.
It is time to deprecate this support (leaving the existing behavior
intact, even if it is buggy), and at a point in the future, require
the use of only 'qemu-img rebase' for adjusting backing chain
relations, saving 'qemu-img amend' for changes unrelated to the
backing chain.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-8-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit: b66ff2c29817f5efa18f5120fd6f089fbf59a933
https://github.com/qemu/qemu/commit/b66ff2c29817f5efa18f5120fd6f089fbf59a933
Author: Eric Blake <eblake@redhat.com>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M tests/qemu-iotests/017
M tests/qemu-iotests/017.out
M tests/qemu-iotests/018
M tests/qemu-iotests/018.out
M tests/qemu-iotests/019
M tests/qemu-iotests/019.out
M tests/qemu-iotests/020
M tests/qemu-iotests/020.out
M tests/qemu-iotests/024
M tests/qemu-iotests/024.out
M tests/qemu-iotests/028
M tests/qemu-iotests/028.out
M tests/qemu-iotests/030
M tests/qemu-iotests/034
M tests/qemu-iotests/034.out
M tests/qemu-iotests/037
M tests/qemu-iotests/037.out
M tests/qemu-iotests/038
M tests/qemu-iotests/038.out
M tests/qemu-iotests/039
M tests/qemu-iotests/039.out
M tests/qemu-iotests/040
M tests/qemu-iotests/041
M tests/qemu-iotests/042
M tests/qemu-iotests/043
M tests/qemu-iotests/043.out
M tests/qemu-iotests/046
M tests/qemu-iotests/046.out
M tests/qemu-iotests/050
M tests/qemu-iotests/050.out
M tests/qemu-iotests/051
M tests/qemu-iotests/051.out
M tests/qemu-iotests/051.pc.out
M tests/qemu-iotests/056
M tests/qemu-iotests/060
M tests/qemu-iotests/060.out
M tests/qemu-iotests/061
M tests/qemu-iotests/061.out
M tests/qemu-iotests/069
M tests/qemu-iotests/069.out
M tests/qemu-iotests/073
M tests/qemu-iotests/073.out
M tests/qemu-iotests/082
M tests/qemu-iotests/082.out
M tests/qemu-iotests/085
M tests/qemu-iotests/085.out
M tests/qemu-iotests/089
M tests/qemu-iotests/089.out
M tests/qemu-iotests/095
M tests/qemu-iotests/095.out
M tests/qemu-iotests/097
M tests/qemu-iotests/097.out
M tests/qemu-iotests/098
M tests/qemu-iotests/098.out
M tests/qemu-iotests/110
M tests/qemu-iotests/110.out
M tests/qemu-iotests/122
M tests/qemu-iotests/122.out
M tests/qemu-iotests/126
M tests/qemu-iotests/126.out
M tests/qemu-iotests/127
M tests/qemu-iotests/127.out
M tests/qemu-iotests/129
M tests/qemu-iotests/133
M tests/qemu-iotests/133.out
M tests/qemu-iotests/139
M tests/qemu-iotests/141
M tests/qemu-iotests/141.out
M tests/qemu-iotests/142
M tests/qemu-iotests/142.out
M tests/qemu-iotests/153
M tests/qemu-iotests/153.out
M tests/qemu-iotests/154
M tests/qemu-iotests/154.out
M tests/qemu-iotests/155
M tests/qemu-iotests/156
M tests/qemu-iotests/156.out
M tests/qemu-iotests/158
M tests/qemu-iotests/158.out
M tests/qemu-iotests/161
M tests/qemu-iotests/161.out
M tests/qemu-iotests/176
M tests/qemu-iotests/176.out
M tests/qemu-iotests/177
M tests/qemu-iotests/177.out
M tests/qemu-iotests/179
M tests/qemu-iotests/179.out
M tests/qemu-iotests/189
M tests/qemu-iotests/189.out
M tests/qemu-iotests/191
M tests/qemu-iotests/191.out
M tests/qemu-iotests/195
M tests/qemu-iotests/195.out
M tests/qemu-iotests/198
M tests/qemu-iotests/198.out
M tests/qemu-iotests/204
M tests/qemu-iotests/204.out
M tests/qemu-iotests/216
M tests/qemu-iotests/224
M tests/qemu-iotests/225
M tests/qemu-iotests/225.out
M tests/qemu-iotests/228
M tests/qemu-iotests/245
M tests/qemu-iotests/249
M tests/qemu-iotests/249.out
M tests/qemu-iotests/252
M tests/qemu-iotests/257
M tests/qemu-iotests/267
M tests/qemu-iotests/267.out
M tests/qemu-iotests/270
M tests/qemu-iotests/270.out
M tests/qemu-iotests/273
M tests/qemu-iotests/273.out
M tests/qemu-iotests/274
M tests/qemu-iotests/274.out
M tests/qemu-iotests/279
M tests/qemu-iotests/279.out
M tests/qemu-iotests/290
M tests/qemu-iotests/290.out
Log Message:
-----------
iotests: Specify explicit backing format where sensible
There are many existing qcow2 images that specify a backing file but
no format. This has been the source of CVEs in the past, but has
become more prominent of a problem now that libvirt has switched to
-blockdev. With older -drive, at least the probing was always done by
qemu (so the only risk of a changed format between successive boots of
a guest was if qemu was upgraded and probed differently). But with
newer -blockdev, libvirt must specify a format; if libvirt guesses raw
where the image was formatted, this results in data corruption visible
to the guest; conversely, if libvirt guesses qcow2 where qemu was
using raw, this can result in potential security holes, so modern
libvirt instead refuses to use images without explicit backing format.
The change in libvirt to reject images without explicit backing format
has pointed out that a number of tools have been far too reliant on
probing in the past. It's time to set a better example in our own
iotests of properly setting this parameter.
iotest calls to create, rebase, and convert are all impacted to some
degree. It's a bit annoying that we are inconsistent on command line
- while all of those accept -o backing_file=...,backing_fmt=..., the
shortcuts are different: create and rebase have -b and -F, while
convert has -B but no -F. (amend has no shortcuts, but the previous
patch just deprecated the use of amend to change backing chains).
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-9-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit: e54ee1b385a9d084b4052b6db7391ea2fd799fa8
https://github.com/qemu/qemu/commit/e54ee1b385a9d084b4052b6db7391ea2fd799fa8
Author: Eric Blake <eblake@redhat.com>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M block.c
M block/qcow2.c
M block/stream.c
M blockdev.c
M include/block/block.h
M qemu-img.c
Log Message:
-----------
block: Add support to warn on backing file change without format
For now, this is a mechanical addition; all callers pass false. But
the next patch will use it to improve 'qemu-img rebase -u' when
selecting a backing file with no format.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Message-Id: <20200706203954.341758-10-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit: d9f059aa6cfccefaffa3532556e966df4a99ece2
https://github.com/qemu/qemu/commit/d9f059aa6cfccefaffa3532556e966df4a99ece2
Author: Eric Blake <eblake@redhat.com>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M block.c
M docs/system/deprecated.rst
M qemu-img.c
M tests/qemu-iotests/114
M tests/qemu-iotests/114.out
M tests/qemu-iotests/301.out
Log Message:
-----------
qemu-img: Deprecate use of -b without -F
Creating an image that requires format probing of the backing image is
potentially unsafe (we've had several CVEs over the years based on
probes leaking information to the guest on a subsequent boot, although
these days tools like libvirt are aware of the issue enough to prevent
the worst effects). For example, if our probing algorithm ever
changes, or if other tools like libvirt determine a different probe
result than we do, then subsequent use of that backing file under a
different format will present corrupted data to the guest.
Fortunately, the worst effects occur only when the backing image is
originally raw, and we at least prevent commit into a probed raw
backing file that would change its probed type.
Still, it is worth starting a deprecation clock so that future
qemu-img can refuse to create backing chains that would rely on
probing, to encourage clients to avoid unsafe practices. Most
warnings are intentionally emitted from bdrv_img_create() in the block
layer, but qemu-img convert uses bdrv_create() which cannot emit its
own warning without causing spurious warnings on other code paths. In
the end, all command-line image creation or backing file rewriting now
performs a check.
Furthermore, if we probe a backing file as non-raw, then it is safe to
explicitly record that result (rather than relying on future probes);
only where we probe a raw image do we care about further warnings to
the user when using such an image (for example, commits into a
probed-raw backing file are prevented), to help them improve their
tooling. But whether or not we make the probe results explicit, we
still warn the user to remind them to upgrade their workflow to supply
-F always.
iotest 114 specifically wants to create an unsafe image for later
amendment rather than defaulting to our new default of recording a
probed format, so it needs an update. While touching it, expand it to
cover all of the various warnings enabled by this patch. iotest 301
also shows a change to qcow messages.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-11-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit: e6cada9231af022ffc2e351c70dfaea8530496e1
https://github.com/qemu/qemu/commit/e6cada9231af022ffc2e351c70dfaea8530496e1
Author: Greg Kurz <groug@kaod.org>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M block/block-backend.c
Log Message:
-----------
block: Avoid stale pointer dereference in blk_get_aio_context()
It is possible for blk_remove_bs() to race with blk_drain_all(), causing
the latter to dereference a stale blk->root pointer:
blk_remove_bs(blk)
bdrv_root_unref_child(blk->root)
child_bs = blk->root->bs
bdrv_detach_child(blk->root)
...
g_free(blk->root) <============== blk->root becomes stale
bdrv_unref(child_bs) <============ yield at some point
A blk_drain_all() can be triggered by some guest action in the
meantime, eg. on POWER, SLOF might disable bus mastering on
a virtio-scsi-pci device:
virtio_write_config()
virtio_pci_stop_ioeventfd()
virtio_bus_stop_ioeventfd()
virtio_scsi_dataplane_stop()
blk_drain_all()
blk_get_aio_context()
bs = blk->root ? blk->root->bs : NULL
^^^^^^^^^
stale
Then, depending on one's luck, QEMU either crashes with SEGV or
hits the assertion in blk_get_aio_context().
blk->root is set by blk_insert_bs() which calls bdrv_root_attach_child()
first. The blk_remove_bs() function should rollback the changes made
by blk_insert_bs() in the opposite order (or it should be documented
somewhere why this isn't the case). Clear blk->root before calling
bdrv_root_unref_child() in blk_remove_bs().
Signed-off-by: Greg Kurz <groug@kaod.org>
Message-Id: <159430264541.389456.11925072456012783045.stgit@bahia.lan>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Commit: d2628b1eb761a5fbf08f367da405eb3314a1f068
https://github.com/qemu/qemu/commit/d2628b1eb761a5fbf08f367da405eb3314a1f068
Author: Peter Maydell <peter.maydell@linaro.org>
Date: 2020-07-14 (Tue, 14 Jul 2020)
Changed paths:
M block.c
M block/block-backend.c
M block/file-posix.c
M block/qcow.c
M block/qcow2.c
M block/sheepdog.c
M block/stream.c
M block/vmdk.c
M blockdev.c
M docs/system/deprecated.rst
M docs/tools/qemu-img.rst
M include/block/block.h
M include/block/block_int.h
M qapi/block-core.json
M qemu-img.c
M tests/qemu-iotests/017
M tests/qemu-iotests/017.out
M tests/qemu-iotests/018
M tests/qemu-iotests/018.out
M tests/qemu-iotests/019
M tests/qemu-iotests/019.out
M tests/qemu-iotests/020
M tests/qemu-iotests/020.out
M tests/qemu-iotests/024
M tests/qemu-iotests/024.out
M tests/qemu-iotests/028
M tests/qemu-iotests/028.out
M tests/qemu-iotests/030
M tests/qemu-iotests/034
M tests/qemu-iotests/034.out
M tests/qemu-iotests/037
M tests/qemu-iotests/037.out
M tests/qemu-iotests/038
M tests/qemu-iotests/038.out
M tests/qemu-iotests/039
M tests/qemu-iotests/039.out
M tests/qemu-iotests/040
M tests/qemu-iotests/041
M tests/qemu-iotests/042
M tests/qemu-iotests/043
M tests/qemu-iotests/043.out
M tests/qemu-iotests/046
M tests/qemu-iotests/046.out
M tests/qemu-iotests/049.out
M tests/qemu-iotests/050
M tests/qemu-iotests/050.out
M tests/qemu-iotests/051
M tests/qemu-iotests/051.out
M tests/qemu-iotests/051.pc.out
M tests/qemu-iotests/054.out
M tests/qemu-iotests/056
M tests/qemu-iotests/059
M tests/qemu-iotests/059.out
M tests/qemu-iotests/060
M tests/qemu-iotests/060.out
M tests/qemu-iotests/061
M tests/qemu-iotests/061.out
M tests/qemu-iotests/069
M tests/qemu-iotests/069.out
M tests/qemu-iotests/073
M tests/qemu-iotests/073.out
M tests/qemu-iotests/079.out
M tests/qemu-iotests/082
M tests/qemu-iotests/082.out
M tests/qemu-iotests/085
M tests/qemu-iotests/085.out
M tests/qemu-iotests/089
M tests/qemu-iotests/089.out
M tests/qemu-iotests/095
M tests/qemu-iotests/095.out
M tests/qemu-iotests/097
M tests/qemu-iotests/097.out
M tests/qemu-iotests/098
M tests/qemu-iotests/098.out
M tests/qemu-iotests/106
M tests/qemu-iotests/110
M tests/qemu-iotests/110.out
M tests/qemu-iotests/111.out
M tests/qemu-iotests/112.out
M tests/qemu-iotests/114
M tests/qemu-iotests/114.out
M tests/qemu-iotests/122
M tests/qemu-iotests/122.out
M tests/qemu-iotests/126
M tests/qemu-iotests/126.out
M tests/qemu-iotests/127
M tests/qemu-iotests/127.out
M tests/qemu-iotests/129
M tests/qemu-iotests/133
M tests/qemu-iotests/133.out
M tests/qemu-iotests/139
M tests/qemu-iotests/141
M tests/qemu-iotests/141.out
M tests/qemu-iotests/142
M tests/qemu-iotests/142.out
M tests/qemu-iotests/153
M tests/qemu-iotests/153.out
M tests/qemu-iotests/154
M tests/qemu-iotests/154.out
M tests/qemu-iotests/155
M tests/qemu-iotests/156
M tests/qemu-iotests/156.out
M tests/qemu-iotests/158
M tests/qemu-iotests/158.out
M tests/qemu-iotests/161
M tests/qemu-iotests/161.out
M tests/qemu-iotests/175
M tests/qemu-iotests/176
M tests/qemu-iotests/176.out
M tests/qemu-iotests/177
M tests/qemu-iotests/177.out
M tests/qemu-iotests/179
M tests/qemu-iotests/179.out
M tests/qemu-iotests/189
M tests/qemu-iotests/189.out
M tests/qemu-iotests/191
M tests/qemu-iotests/191.out
M tests/qemu-iotests/195
M tests/qemu-iotests/195.out
M tests/qemu-iotests/198
M tests/qemu-iotests/198.out
M tests/qemu-iotests/204
M tests/qemu-iotests/204.out
M tests/qemu-iotests/216
M tests/qemu-iotests/224
M tests/qemu-iotests/225
M tests/qemu-iotests/225.out
M tests/qemu-iotests/228
M tests/qemu-iotests/243
M tests/qemu-iotests/245
M tests/qemu-iotests/249
M tests/qemu-iotests/249.out
M tests/qemu-iotests/252
M tests/qemu-iotests/257
M tests/qemu-iotests/259.out
M tests/qemu-iotests/267
M tests/qemu-iotests/267.out
M tests/qemu-iotests/270
M tests/qemu-iotests/270.out
M tests/qemu-iotests/273
M tests/qemu-iotests/273.out
M tests/qemu-iotests/274
M tests/qemu-iotests/274.out
M tests/qemu-iotests/279
M tests/qemu-iotests/279.out
M tests/qemu-iotests/282.out
M tests/qemu-iotests/290
M tests/qemu-iotests/290.out
A tests/qemu-iotests/301
A tests/qemu-iotests/301.out
M tests/qemu-iotests/common.filter
M tests/qemu-iotests/group
Log Message:
-----------
Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging
Block layer patches:
- file-posix: Mitigate file fragmentation with extent size hints
- Tighten qemu-img rules on missing backing format
- qemu-img map: Don't limit block status request size
- Fix crash with virtio-scsi and iothreads
# gpg: Signature made Tue 14 Jul 2020 14:24:19 BST
# gpg: using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg: issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6
* remotes/kevin/tags/for-upstream:
block: Avoid stale pointer dereference in blk_get_aio_context()
qemu-img: Deprecate use of -b without -F
block: Add support to warn on backing file change without format
iotests: Specify explicit backing format where sensible
qcow2: Deprecate use of qemu-img amend to change backing file
block: Error if backing file fails during creation without -u
qcow: Tolerate backing_fmt=
vmdk: Add trivial backing_fmt support
sheepdog: Add trivial backing_fmt support
block: Finish deprecation of 'qemu-img convert -n -o'
qemu-img: Flush stdout before before potential stderr messages
file-posix: Mitigate file fragmentation with extent size hints
iotests/059: Filter out disk size with more standard filter
qemu-img map: Don't limit block status request size
iotests: Simplify _filter_img_create() a bit
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Compare: https://github.com/qemu/qemu/compare/aeb07b5f6e69...d2628b1eb761
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- [Qemu-commits] [qemu/qemu] 4b196c: iotests: Simplify _filter_img_create() a bit,
Peter Maydell <=