[Qemu-commits] [qemu/qemu] 6caaad: block/vhdx: add check for truncated image files

[Peter Maydell]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 6caaad46de02c02fb1eeaae64fcdc74f9e0a5eb0
      
https://github.com/qemu/qemu/commit/6caaad46de02c02fb1eeaae64fcdc74f9e0a5eb0
  Author: Peter Lieven <address@hidden>
  Date:   2019-10-14 (Mon, 14 Oct 2019)

  Changed paths:
    M block/vhdx.c

  Log Message:
  -----------
  block/vhdx: add check for truncated image files

qemu is currently not able to detect truncated vhdx image files.
Add a basic check if all allocated blocks are reachable at open and
report all errors during bdrv_co_check.

Signed-off-by: Peter Lieven <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 3c6c4348f23131155846dc53ce19aca309796fda
      
https://github.com/qemu/qemu/commit/3c6c4348f23131155846dc53ce19aca309796fda
  Author: Pavel Dovgalyuk <address@hidden>
  Date:   2019-10-14 (Mon, 14 Oct 2019)

  Changed paths:
    M block/blkreplay.c

  Log Message:
  -----------
  block: implement bdrv_snapshot_goto for blkreplay

This patch enables making snapshots with blkreplay used in
block devices.
This function is required to make bdrv_snapshot_goto without
calling .bdrv_open which is not implemented.

Signed-off-by: Pavel Dovgalyuk <address@hidden>
Acked-by: Kevin Wolf <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 258639754461e39d307ce3d340d213e0e5f339e3
      
https://github.com/qemu/qemu/commit/258639754461e39d307ce3d340d213e0e5f339e3
  Author: Pavel Dovgalyuk <address@hidden>
  Date:   2019-10-14 (Mon, 14 Oct 2019)

  Changed paths:
    M vl.c

  Log Message:
  -----------
  replay: disable default snapshot for record/replay

This patch disables setting '-snapshot' option on by default
in record/replay mode. This is needed for creating vmstates in record
and replay modes.

Signed-off-by: Pavel Dovgalyuk <address@hidden>
Acked-by: Kevin Wolf <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: de499eb67cda5c852605fe365950844412cbe789
      
https://github.com/qemu/qemu/commit/de499eb67cda5c852605fe365950844412cbe789
  Author: Pavel Dovgalyuk <address@hidden>
  Date:   2019-10-14 (Mon, 14 Oct 2019)

  Changed paths:
    M docs/replay.txt

  Log Message:
  -----------
  replay: update docs for record/replay with block devices

This patch updates the description of the command lines for using
record/replay with attached block devices.

Signed-off-by: Pavel Dovgalyuk <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: c8aa7895eb536e1fc9ef2740e4e32bb9ec6f3afa
      
https://github.com/qemu/qemu/commit/c8aa7895eb536e1fc9ef2740e4e32bb9ec6f3afa
  Author: Pavel Dovgalyuk <address@hidden>
  Date:   2019-10-14 (Mon, 14 Oct 2019)

  Changed paths:
    M block/io.c
    M cpus.c

  Log Message:
  -----------
  replay: don't drain/flush bdrv queue while RR is working

In record/replay mode bdrv queue is controlled by replay mechanism.
It does not allow saving or loading the snapshots
when bdrv queue is not empty. Stopping the VM is not blocked by nonempty
queue, but flushing the queue is still impossible there,
because it may cause deadlocks in replay mode.
This patch disables bdrv_drain_all and bdrv_flush_all in
record/replay mode.

Stopping the machine when the IO requests are not finished is needed
for the debugging. E.g., breakpoint may be set at the specified step,
and forcing the IO requests to finish may break the determinism
of the execution.

Signed-off-by: Pavel Dovgalyuk <address@hidden>
Acked-by: Kevin Wolf <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: ae25dccb7df64ec9cbb52d7a0a67308d74372cfd
      
https://github.com/qemu/qemu/commit/ae25dccb7df64ec9cbb52d7a0a67308d74372cfd
  Author: Pavel Dovgalyuk <address@hidden>
  Date:   2019-10-14 (Mon, 14 Oct 2019)

  Changed paths:
    M replay/replay.c
    M vl.c

  Log Message:
  -----------
  replay: finish record/replay before closing the disks

After recent updates block devices cannot be closed on qemu exit.
This happens due to the block request polling when replay is not finished.
Therefore now we stop execution recording before closing the block devices.

Signed-off-by: Pavel Dovgalyuk <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: e4ec5ad464e48ab8d978b4dd8aacd05c1c4a87cc
      
https://github.com/qemu/qemu/commit/e4ec5ad464e48ab8d978b4dd8aacd05c1c4a87cc
  Author: Pavel Dovgalyuk <address@hidden>
  Date:   2019-10-14 (Mon, 14 Oct 2019)

  Changed paths:
    M block/block-backend.c
    M block/io.c
    M block/iscsi.c
    M block/nfs.c
    M block/null.c
    M block/nvme.c
    M block/rbd.c
    M block/vxhs.c
    M include/sysemu/replay.h
    M replay/replay-events.c
    M replay/replay-internal.h
    M stubs/Makefile.objs
    A stubs/replay-user.c

  Log Message:
  -----------
  replay: add BH oneshot event for block layer

Replay is capable of recording normal BH events, but sometimes
there are single use callbacks scheduled with aio_bh_schedule_oneshot
function. This patch enables recording and replaying such callbacks.
Block layer uses these events for calling the completion function.
Replaying these calls makes the execution deterministic.

Signed-off-by: Pavel Dovgalyuk <address@hidden>
Acked-by: Kevin Wolf <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: f2208fdc5ba2f4d0cbc023e2530b5e3e110d71c2
      
https://github.com/qemu/qemu/commit/f2208fdc5ba2f4d0cbc023e2530b5e3e110d71c2
  Author: Alberto Garcia <address@hidden>
  Date:   2019-10-14 (Mon, 14 Oct 2019)

  Changed paths:
    M block/io.c
    A tests/qemu-iotests/268
    A tests/qemu-iotests/268.out
    M tests/qemu-iotests/group

  Log Message:
  -----------
  block: Reject misaligned write requests with BDRV_REQ_NO_FALLBACK

The BDRV_REQ_NO_FALLBACK flag means that an operation should only be
performed if it can be offloaded or otherwise performed efficiently.

However a misaligned write request requires a RMW so we should return
an error and let the caller decide how to proceed.

This hits an assertion since commit c8bb23cbdb if the required
alignment is larger than the cluster size:

qemu-img create -f qcow2 -o cluster_size=2k img.qcow2 4G
qemu-io -c "open -o driver=qcow2,file.align=4k blkdebug::img.qcow2" \
        -c 'write 0 512'
qemu-io: block/io.c:1127: bdrv_driver_pwritev: Assertion `!(flags & 
BDRV_REQ_NO_FALLBACK)' failed.
Aborted

The reason is that when writing to an unallocated cluster we try to
skip the copy-on-write part and zeroize it using BDRV_REQ_NO_FALLBACK
instead, resulting in a write request that is too small (2KB cluster
size vs 4KB required alignment).

Signed-off-by: Alberto Garcia <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 48c8d3ce6daad66d86ba972e8930adc54c02db1a
      
https://github.com/qemu/qemu/commit/48c8d3ce6daad66d86ba972e8930adc54c02db1a
  Author: Max Reitz <address@hidden>
  Date:   2019-10-14 (Mon, 14 Oct 2019)

  Changed paths:
    M tests/qemu-iotests/028
    M tests/qemu-iotests/028.out

  Log Message:
  -----------
  iotests/028: Fix for long $TEST_DIRs

For long test image paths, the order of the "Formatting" line and the
"(qemu)" prompt after a drive_backup HMP command may be reversed.  In
fact, the interaction between the prompt and the line may lead to the
"Formatting" to being greppable at all after "read"-ing it (if the
prompt injects an IFS character into the "Formatting" string).

So just wait until we get a prompt.  At that point, the block job must
have been started, so "info block-jobs" will only return "No active
jobs" once it is done.

Reported-by: Thomas Huth <address@hidden>
Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: John Snow <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 3e9297f3659701a72110f0f560b4cc22452972f1
      
https://github.com/qemu/qemu/commit/3e9297f3659701a72110f0f560b4cc22452972f1
  Author: Kevin Wolf <address@hidden>
  Date:   2019-10-14 (Mon, 14 Oct 2019)

  Changed paths:
    M include/qom/object_interfaces.h
    M qom/object_interfaces.c
    M vl.c

  Log Message:
  -----------
  vl: Split off user_creatable_print_help()

Printing help for --object is something that we not only want in the
system emulator, but also in tools that support --object. Move it into a
separate function in qom/object_interfaces.c to make the code accessible
for tools.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Eric Blake <address@hidden>


  Commit: 4fa1f0dc052968b41d2bdffd9cb9fcab8137cdaf
      
https://github.com/qemu/qemu/commit/4fa1f0dc052968b41d2bdffd9cb9fcab8137cdaf
  Author: Kevin Wolf <address@hidden>
  Date:   2019-10-14 (Mon, 14 Oct 2019)

  Changed paths:
    M qemu-io.c

  Log Message:
  -----------
  qemu-io: Support help options for --object

Instead of parsing help options as normal object properties and
returning an error, provide the same help functionality as the system
emulator in qemu-io, too.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Eric Blake <address@hidden>


  Commit: c6e5cdfd4b06918a6bcefa2039af7253dbd5e3c0
      
https://github.com/qemu/qemu/commit/c6e5cdfd4b06918a6bcefa2039af7253dbd5e3c0
  Author: Kevin Wolf <address@hidden>
  Date:   2019-10-14 (Mon, 14 Oct 2019)

  Changed paths:
    M qemu-img.c

  Log Message:
  -----------
  qemu-img: Support help options for --object

Instead of parsing help options as normal object properties and
returning an error, provide the same help functionality as the system
emulator in qemu-img, too.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Eric Blake <address@hidden>


  Commit: 495bf893b0db2621726815b5625a6a788d3958aa
      
https://github.com/qemu/qemu/commit/495bf893b0db2621726815b5625a6a788d3958aa
  Author: Kevin Wolf <address@hidden>
  Date:   2019-10-14 (Mon, 14 Oct 2019)

  Changed paths:
    M qemu-nbd.c

  Log Message:
  -----------
  qemu-nbd: Support help options for --object

Instead of parsing help options as normal object properties and
returning an error, provide the same help functionality as the system
emulator in qemu-nbd, too.

Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Eric Blake <address@hidden>


  Commit: d1b9d19f99586b33795e20a79f645186ccbc070f
      
https://github.com/qemu/qemu/commit/d1b9d19f99586b33795e20a79f645186ccbc070f
  Author: Max Reitz <address@hidden>
  Date:   2019-10-14 (Mon, 14 Oct 2019)

  Changed paths:
    M block/qcow2-cluster.c

  Log Message:
  -----------
  qcow2: Limit total allocation range to INT_MAX

When the COW areas are included, the size of an allocation can exceed
INT_MAX.  This is kind of limited by handle_alloc() in that it already
caps avail_bytes at INT_MAX, but the number of clusters still reflects
the original length.

This can have all sorts of effects, ranging from the storage layer write
call failing to image corruption.  (If there were no image corruption,
then I suppose there would be data loss because the .cow_end area is
forced to be empty, even though there might be something we need to
COW.)

Fix all of it by limiting nb_clusters so the equivalent number of bytes
will not exceed INT_MAX.

Cc: address@hidden
Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Reviewed-by: Philippe Mathieu-Daudé <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: a1406a9262a087d9ec9627b88da13c4590b61dae
      
https://github.com/qemu/qemu/commit/a1406a9262a087d9ec9627b88da13c4590b61dae
  Author: Max Reitz <address@hidden>
  Date:   2019-10-14 (Mon, 14 Oct 2019)

  Changed paths:
    A tests/qemu-iotests/270
    A tests/qemu-iotests/270.out
    M tests/qemu-iotests/group

  Log Message:
  -----------
  iotests: Test large write request to qcow2 file

Without HEAD^, the following happens when you attempt a large write
request to a qcow2 file such that the number of bytes covered by all
clusters involved in a single allocation will exceed INT_MAX:

(A) handle_alloc_space() decides to fill the whole area with zeroes and
    fails because bdrv_co_pwrite_zeroes() fails (the request is too
    large).

(B) If handle_alloc_space() does not do anything, but merge_cow()
    decides that the requests can be merged, it will create a too long
    IOV that later cannot be written.

(C) Otherwise, all parts will be written separately, so those requests
    will work.

In either B or C, though, qcow2_alloc_cluster_link_l2() will have an
overflow: We use an int (i) to iterate over nb_clusters, and then
calculate the L2 entry based on "i << s->cluster_bits" -- which will
overflow if the range covers more than INT_MAX bytes.  This then leads
to image corruption because the L2 entry will be wrong (it will be
recognized as a compressed cluster).

Even if that were not the case, the .cow_end area would be empty
(because handle_alloc() will cap avail_bytes and nb_bytes at INT_MAX, so
their difference (which is the .cow_end size) will be 0).

So this test checks that on such large requests, the image will not be
corrupted.  Unfortunately, we cannot check whether COW will be handled
correctly, because that data is discarded when it is written to null-co
(but we have to use null-co, because writing 2 GB of data in a test is
not quite reasonable).

Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 3af78db68176a049e2570822f64604e0692c1447
      
https://github.com/qemu/qemu/commit/3af78db68176a049e2570822f64604e0692c1447
  Author: Peter Maydell <address@hidden>
  Date:   2019-10-15 (Tue, 15 Oct 2019)

  Changed paths:
    M block/blkreplay.c
    M block/block-backend.c
    M block/io.c
    M block/iscsi.c
    M block/nfs.c
    M block/null.c
    M block/nvme.c
    M block/qcow2-cluster.c
    M block/rbd.c
    M block/vhdx.c
    M block/vxhs.c
    M cpus.c
    M docs/replay.txt
    M include/qom/object_interfaces.h
    M include/sysemu/replay.h
    M qemu-img.c
    M qemu-io.c
    M qemu-nbd.c
    M qom/object_interfaces.c
    M replay/replay-events.c
    M replay/replay-internal.h
    M replay/replay.c
    M stubs/Makefile.objs
    A stubs/replay-user.c
    M tests/qemu-iotests/028
    M tests/qemu-iotests/028.out
    A tests/qemu-iotests/268
    A tests/qemu-iotests/268.out
    A tests/qemu-iotests/270
    A tests/qemu-iotests/270.out
    M tests/qemu-iotests/group
    M vl.c

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- block: Fix crash with qcow2 partial cluster COW with small cluster
  sizes (misaligned write requests with BDRV_REQ_NO_FALLBACK)
- qcow2: Fix integer overflow potentially causing corruption with huge
  requests
- vhdx: Detect truncated image files
- tools: Support help options for --object
- Various block-related replay improvements
- iotests/028: Fix for long $TEST_DIRs

# gpg: Signature made Mon 14 Oct 2019 17:02:54 BST
# gpg:                using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <address@hidden>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74  56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream:
  iotests: Test large write request to qcow2 file
  qcow2: Limit total allocation range to INT_MAX
  qemu-nbd: Support help options for --object
  qemu-img: Support help options for --object
  qemu-io: Support help options for --object
  vl: Split off user_creatable_print_help()
  iotests/028: Fix for long $TEST_DIRs
  block: Reject misaligned write requests with BDRV_REQ_NO_FALLBACK
  replay: add BH oneshot event for block layer
  replay: finish record/replay before closing the disks
  replay: don't drain/flush bdrv queue while RR is working
  replay: update docs for record/replay with block devices
  replay: disable default snapshot for record/replay
  block: implement bdrv_snapshot_goto for blkreplay
  block/vhdx: add check for truncated image files

Signed-off-by: Peter Maydell <address@hidden>


Compare: https://github.com/qemu/qemu/compare/9020e9526cd0...3af78db68176