qemu-commits
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Qemu-commits] [qemu/qemu] 3ae3fc: docs: improve qcow2 spec about extend


From: Peter Maydell
Subject: [Qemu-commits] [qemu/qemu] 3ae3fc: docs: improve qcow2 spec about extending image header
Date: Fri, 21 Feb 2020 06:30:12 -0800

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 3ae3fcfae533086d3e210589c10f317717c9d789
      
https://github.com/qemu/qemu/commit/3ae3fcfae533086d3e210589c10f317717c9d789
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M docs/interop/qcow2.txt

  Log Message:
  -----------
  docs: improve qcow2 spec about extending image header

Make it more obvious how to add new fields to the version 3 header and
how to interpret them.

The specification is adjusted so that for new defined optional fields:

1. Software may support some of these optional fields and ignore the
   others, which means that features may be backported to downstream
   Qemu independently.
2. If we want to add incompatible field (or a field, for which some of
   its values would be incompatible), it must be accompanied by
   incompatible feature bit.

Also the concept of "default is zero" is clarified, as it's strange to
say that the value of the field is assumed to be zero for the software
version which don't know about the field at all and don't know how to
treat it be it zero or not.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Alberto Garcia <address@hidden>
[mreitz: s/some its/some of its/]
Signed-off-by: Max Reitz <address@hidden>


  Commit: 66fcbca5a7202d2ff2973b61a30f915204917eb7
      
https://github.com/qemu/qemu/commit/66fcbca5a7202d2ff2973b61a30f915204917eb7
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M docs/interop/qcow2.txt

  Log Message:
  -----------
  docs: qcow2: introduce compression type feature

The patch adds a new additional field to the qcow2 header: compression_type,
which specifies compression type. If field is absent or zero, default
compression type is set: ZLIB, which corresponds to current behavior.

New compression type (ZSTD) is to be added in further commit.

Suggested-by: Denis Plotnikov <address@hidden>
Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-Id: <address@hidden>
[mreitz: s/Bits 3-63:  Reserved/Bits 4-63:  Reserved/]
Signed-off-by: Max Reitz <address@hidden>


  Commit: 6756696e38117f9e7c769dd33b339f5ab71f7d47
      
https://github.com/qemu/qemu/commit/6756696e38117f9e7c769dd33b339f5ab71f7d47
  Author: Thomas Huth <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M tests/qemu-iotests/139

  Log Message:
  -----------
  iotests: Remove the superfluous 2nd check for the availability of quorum

Commit d9df28e7b07 ("iotests: check whitelisted formats") added the
modern @iotests.skip_if_unsupported() to the functions in this test,
so we don't need the old explicit test here anymore.

Signed-off-by: Thomas Huth <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Alberto Garcia <address@hidden>
Reviewed-by: Andrey Shinkevich <address@hidden>
Tested-by: Andrey Shinkevich <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: b0c4cf21b0f26533e4aa6ca1930b2c9f52371bfa
      
https://github.com/qemu/qemu/commit/b0c4cf21b0f26533e4aa6ca1930b2c9f52371bfa
  Author: Max Reitz <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M tests/qemu-iotests/147

  Log Message:
  -----------
  iotests/147: Fix drive parameters

8dff69b94 added an aio parameter to the drive parameter but forgot to
add a comma before, thus breaking the test.  Fix it again.

Fixes: 8dff69b9415b4287e900358744b732195e1ab2e2
Signed-off-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Tested-by: Eric Blake <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: facda5443f5a8676fb635b82ac1046ac6b6a67ce
      
https://github.com/qemu/qemu/commit/facda5443f5a8676fb635b82ac1046ac6b6a67ce
  Author: Peter Krempa <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M block.c
    M block/qapi.c
    M blockdev.c
    M include/block/block.h
    M include/block/qapi.h
    M monitor/hmp-cmds.c
    M qapi/block-core.json

  Log Message:
  -----------
  qapi: Allow getting flat output from 'query-named-block-nodes'

When a management application manages node names there's no reason to
recurse into backing images in the output of query-named-block-nodes.

Add a parameter to the command which will return just the top level
structs.

Signed-off-by: Peter Krempa <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
[mreitz: Fixed coding style]
Signed-off-by: Max Reitz <address@hidden>


  Commit: 168468fe19c872d31d17581e6a6cf7b69bd105b1
      
https://github.com/qemu/qemu/commit/168468fe19c872d31d17581e6a6cf7b69bd105b1
  Author: David Edmondson <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M docs/interop/qemu-img.rst
    M qemu-img-cmds.hx
    M qemu-img.c

  Log Message:
  -----------
  qemu-img: Add --target-is-zero to convert

In many cases the target of a convert operation is a newly provisioned
target that the user knows is blank (reads as zero). In this situation
there is no requirement for qemu-img to wastefully zero out the entire
device.

Add a new option, --target-is-zero, allowing the user to indicate that
an existing target device will return zeros for all reads.

Signed-off-by: David Edmondson <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 087ab8e775f48766068e65de1bc99d03b40d1670
      
https://github.com/qemu/qemu/commit/087ab8e775f48766068e65de1bc99d03b40d1670
  Author: Daniel P. Berrangé <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M block/qcow2.c
    A tests/qemu-iotests/284
    A tests/qemu-iotests/284.out
    M tests/qemu-iotests/group

  Log Message:
  -----------
  block: always fill entire LUKS header space with zeros

When initializing the LUKS header the size with default encryption
parameters will currently be 2068480 bytes. This is rounded up to
a multiple of the cluster size, 2081792, with 64k sectors. If the
end of the header is not the same as the end of the cluster we fill
the extra space with zeros. This was forgetting that not even the
space allocated for the header will be fully initialized, as we
only write key material for the first key slot. The space left
for the other 7 slots is never written to.

An optimization to the ref count checking code:

  commit a5fff8d4b4d928311a5005efa12d0991fe3b66f9 (refs/bisect/bad)
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   Wed Feb 27 16:14:30 2019 +0300

    qcow2-refcount: avoid eating RAM

made the assumption that every cluster which was allocated would
have at least some data written to it. This was violated by way
the LUKS header is only partially written, with much space simply
reserved for future use.

Depending on the cluster size this problem was masked by the
logic which wrote zeros between the end of the LUKS header and
the end of the cluster.

$ qemu-img create --object secret,id=cluster_encrypt0,data=123456 \
   -f qcow2 -o cluster_size=2k,encrypt.iter-time=1,\
               encrypt.format=luks,encrypt.key-secret=cluster_encrypt0 \
               cluster_size_check.qcow2 100M
  Formatting 'cluster_size_check.qcow2', fmt=qcow2 size=104857600
    encrypt.format=luks encrypt.key-secret=cluster_encrypt0
    encrypt.iter-time=1 cluster_size=2048 lazy_refcounts=off refcount_bits=16

$ qemu-img check --object secret,id=cluster_encrypt0,data=redhat \
    'json:{"driver": "qcow2", "encrypt.format": "luks", \
           "encrypt.key-secret": "cluster_encrypt0", \
           "file.driver": "file", "file.filename": "cluster_size_check.qcow2"}'
ERROR: counting reference for region exceeding the end of the file by one 
cluster or more: offset 0x2000 size 0x1f9000
Leaked cluster 4 refcount=1 reference=0
...snip...
Leaked cluster 130 refcount=1 reference=0

1 errors were found on the image.
Data may be corrupted, or further writes to the image may corrupt it.

127 leaked clusters were found on the image.
This means waste of disk space, but no harm to data.
Image end offset: 268288

The problem only exists when the disk image is entirely empty. Writing
data to the disk image payload will solve the problem by causing the
end of the file to be extended further.

The change fixes it by ensuring that the entire allocated LUKS header
region is fully initialized with zeros. The qemu-img check will still
fail for any pre-existing disk images created prior to this change,
unless at least 1 byte of the payload is written to.

Fully writing zeros to the entire LUKS header is a good idea regardless
as it ensures that space has been allocated on the host filesystem (or
whatever block storage backend is used).

Signed-off-by: Daniel P. Berrangé <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 4bc267a7c7cf204b05d3d1f01625691540a7b965
      
https://github.com/qemu/qemu/commit/4bc267a7c7cf204b05d3d1f01625691540a7b965
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M block/backup-top.c

  Log Message:
  -----------
  block/backup-top: fix flags handling

backup-top "supports" write-unchanged, by skipping CBW operation in
backup_top_co_pwritev. But it forgets to do the same in
backup_top_co_pwrite_zeroes, as well as declare support for
BDRV_REQ_WRITE_UNCHANGED.

Fix this, and, while being here, declare also support for flags
supported by source child.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: b7e9eae98c160f3e7eb9ffbedba8821e358d963c
      
https://github.com/qemu/qemu/commit/b7e9eae98c160f3e7eb9ffbedba8821e358d963c
  Author: Max Reitz <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M tests/qemu-iotests/279

  Log Message:
  -----------
  iotests/279: Fix for non-qcow2 formats

First, driver=qcow2 will not work so well with non-qcow2 formats (and
this test claims to support qcow, qed, and vmdk).

Second, vmdk will always report the backing file format to be vmdk.
Filter that out so the output looks like for all other formats.

Third, the flat vmdk subformats do not support backing files, so they
will not work with this test.

Signed-off-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Tested-by: Thomas Huth <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 78c81a3f108870d325b0a39d88711366afe6f703
      
https://github.com/qemu/qemu/commit/78c81a3f108870d325b0a39d88711366afe6f703
  Author: Max Reitz <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M block/nbd.c

  Log Message:
  -----------
  block/nbd: Fix hang in .bdrv_close()

When nbd_close() is called from a coroutine, the connection_co never
gets to run, and thus nbd_teardown_connection() hangs.

This is because aio_co_enter() only puts the connection_co into the main
coroutine's wake-up queue, so this main coroutine needs to yield and
wait for connection_co to terminate.

Suggested-by: Kevin Wolf <address@hidden>
Signed-off-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Reviewed-by: Maxim Levitsky <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: fd17146cd93d1704cd96d7c2757b325fc7aac6fd
      
https://github.com/qemu/qemu/commit/fd17146cd93d1704cd96d7c2757b325fc7aac6fd
  Author: Max Reitz <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M block.c

  Log Message:
  -----------
  block: Generic file creation fallback

If a protocol driver does not support image creation, we can see whether
maybe the file exists already.  If so, just truncating it will be
sufficient.

Signed-off-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 87ca3b8fa615b278b33cabf9ed22b3f44b5214ba
      
https://github.com/qemu/qemu/commit/87ca3b8fa615b278b33cabf9ed22b3f44b5214ba
  Author: Max Reitz <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M block/file-posix.c

  Log Message:
  -----------
  file-posix: Drop hdev_co_create_opts()

The generic fallback implementation effectively does the same.

Reviewed-by: Maxim Levitsky <address@hidden>
Signed-off-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 80f0900905b555f00d644894c786b6d66ac2e00e
      
https://github.com/qemu/qemu/commit/80f0900905b555f00d644894c786b6d66ac2e00e
  Author: Max Reitz <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M block/iscsi.c

  Log Message:
  -----------
  iscsi: Drop iscsi_co_create_opts()

The generic fallback implementation effectively does the same.

Reviewed-by: Maxim Levitsky <address@hidden>
Signed-off-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 4dddeac115c5a2c5f74731fda0afd031a0b45490
      
https://github.com/qemu/qemu/commit/4dddeac115c5a2c5f74731fda0afd031a0b45490
  Author: Max Reitz <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    A tests/qemu-iotests/259
    A tests/qemu-iotests/259.out
    M tests/qemu-iotests/group

  Log Message:
  -----------
  iotests: Add test for image creation fallback

Signed-off-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Reviewed-by: Maxim Levitsky <address@hidden>
[mreitz: Added a note that NBD does not support resizing, which is why
         the second case is expected to fail]
Signed-off-by: Max Reitz <address@hidden>


  Commit: c69291e712ae4ef95f628424db6586473da61d43
      
https://github.com/qemu/qemu/commit/c69291e712ae4ef95f628424db6586473da61d43
  Author: Max Reitz <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M qemu-img.c

  Log Message:
  -----------
  qemu-img: Fix convert -n -B for backing-less targets

s.target_has_backing does not reflect whether the target BDS has a
backing file; it only tells whether we should use a backing file during
conversion (specified by -B).

As such, if you use convert -n, the target does not necessarily actually
have a backing file, and then dereferencing out_bs->backing fails here.

When converting to an existing file, we should set
target_backing_sectors to a negative value, because first, as the
comment explains, this value is only used for optimization, so it is
always fine to do that.

Second, we use this value to determine where the target must be
initialized to zeroes (overlays are initialized to zero after the end of
their backing file).  When converting to an existing file, we cannot
assume that to be true.

Cc: address@hidden
Fixes: 351c8efff9ad809c822d55620df54d575d536f68
       ("qemu-img: Special post-backing convert handling")
Signed-off-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: John Snow <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: f535cc90b50b1be2e6c204b844eab991c872b78b
      
https://github.com/qemu/qemu/commit/f535cc90b50b1be2e6c204b844eab991c872b78b
  Author: Max Reitz <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M tests/qemu-iotests/122
    M tests/qemu-iotests/122.out

  Log Message:
  -----------
  iotests: Test convert -n -B to backing-less target

This must not crash.

Signed-off-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: John Snow <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 804359b8b90f76d9d8fbe8d85a6544b68f107f10
      
https://github.com/qemu/qemu/commit/804359b8b90f76d9d8fbe8d85a6544b68f107f10
  Author: Max Reitz <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    M block/qapi.c

  Log Message:
  -----------
  block: Fix VM size field width in snapshot dump

When printing the snapshot list (e.g. with qemu-img snapshot -l), the VM
size field is only seven characters wide.  As of de38b5005e9, this is
not necessarily sufficient: We generally print three digits, and this
may require a decimal point.  Also, the unit field grew from something
as plain as "M" to " MiB".  This means that number and unit may take up
eight characters in total; but we also want spaces in front.

Considering previously the maximum width was four characters and the
field width was chosen to be three characters wider, let us adjust the
field width to be eleven now.

Fixes: de38b5005e946aa3714963ea4c501e279e7d3666
Buglink: https://bugs.launchpad.net/qemu/+bug/1859989
Signed-off-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: dff8d44c96f128480430b0c59ed8760917dbd427
      
https://github.com/qemu/qemu/commit/dff8d44c96f128480430b0c59ed8760917dbd427
  Author: Max Reitz <address@hidden>
  Date:   2020-02-20 (Thu, 20 Feb 2020)

  Changed paths:
    A tests/qemu-iotests/286
    A tests/qemu-iotests/286.out
    M tests/qemu-iotests/group

  Log Message:
  -----------
  iotests: Test snapshot -l field separation

Add a test that all fields in "qemu-img snapshot -l"s output are
separated by spaces.

Signed-off-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
[mreitz: Renamed test from 284 to 286]
Signed-off-by: Max Reitz <address@hidden>


  Commit: 9e6b7f7eb89580cd8dfc1dd4983db9c158517d0a
      
https://github.com/qemu/qemu/commit/9e6b7f7eb89580cd8dfc1dd4983db9c158517d0a
  Author: Peter Maydell <address@hidden>
  Date:   2020-02-21 (Fri, 21 Feb 2020)

  Changed paths:
    M block.c
    M block/backup-top.c
    M block/file-posix.c
    M block/iscsi.c
    M block/nbd.c
    M block/qapi.c
    M block/qcow2.c
    M blockdev.c
    M docs/interop/qcow2.txt
    M docs/interop/qemu-img.rst
    M include/block/block.h
    M include/block/qapi.h
    M monitor/hmp-cmds.c
    M qapi/block-core.json
    M qemu-img-cmds.hx
    M qemu-img.c
    M tests/qemu-iotests/122
    M tests/qemu-iotests/122.out
    M tests/qemu-iotests/139
    M tests/qemu-iotests/147
    A tests/qemu-iotests/259
    A tests/qemu-iotests/259.out
    M tests/qemu-iotests/279
    A tests/qemu-iotests/284
    A tests/qemu-iotests/284.out
    A tests/qemu-iotests/286
    A tests/qemu-iotests/286.out
    M tests/qemu-iotests/group

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-02-20' 
into staging

Block patches:
- qemu-img convert: New --target-is-zero parameter
- qcow2: Specify non-default compression type flag
- optionally flat output for query-named-block-nodes
- some fixes
- pseudo-creation of images on block devices is now done by a generic
  block layer function

# gpg: Signature made Thu 20 Feb 2020 16:05:34 GMT
# gpg:                using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg:                issuer "address@hidden"
# gpg: Good signature from "Max Reitz <address@hidden>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1  1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-02-20:
  iotests: Test snapshot -l field separation
  block: Fix VM size field width in snapshot dump
  iotests: Test convert -n -B to backing-less target
  qemu-img: Fix convert -n -B for backing-less targets
  iotests: Add test for image creation fallback
  iscsi: Drop iscsi_co_create_opts()
  file-posix: Drop hdev_co_create_opts()
  block: Generic file creation fallback
  block/nbd: Fix hang in .bdrv_close()
  iotests/279: Fix for non-qcow2 formats
  block/backup-top: fix flags handling
  block: always fill entire LUKS header space with zeros
  qemu-img: Add --target-is-zero to convert
  qapi: Allow getting flat output from 'query-named-block-nodes'
  iotests/147: Fix drive parameters
  iotests: Remove the superfluous 2nd check for the availability of quorum
  docs: qcow2: introduce compression type feature
  docs: improve qcow2 spec about extending image header

Signed-off-by: Peter Maydell <address@hidden>


Compare: https://github.com/qemu/qemu/compare/b651b80822fa...9e6b7f7eb895



reply via email to

[Prev in Thread] Current Thread [Next in Thread]