[Qemu-commits] [qemu/qemu] ecbda7: qcow2: Flush pending discards before allocating cl...

[GitHub]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: ecbda7a22576591a84f44de1be0150faf6001f1c
      
https://github.com/qemu/qemu/commit/ecbda7a22576591a84f44de1be0150faf6001f1c
  Author: Kevin Wolf <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M block/qcow2-refcount.c

  Log Message:
  -----------
  qcow2: Flush pending discards before allocating cluster

Before a freed cluster can be reused, pending discards for this cluster
must be processed.

The original assumption was that this was not a problem because discards
are only cached during discard/write zeroes operations, which are
synchronous so that no concurrent write requests can cause cluster
allocations.

However, the discard/write zeroes operation itself can allocate a new L2
table (and it has to in order to put zero flags there), so make sure we
can cope with the situation.

This fixes https://bugs.launchpad.net/bugs/1349972.

Cc: address@hidden
Signed-off-by: Kevin Wolf <address@hidden>
Reviewed-by: Max Reitz <address@hidden>


  Commit: aacd5650c68ef2e9e19079ba60cb0df51e15880c
      
https://github.com/qemu/qemu/commit/aacd5650c68ef2e9e19079ba60cb0df51e15880c
  Author: Christoph Hellwig <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M hw/block/nvme.c

  Log Message:
  -----------
  nvme: support NVME_VOLATILE_WRITE_CACHE feature

The SCSI emulation in the Linux NVMe driver really wants to know
if a device has a volatile write cache.  Given that qemu has moved
away from a model where we report the backing store WCE bit to
one where the WCE bit is supposed to be part of the migratable
guest-visible state we always return 1 here.

Signed-off-by: Christoph Hellwig <address@hidden>
Acked-by: Keith Busch <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 5e82a31eb967db135fc4e688b134fb0972d62de3
      
https://github.com/qemu/qemu/commit/5e82a31eb967db135fc4e688b134fb0972d62de3
  Author: Fam Zheng <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M block/vmdk.c

  Log Message:
  -----------
  vmdk: Fix next_cluster_sector for compressed write

This fixes the bug introduced by commit c6ac36e (vmdk: Optimize cluster
allocation).

Sometimes, write_len could be larger than cluster size, because it
contains both data and marker.  We must advance next_cluster_sector in
this case, otherwise the image gets corrupted.

Cc: address@hidden
Reported-by: Antoni Villalonga <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 13c4941cdd8685d28c7e3a09e393a5579b58db46
      
https://github.com/qemu/qemu/commit/13c4941cdd8685d28c7e3a09e393a5579b58db46
  Author: Fam Zheng <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M block/vmdk.c

  Log Message:
  -----------
  vmdk: Fix overflow if l1_size is 0x20000000

Richard Jones caught this bug with afl fuzzer.

In fact, that's the only possible value to overflow (extent->l1_size =
0x20000000) l1_size:

l1_size = extent->l1_size * sizeof(long) => 0x80000000;

g_try_malloc returns NULL because l1_size is interpreted as negative
during type casting from 'int' to 'gsize', which yields a enormous
value. Hence, by coincidence, we get a "not too bad" behavior:

qemu-img: Could not open '/tmp/afl6.img': Could not open
'/tmp/afl6.img': Cannot allocate memory

Values larger than 0x20000000 will be refused by the validation in
vmdk_add_extent.

Values smaller than 0x20000000 will not overflow l1_size.

Cc: address@hidden
Reported-by: Richard W.M. Jones <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Tested-by: Richard W.M. Jones <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 72e80b89015bab196f0f0e83b12b0eee75fa0574
      
https://github.com/qemu/qemu/commit/72e80b89015bab196f0f0e83b12b0eee75fa0574
  Author: Alberto Garcia <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M block/qcow2-cache.c
    M block/qcow2-cluster.c
    M block/qcow2-refcount.c
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: use one single memory block for the L2/refcount cache tables

The qcow2 L2/refcount cache contains one separate table for each cache
entry. Doing one allocation per table adds unnecessary overhead and it
also requires us to store the address of each table separately.

Since the size of the cache is constant during its lifetime, it's
better to have an array that contains all the tables using one single
allocation.

In my tests measuring freshly created caches with sizes 128MB (L2) and
32MB (refcount) this uses around 10MB of RAM less.

Signed-off-by: Alberto Garcia <address@hidden>
Reviewed-by: Stefan Hajnoczi <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: baf07d60f5c5d5d0f0c9e844cde75691f1ceb3d1
      
https://github.com/qemu/qemu/commit/baf07d60f5c5d5d0f0c9e844cde75691f1ceb3d1
  Author: Alberto Garcia <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M block/qcow2-cache.c

  Log Message:
  -----------
  qcow2: simplify qcow2_cache_put() and qcow2_cache_entry_mark_dirty()

Since all tables are now stored together, it is possible to obtain
the position of a particular table directly from its address, so the
operation becomes O(1).

Signed-off-by: Alberto Garcia <address@hidden>
Reviewed-by: Stefan Hajnoczi <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 2693310eccccf8351678ddd6f3b050163e51dba0
      
https://github.com/qemu/qemu/commit/2693310eccccf8351678ddd6f3b050163e51dba0
  Author: Alberto Garcia <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M block/qcow2-cache.c

  Log Message:
  -----------
  qcow2: use an LRU algorithm to replace entries from the L2 cache

The current algorithm to evict entries from the cache gives always
preference to those in the lowest positions. As the size of the cache
increases, the chances of the later elements of being removed decrease
exponentially.

In a scenario with random I/O and lots of cache misses, entries in
positions 8 and higher are rarely (if ever) evicted. This can be seen
even with the default cache size, but with larger caches the problem
becomes more obvious.

Using an LRU algorithm makes the chances of being removed from the
cache independent from the position.

Signed-off-by: Alberto Garcia <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: fdfbca82a0874916007ca76323cd35f2af8a2ef3
      
https://github.com/qemu/qemu/commit/fdfbca82a0874916007ca76323cd35f2af8a2ef3
  Author: Alberto Garcia <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M block/qcow2-cache.c

  Log Message:
  -----------
  qcow2: remove qcow2_cache_find_entry_to_replace()

A cache miss means that the whole array was traversed and the entry
we were looking for was not found, so there's no need to traverse it
again in order to select an entry to replace.

Signed-off-by: Alberto Garcia <address@hidden>
Reviewed-by: Stefan Hajnoczi <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 812e4082cae73e12fd425cace4fd3a715a7c1d32
      
https://github.com/qemu/qemu/commit/812e4082cae73e12fd425cace4fd3a715a7c1d32
  Author: Alberto Garcia <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M block/qcow2-cache.c

  Log Message:
  -----------
  qcow2: use a hash to look for entries in the L2 cache

The current cache algorithm traverses the array starting always from
the beginning, so the average number of comparisons needed to perform
a lookup is proportional to the size of the array.

By using a hash of the offset as the starting point, lookups are
faster and independent from the array size.

The hash is computed using the cluster number of the table, multiplied
by 4 to make it perform better when there are collisions.

In my tests, using a cache with 2048 entries, this reduces the average
number of comparisons per lookup from 430 to 2.5.

Signed-off-by: Alberto Garcia <address@hidden>
Reviewed-by: Stefan Hajnoczi <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: a3f1afb43a09e4577571c044c48f2ba9e6e4ad06
      
https://github.com/qemu/qemu/commit/a3f1afb43a09e4577571c044c48f2ba9e6e4ad06
  Author: Alberto Garcia <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M block/qcow2-cache.c
    M block/qcow2-cluster.c
    M block/qcow2-refcount.c
    M block/qcow2.h

  Log Message:
  -----------
  qcow2: make qcow2_cache_put() a void function

This function never receives an invalid table pointer, so we can make
it void and remove all the error checking code.

Signed-off-by: Alberto Garcia <address@hidden>
Reviewed-by: Stefan Hajnoczi <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: d1b4efe5c4088fd2289e39b95bbdf73b3dcb7432
      
https://github.com/qemu/qemu/commit/d1b4efe5c4088fd2289e39b95bbdf73b3dcb7432
  Author: Alberto Garcia <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M block/qcow2-cache.c

  Log Message:
  -----------
  qcow2: style fixes in qcow2-cache.c

Fix pointer declaration to make it consistent with the rest of the
code.

Signed-off-by: Alberto Garcia <address@hidden>
Reviewed-by: Stefan Hajnoczi <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: b062ad86dcd33ab39be5060b0655d8e13834b167
      
https://github.com/qemu/qemu/commit/b062ad86dcd33ab39be5060b0655d8e13834b167
  Author: Eric Blake <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M qemu-io-cmds.c
    M qemu-io.c

  Log Message:
  -----------
  qemu-io: Use getopt() correctly

POSIX says getopt() returns -1 on completion.  While Linux happens
to define EOF as -1, this definition is not required by POSIX, and
there may be platforms where checking for EOF instead of -1 would
lead to an infinite loop.

Signed-off-by: Eric Blake <address@hidden>
Reviewed-by: Alberto Garcia <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 4a9c9ea0d318bec2f67848c5ceaf4ad5bcb91d09
      
https://github.com/qemu/qemu/commit/4a9c9ea0d318bec2f67848c5ceaf4ad5bcb91d09
  Author: Fam Zheng <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M block.c

  Log Message:
  -----------
  block: Detect multiplication overflow in bdrv_getlength

Bogus image may have a large total_sectors that will overflow the
multiplication. For cleanness, fix the return code so the error message
will be meaningful.

Reported-by: Richard W.M. Jones <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>
Reviewed-by: Alberto Garcia <address@hidden>
Reviewed-by: Markus Armbruster <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: b93bbf4ee9035ae077679482305d5beb38df4d7d
      
https://github.com/qemu/qemu/commit/b93bbf4ee9035ae077679482305d5beb38df4d7d
  Author: Fam Zheng <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M tests/qemu-iotests/059
    M tests/qemu-iotests/059.out
    A tests/qemu-iotests/sample_images/afl9.vmdk.bz2

  Log Message:
  -----------
  qemu-iotests: qemu-img info on afl VMDK image with a huge capacity

The image is contributed by Richard W.M. Jones.

Cc: Richard W.M. Jones <address@hidden>
Signed-off-by: Fam Zheng <address@hidden>
Reviewed-by: Alberto Garcia <address@hidden>
Reviewed-by: Markus Armbruster <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: aa4f592a1dd9aea5f5c36f6ff4b22b5bd208162a
      
https://github.com/qemu/qemu/commit/aa4f592a1dd9aea5f5c36f6ff4b22b5bd208162a
  Author: Fam Zheng <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M tests/qemu-iotests/check
    M tests/qemu-iotests/common
    M tests/qemu-iotests/iotests.py

  Log Message:
  -----------
  qemu-iotests: Make debugging python tests easier

Adding "-d" option. The output goes to "tee" so it appears in your
console. Also, raise the verbosity of unnitest runner.

When testing a topic branch, it's possible that a bug introduced by a
code change makes the python test case hang, with debug output, it is
much easier to locate the problem.

This can also be helpful if you want to watch the progress of a python
test, it offers you a way to sense the speed of each test case method
you're writing.

Note: because there is no easy way to get *both* the verbose output and
the output expected by ./check comparison, the case would always fail
with an "output mismatch". The sole purpose of using this option is
giving developers a quick way to debug when things go wrong.

Signed-off-by: Fam Zheng <address@hidden>
Reviewed-by: Stefan Hajnoczi <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 8336aafae1451d54c81dd2b187b45f7c45d2428e
      
https://github.com/qemu/qemu/commit/8336aafae1451d54c81dd2b187b45f7c45d2428e
  Author: Daniel P. Berrange <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M block/qcow.c
    M block/qcow2-cluster.c
    M block/qcow2.c

  Log Message:
  -----------
  qcow2/qcow: protect against uninitialized encryption key

When a qcow[2] file is opened, if the header reports an
encryption method, this is used to set the 'crypt_method_header'
field on the BDRVQcow[2]State struct, and the 'encrypted' flag
in the BDRVState struct.

When doing I/O operations, the 'crypt_method' field on the
BDRVQcow[2]State struct is checked to determine if encryption
needs to be applied.

The crypt_method_header value is copied into crypt_method when
the bdrv_set_key() method is called.

The QEMU code which opens a block device is expected to always
do a check

   if (bdrv_is_encrypted(bs)) {
       bdrv_set_key(bs, ....key...);
   }

If code forgets to do this, then 'crypt_method' is never set
and so when I/O is performed, QEMU writes plain text data
into a sector which is expected to contain cipher text, or
when reading, will return cipher text instead of plain
text.

Change the qcow[2] code to consult bs->encrypted when deciding
whether encryption is required, and assert(s->crypt_method)
to protect against cases where the caller forgets to set the
encryption key.

Also put an assert in the set_key methods to protect against
the case where the caller sets an encryption key on a block
device that does not have encryption

Signed-off-by: Daniel P. Berrange <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: d57e4e482e3997b1382625c84149ad0b69155fc0
      
https://github.com/qemu/qemu/commit/d57e4e482e3997b1382625c84149ad0b69155fc0
  Author: Daniel P. Berrange <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M include/qemu/osdep.h
    M qemu-img.c
    M util/oslib-posix.c
    M util/oslib-win32.c

  Log Message:
  -----------
  util: move read_password method out of qemu-img into osdep/oslib

The qemu-img.c file has a read_password() method impl that is
used to prompt for passwords on the console, with impls for
POSIX and Windows. This will be needed by qemu-io.c too, so
move it into the QEMU osdep/oslib files where it can be shared
without code duplication

Signed-off-by: Daniel P. Berrange <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 6a11d5183fb7564a3d32007b46846312fd61a1c5
      
https://github.com/qemu/qemu/commit/6a11d5183fb7564a3d32007b46846312fd61a1c5
  Author: Daniel P. Berrange <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M util/oslib-posix.c

  Log Message:
  -----------
  util: allow \n to terminate password input

The qemu_read_password() method looks for \r to terminate the
reading of the a password. This is what will be seen when
reading the password from a TTY. When scripting though, it is
useful to be able to send the password via a pipe, in which
case we must look for \n to terminate password input.

Signed-off-by: Daniel P. Berrange <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 8caf02127e92939fff39b63a7ff1a5834d320191
      
https://github.com/qemu/qemu/commit/8caf02127e92939fff39b63a7ff1a5834d320191
  Author: Daniel P. Berrange <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M qemu-io.c

  Log Message:
  -----------
  qemu-io: prompt for encryption keys when required

The qemu-io tool does not check if the image is encrypted so
historically would silently corrupt the sectors by writing
plain text data into them instead of cipher text. The earlier
commit turns this mistake into a fatal abort, so check for
encryption and prompt for key when required.

This enables us to add unit tests to ensure we don't break
the ability of qemu-img to convert existing encrypted qcow2
files into a non-encrypted format.

Signed-off-by: Daniel P. Berrange <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: f7ac119cfac735b24412db8d53b9be13e5ff23b0
      
https://github.com/qemu/qemu/commit/f7ac119cfac735b24412db8d53b9be13e5ff23b0
  Author: Daniel P. Berrange <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    A tests/qemu-iotests/134
    A tests/qemu-iotests/134.out
    M tests/qemu-iotests/group

  Log Message:
  -----------
  tests: add test case for encrypted qcow2 read/write

Add a simple test case for qemu-iotests that covers read/write
with encrypted qcow2 files.

Signed-off-by: Daniel P. Berrange <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 4c346e0bb9300afe3036560c21baa7fdfb253d9b
      
https://github.com/qemu/qemu/commit/4c346e0bb9300afe3036560c21baa7fdfb253d9b
  Author: Kevin Wolf <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M MAINTAINERS

  Log Message:
  -----------
  MAINTAINERS: Add header files to Block Layer Core section

Suggested-by: Markus Armbruster <address@hidden>
Reviewed-by: Alberto Garcia <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 4120201d2fcfc24404fe6eb6b761b66bc35bca16
      
https://github.com/qemu/qemu/commit/4120201d2fcfc24404fe6eb6b761b66bc35bca16
  Author: Markus Armbruster <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M MAINTAINERS

  Log Message:
  -----------
  MAINTAINERS: Split "Block QAPI, monitor, command line" off core

Kevin and Stefan asked me to take care of this part.

Signed-off-by: Markus Armbruster <address@hidden>
Signed-off-by: Kevin Wolf <address@hidden>


  Commit: 0d2ed6039cf86fe3a78671e32b5e3eb17d725762
      
https://github.com/qemu/qemu/commit/0d2ed6039cf86fe3a78671e32b5e3eb17d725762
  Author: Peter Maydell <address@hidden>
  Date:   2015-05-22 (Fri, 22 May 2015)

  Changed paths:
    M MAINTAINERS
    M block.c
    M block/qcow.c
    M block/qcow2-cache.c
    M block/qcow2-cluster.c
    M block/qcow2-refcount.c
    M block/qcow2.c
    M block/qcow2.h
    M block/vmdk.c
    M hw/block/nvme.c
    M include/qemu/osdep.h
    M qemu-img.c
    M qemu-io-cmds.c
    M qemu-io.c
    M tests/qemu-iotests/059
    M tests/qemu-iotests/059.out
    A tests/qemu-iotests/134
    A tests/qemu-iotests/134.out
    M tests/qemu-iotests/check
    M tests/qemu-iotests/common
    M tests/qemu-iotests/group
    M tests/qemu-iotests/iotests.py
    A tests/qemu-iotests/sample_images/afl9.vmdk.bz2
    M util/oslib-posix.c
    M util/oslib-win32.c

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer core and image format patches

# gpg: Signature made Fri May 22 16:21:03 2015 BST using RSA key ID C88F2FD6
# gpg: Good signature from "Kevin Wolf <address@hidden>"

* remotes/kevin/tags/for-upstream: (22 commits)
  MAINTAINERS: Split "Block QAPI, monitor, command line" off core
  MAINTAINERS: Add header files to Block Layer Core section
  tests: add test case for encrypted qcow2 read/write
  qemu-io: prompt for encryption keys when required
  util: allow \n to terminate password input
  util: move read_password method out of qemu-img into osdep/oslib
  qcow2/qcow: protect against uninitialized encryption key
  qemu-iotests: Make debugging python tests easier
  qemu-iotests: qemu-img info on afl VMDK image with a huge capacity
  block: Detect multiplication overflow in bdrv_getlength
  qemu-io: Use getopt() correctly
  qcow2: style fixes in qcow2-cache.c
  qcow2: make qcow2_cache_put() a void function
  qcow2: use a hash to look for entries in the L2 cache
  qcow2: remove qcow2_cache_find_entry_to_replace()
  qcow2: use an LRU algorithm to replace entries from the L2 cache
  qcow2: simplify qcow2_cache_put() and qcow2_cache_entry_mark_dirty()
  qcow2: use one single memory block for the L2/refcount cache tables
  vmdk: Fix overflow if l1_size is 0x20000000
  vmdk: Fix next_cluster_sector for compressed write
  ...

Signed-off-by: Peter Maydell <address@hidden>


Compare: https://github.com/qemu/qemu/compare/bb2fa17f182e...0d2ed6039cf8