[Qemu-commits] [qemu/qemu] 1cc354: nvme: do not advertise support for unsupported arb...

[Peter Maydell]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 1cc354ac9868a35279ff01857144212f6aefb697
      
https://github.com/qemu/qemu/commit/1cc354ac9868a35279ff01857144212f6aefb697
  Author: Klaus Birkelund Jensen <address@hidden>
  Date:   2019-06-24 (Mon, 24 Jun 2019)

  Changed paths:
    M hw/block/nvme.c

  Log Message:
  -----------
  nvme: do not advertise support for unsupported arbitration mechanism

The device mistakenly reports that the Weighted Round Robin with Urgent
Priority Class arbitration mechanism is supported.

It is not.

Signed-off-by: Klaus Birkelund Jensen <address@hidden>
Message-id: address@hidden
Acked-by: Maxim Levitsky <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 85c9d133fb53ff098505ad03f139a3f4c99be863
      
https://github.com/qemu/qemu/commit/85c9d133fb53ff098505ad03f139a3f4c99be863
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2019-06-24 (Mon, 24 Jun 2019)

  Changed paths:
    M blockdev.c

  Log Message:
  -----------
  blockdev: enable non-root nodes for transaction drive-backup source

We forget to enable it for transaction .prepare, while it is already
enabled in do_drive_backup since commit a2d665c1bc362
    "blockdev: loosen restrictions on drive-backup source node"

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Reviewed-by: John Snow <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 6ec889eb85c8fe96827c57e3900d8a1079554f3d
      
https://github.com/qemu/qemu/commit/6ec889eb85c8fe96827c57e3900d8a1079554f3d
  Author: Anton Nefedov <address@hidden>
  Date:   2019-06-24 (Mon, 24 Jun 2019)

  Changed paths:
    M tests/qemu-iotests/134
    M tests/qemu-iotests/134.out

  Log Message:
  -----------
  iotest 134: test cluster-misaligned encrypted write

COW (even empty/zero) areas require encryption too

Signed-off-by: Anton Nefedov <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Reviewed-by: Alberto Garcia <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: 940a2cd5d2eede3c81b83712c55cc6d3c5e6c463
      
https://github.com/qemu/qemu/commit/940a2cd5d2eede3c81b83712c55cc6d3c5e6c463
  Author: Sam Eiderman <address@hidden>
  Date:   2019-06-24 (Mon, 24 Jun 2019)

  Changed paths:
    M block/vmdk.c

  Log Message:
  -----------
  vmdk: Fix comment regarding max l1_size coverage

Commit b0651b8c246d ("vmdk: Move l1_size check into vmdk_add_extent")
extended the l1_size check from VMDK4 to VMDK3 but did not update the
default coverage in the moved comment.

The previous vmdk4 calculation:

    (512 * 1024 * 1024) * 512(l2 entries) * 65536(grain) = 16PB

The added vmdk3 calculation:

    (512 * 1024 * 1024) * 4096(l2 entries) * 512(grain) = 1PB

Adding the calculation of vmdk3 to the comment.

In any case, VMware does not offer virtual disks more than 2TB for
vmdk4/vmdk3 or 64TB for the new undocumented seSparse format which is
not implemented yet in qemu.

Reviewed-by: Karl Heubaum <address@hidden>
Reviewed-by: Eyal Moscovici <address@hidden>
Reviewed-by: Liran Alon <address@hidden>
Reviewed-by: Arbel Moshe <address@hidden>
Signed-off-by: Sam Eiderman <address@hidden>
Message-id: address@hidden
Reviewed-by: yuchenlin <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 59d6ee485020fdc45cb0f9f748d8b8850fad9f8c
      
https://github.com/qemu/qemu/commit/59d6ee485020fdc45cb0f9f748d8b8850fad9f8c
  Author: Sam Eiderman <address@hidden>
  Date:   2019-06-24 (Mon, 24 Jun 2019)

  Changed paths:
    M block/vmdk.c
    M tests/qemu-iotests/059.out

  Log Message:
  -----------
  vmdk: Reduce the max bound for L1 table size

512M of L1 entries is a very loose bound, only 32M are required to store
the maximal supported VMDK file size of 2TB.

Fixed qemu-iotest 59# - now failure occures before on impossible L1
table size.

Reviewed-by: Karl Heubaum <address@hidden>
Reviewed-by: Eyal Moscovici <address@hidden>
Reviewed-by: Liran Alon <address@hidden>
Reviewed-by: Arbel Moshe <address@hidden>
Signed-off-by: Sam Eiderman <address@hidden>
Message-id: address@hidden
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 98eb9733f4cf2eeab6d12db7e758665d2fd5367b
      
https://github.com/qemu/qemu/commit/98eb9733f4cf2eeab6d12db7e758665d2fd5367b
  Author: Sam Eiderman <address@hidden>
  Date:   2019-06-24 (Mon, 24 Jun 2019)

  Changed paths:
    M block/vmdk.c

  Log Message:
  -----------
  vmdk: Add read-only support for seSparse snapshots

Until ESXi 6.5 VMware used the vmfsSparse format for snapshots (VMDK3 in
QEMU).

This format was lacking in the following:

    * Grain directory (L1) and grain table (L2) entries were 32-bit,
      allowing access to only 2TB (slightly less) of data.
    * The grain size (default) was 512 bytes - leading to data
      fragmentation and many grain tables.
    * For space reclamation purposes, it was necessary to find all the
      grains which are not pointed to by any grain table - so a reverse
      mapping of "offset of grain in vmdk" to "grain table" must be
      constructed - which takes large amounts of CPU/RAM.

The format specification can be found in VMware's documentation:
https://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf

In ESXi 6.5, to support snapshot files larger than 2TB, a new format was
introduced: SESparse (Space Efficient).

This format fixes the above issues:

    * All entries are now 64-bit.
    * The grain size (default) is 4KB.
    * Grain directory and grain tables are now located at the beginning
      of the file.
      + seSparse format reserves space for all grain tables.
      + Grain tables can be addressed using an index.
      + Grains are located in the end of the file and can also be
        addressed with an index.
      - seSparse vmdks of large disks (64TB) have huge preallocated
        headers - mainly due to L2 tables, even for empty snapshots.
    * The header contains a reverse mapping ("backmap") of "offset of
      grain in vmdk" to "grain table" and a bitmap ("free bitmap") which
      specifies for each grain - whether it is allocated or not.
      Using these data structures we can implement space reclamation
      efficiently.
    * Due to the fact that the header now maintains two mappings:
        * The regular one (grain directory & grain tables)
        * A reverse one (backmap and free bitmap)
      These data structures can lose consistency upon crash and result
      in a corrupted VMDK.
      Therefore, a journal is also added to the VMDK and is replayed
      when the VMware reopens the file after a crash.

Since ESXi 6.7 - SESparse is the only snapshot format available.

Unfortunately, VMware does not provide documentation regarding the new
seSparse format.

This commit is based on black-box research of the seSparse format.
Various in-guest block operations and their effect on the snapshot file
were tested.

The only VMware provided source of information (regarding the underlying
implementation) was a log file on the ESXi:

    /var/log/hostd.log

Whenever an seSparse snapshot is created - the log is being populated
with seSparse records.

Relevant log records are of the form:

[...] Const Header:
[...]  constMagic     = 0xcafebabe
[...]  version        = 2.1
[...]  capacity       = 204800
[...]  grainSize      = 8
[...]  grainTableSize = 64
[...]  flags          = 0
[...] Extents:
[...]  Header         : <1 : 1>
[...]  JournalHdr     : <2 : 2>
[...]  Journal        : <2048 : 2048>
[...]  GrainDirectory : <4096 : 2048>
[...]  GrainTables    : <6144 : 2048>
[...]  FreeBitmap     : <8192 : 2048>
[...]  BackMap        : <10240 : 2048>
[...]  Grain          : <12288 : 204800>
[...] Volatile Header:
[...] volatileMagic     = 0xcafecafe
[...] FreeGTNumber      = 0
[...] nextTxnSeqNumber  = 0
[...] replayJournal     = 0

The sizes that are seen in the log file are in sectors.
Extents are of the following format: <offset : size>

This commit is a strict implementation which enforces:
    * magics
    * version number 2.1
    * grain size of 8 sectors  (4KB)
    * grain table size of 64 sectors
    * zero flags
    * extent locations

Additionally, this commit proivdes only a subset of the functionality
offered by seSparse's format:
    * Read-only
    * No journal replay
    * No space reclamation
    * No unmap support

Hence, journal header, journal, free bitmap and backmap extents are
unused, only the "classic" (L1 -> L2 -> data) grain access is
implemented.

However there are several differences in the grain access itself.
Grain directory (L1):
    * Grain directory entries are indexes (not offsets) to grain
      tables.
    * Valid grain directory entries have their highest nibble set to
      0x1.
    * Since grain tables are always located in the beginning of the
      file - the index can fit into 32 bits - so we can use its low
      part if it's valid.
Grain table (L2):
    * Grain table entries are indexes (not offsets) to grains.
    * If the highest nibble of the entry is:
        0x0:
            The grain in not allocated.
            The rest of the bytes are 0.
        0x1:
            The grain is unmapped - guest sees a zero grain.
            The rest of the bits point to the previously mapped grain,
            see 0x3 case.
        0x2:
            The grain is zero.
        0x3:
            The grain is allocated - to get the index calculate:
            ((entry & 0x0fff000000000000) >> 48) |
            ((entry & 0x0000ffffffffffff) << 12)
    * The difference between 0x1 and 0x2 is that 0x1 is an unallocated
      grain which results from the guest using sg_unmap to unmap the
      grain - but the grain itself still exists in the grain extent - a
      space reclamation procedure should delete it.
      Unmapping a zero grain has no effect (0x2 will not change to 0x1)
      but unmapping an unallocated grain will (0x0 to 0x1) - naturally.

In order to implement seSparse some fields had to be changed to support
both 32-bit and 64-bit entry sizes.

Reviewed-by: Karl Heubaum <address@hidden>
Reviewed-by: Eyal Moscovici <address@hidden>
Reviewed-by: Arbel Moshe <address@hidden>
Signed-off-by: Sam Eiderman <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: b10d49d7619e4957b4b971f816661b57e5061d71
      
https://github.com/qemu/qemu/commit/b10d49d7619e4957b4b971f816661b57e5061d71
  Author: Pino Toscano <address@hidden>
  Date:   2019-06-24 (Mon, 24 Jun 2019)

  Changed paths:
    M .travis.yml
    M block/Makefile.objs
    M block/ssh.c
    M block/trace-events
    M configure
    M docs/qemu-block-drivers.texi
    M tests/docker/dockerfiles/debian-win32-cross.docker
    M tests/docker/dockerfiles/debian-win64-cross.docker
    M tests/docker/dockerfiles/fedora.docker
    M tests/docker/dockerfiles/ubuntu.docker
    M tests/docker/dockerfiles/ubuntu1804.docker
    M tests/qemu-iotests/207
    M tests/qemu-iotests/207.out

  Log Message:
  -----------
  ssh: switch from libssh2 to libssh

Rewrite the implementation of the ssh block driver to use libssh instead
of libssh2.  The libssh library has various advantages over libssh2:
- easier API for authentication (for example for using ssh-agent)
- easier API for known_hosts handling
- supports newer types of keys in known_hosts

Use APIs/features available in libssh 0.8 conditionally, to support
older versions (which are not recommended though).

Adjust the iotest 207 according to the different error message, and to
find the default key type for localhost (to properly compare the
fingerprint with).
Contributed-by: Max Reitz <address@hidden>

Adjust the various Docker/Travis scripts to use libssh when available
instead of libssh2. The mingw/mxe testing is dropped for now, as there
are no packages for it.

Signed-off-by: Pino Toscano <address@hidden>
Tested-by: Philippe Mathieu-Daudé <address@hidden>
Acked-by: Alex Bennée <address@hidden>
Message-id: address@hidden
Reviewed-by: Philippe Mathieu-Daudé <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: ab5d4a30f7f3803ca5106b370969c1b7b54136f8
      
https://github.com/qemu/qemu/commit/ab5d4a30f7f3803ca5106b370969c1b7b54136f8
  Author: Max Reitz <address@hidden>
  Date:   2019-06-24 (Mon, 24 Jun 2019)

  Changed paths:
    M tests/qemu-iotests/205

  Log Message:
  -----------
  iotests: Fix 205 for concurrent runs

Tests should place their files into the test directory.  This includes
Unix sockets.  205 currently fails to do so, which prevents it from
being run concurrently.

Signed-off-by: Max Reitz <address@hidden>
Message-id: address@hidden
Reviewed-by: Eric Blake <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 7fec76a02267598a4e437ddfdaeaeb6de09b92f3
      
https://github.com/qemu/qemu/commit/7fec76a02267598a4e437ddfdaeaeb6de09b92f3
  Author: Peter Maydell <address@hidden>
  Date:   2019-07-01 (Mon, 01 Jul 2019)

  Changed paths:
    M .travis.yml
    M block/Makefile.objs
    M block/ssh.c
    M block/trace-events
    M block/vmdk.c
    M blockdev.c
    M configure
    M docs/qemu-block-drivers.texi
    M hw/block/nvme.c
    M tests/docker/dockerfiles/debian-win32-cross.docker
    M tests/docker/dockerfiles/debian-win64-cross.docker
    M tests/docker/dockerfiles/fedora.docker
    M tests/docker/dockerfiles/ubuntu.docker
    M tests/docker/dockerfiles/ubuntu1804.docker
    M tests/qemu-iotests/059.out
    M tests/qemu-iotests/134
    M tests/qemu-iotests/134.out
    M tests/qemu-iotests/205
    M tests/qemu-iotests/207
    M tests/qemu-iotests/207.out

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2019-06-24' 
into staging

Block patches:
- The SSH block driver now uses libssh instead of libssh2
- The VMDK block driver gets read-only support for the seSparse
  subformat
- Various fixes

# gpg: Signature made Mon 24 Jun 2019 15:42:56 BST
# gpg:                using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg:                issuer "address@hidden"
# gpg: Good signature from "Max Reitz <address@hidden>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1  1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2019-06-24:
  iotests: Fix 205 for concurrent runs
  ssh: switch from libssh2 to libssh
  vmdk: Add read-only support for seSparse snapshots
  vmdk: Reduce the max bound for L1 table size
  vmdk: Fix comment regarding max l1_size coverage
  iotest 134: test cluster-misaligned encrypted write
  blockdev: enable non-root nodes for transaction drive-backup source
  nvme: do not advertise support for unsupported arbitration mechanism

Signed-off-by: Peter Maydell <address@hidden>


Compare: https://github.com/qemu/qemu/compare/474f3938d79a...7fec76a02267