[Qemu-commits] [qemu/qemu] 27af7d: xbzrle: optimize XBZRLE to decrease the cache miss...

[GitHub]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 27af7d6ea5015e5ef1f7985eab94a8a218267a2b
      
https://github.com/qemu/qemu/commit/27af7d6ea5015e5ef1f7985eab94a8a218267a2b
  Author: ChenLiang <address@hidden>
  Date:   2015-01-15 (Thu, 15 Jan 2015)

  Changed paths:
    M arch_init.c
    M docs/xbzrle.txt
    M include/migration/page_cache.h
    M page_cache.c

  Log Message:
  -----------
  xbzrle: optimize XBZRLE to decrease the cache misses

Avoid hot pages being replaced by others to remarkably decrease cache
misses

Sample results with the test program which quote from xbzrle.txt ran in
vm:(migrate bandwidth:1GE and xbzrle cache size 8MB)

the test program:

include <stdlib.h>
include <stdio.h>
int main()
 {
  char *buf = (char *) calloc(4096, 4096);
  while (1) {
      int i;
      for (i = 0; i < 4096 * 4; i++) {
          buf[i * 4096 / 4]++;
      }
      printf(".");
  }
 }

before this patch:
virsh qemu-monitor-command test_vm '{"execute": "query-migrate"}'
{"return":{"expected-downtime":1020,"xbzrle-cache":{"bytes":1108284,
"cache-size":8388608,"cache-miss-rate":0.987013,"pages":18297,"overflow":8,
"cache-miss":1228737},"status":"active","setup-time":10,"total-time":52398,
"ram":{"total":12466991104,"remaining":1695744,"mbps":935.559472,
"transferred":5780760580,"dirty-sync-counter":271,"duplicate":2878530,
"dirty-pages-rate":29130,"skipped":0,"normal-bytes":5748592640,
"normal":1403465}},"id":"libvirt-706"}

18k pages sent compressed in 52 seconds.
cache-miss-rate is 98.7%, totally miss.

after optimizing:
virsh qemu-monitor-command test_vm '{"execute": "query-migrate"}'
{"return":{"expected-downtime":2054,"xbzrle-cache":{"bytes":5066763,
"cache-size":8388608,"cache-miss-rate":0.485924,"pages":194823,"overflow":0,
"cache-miss":210653},"status":"active","setup-time":11,"total-time":18729,
"ram":{"total":12466991104,"remaining":3895296,"mbps":937.663549,
"transferred":1615042219,"dirty-sync-counter":98,"duplicate":2869840,
"dirty-pages-rate":58781,"skipped":0,"normal-bytes":1588404224,
"normal":387794}},"id":"libvirt-266"}

194k pages sent compressed in 18 seconds.
The value of cache-miss-rate decrease to 48.59%.

Signed-off-by: ChenLiang <address@hidden>
Signed-off-by: Gonglei <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Signed-off-by: Amit Shah <address@hidden>


  Commit: 1b826f277814dd9496fe3cc71cbe6ab7b203cadf
      
https://github.com/qemu/qemu/commit/1b826f277814dd9496fe3cc71cbe6ab7b203cadf
  Author: ChenLiang <address@hidden>
  Date:   2015-01-15 (Thu, 15 Jan 2015)

  Changed paths:
    M page_cache.c

  Log Message:
  -----------
  xbzrle: rebuild the cache_is_cached function

Rebuild the cache_is_cached function by cache_get_by_addr. And
drops the asserts because the caller is also asserting the same
thing.

Signed-off-by: ChenLiang <address@hidden>
Signed-off-by: Gonglei <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Signed-off-by: Amit Shah <address@hidden>


  Commit: f018d8cd2123f495300935d5019931abbee4e5d9
      
https://github.com/qemu/qemu/commit/f018d8cd2123f495300935d5019931abbee4e5d9
  Author: Yang Hongyang <address@hidden>
  Date:   2015-01-16 (Fri, 16 Jan 2015)

  Changed paths:
    M migration/qemu-file-buf.c

  Log Message:
  -----------
  QEMUSizedBuffer: only free qsb that qemu_bufopen allocated

Only free qsb that qemu_bufopen allocated, and also allow
qemu_bufopen accept qsb as input for write operation. It
will make the API more logical:
1.If you create the QEMUSizedBuffer yourself, you need to
  free it by using qsb_free() but not depends on other API
  like qemu_fclose.
2.allow qemu_bufopen() accept QEMUSizedBuffer as input for
  write operation, otherwise, it will be a little strange
  for this API won't accept the second parameter.

This brings API change, since there are only 3
users of this API currently, this change only impact the
first one which will be fixed in patch 2 of this patchset,
so I think it is safe to do this change.

1     70  tests/test-vmstate.c <<open_mem_file_read>>
      return qemu_bufopen("r", qsb);
2    404  tests/test-vmstate.c <<test_save_noskip>>
      QEMUFile *fsave = qemu_bufopen("w", NULL);
3    424  tests/test-vmstate.c <<test_save_skip>>
      QEMUFile *fsave = qemu_bufopen("w", NULL);

Signed-off-by: Yang Hongyang <address@hidden>
Cc: Dr. David Alan Gilbert <address@hidden>
Cc: Juan Quintela <address@hidden>
Cc: Amit Shah <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Signed-off-by: Amit Shah <address@hidden>


  Commit: 8580b06498a5dffe554e7ac627726b1d7775c591
      
https://github.com/qemu/qemu/commit/8580b06498a5dffe554e7ac627726b1d7775c591
  Author: Yang Hongyang <address@hidden>
  Date:   2015-01-16 (Fri, 16 Jan 2015)

  Changed paths:
    M tests/test-vmstate.c

  Log Message:
  -----------
  Tests: QEMUSizedBuffer/QEMUBuffer

Modify some of tests/test-vmstate.c due to qemu_bufopen() change.
If you create a QEMUSizedBuffer yourself, you have to explicitly
free it.

Signed-off-by: Yang Hongyang <address@hidden>
Cc: Dr. David Alan Gilbert <address@hidden>
Cc: Juan Quintela <address@hidden>
Cc: Amit Shah <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Signed-off-by: Amit Shah <address@hidden>


  Commit: e1a8c9b67fc97d293211773edcae9e8e2f3367ab
      
https://github.com/qemu/qemu/commit/e1a8c9b67fc97d293211773edcae9e8e2f3367ab
  Author: Dr. David Alan Gilbert <address@hidden>
  Date:   2015-01-16 (Fri, 16 Jan 2015)

  Changed paths:
    M include/migration/qemu-file.h
    M include/qemu/sockets.h
    M migration/qemu-file-unix.c
    M migration/qemu-file.c

  Log Message:
  -----------
  socket shutdown

Add QEMUFile interface to allow a socket to be 'shut down' - i.e. any
reads/writes will fail (and any blocking read/write will be woken).

Signed-off-by: Dr. David Alan Gilbert <address@hidden>
Reviewed-by: Paolo Bonzini <address@hidden>
Reviewed-by: Amit Shah <address@hidden>
Signed-off-by: Amit Shah <address@hidden>


  Commit: 131fe9b843f9a1e55fcbf2457c9cb25c3711b9d8
      
https://github.com/qemu/qemu/commit/131fe9b843f9a1e55fcbf2457c9cb25c3711b9d8
  Author: Cristian Klein <address@hidden>
  Date:   2015-01-16 (Fri, 16 Jan 2015)

  Changed paths:
    M migration/fd.c

  Log Message:
  -----------
  Handle bi-directional communication for fd migration

libvirt prefers opening the TCP connection itself, for two reasons.
First, connection failed errors can be detected easier, without having
to parse qemu's error output.
Second, libvirt might be asked to secure the transfer by tunnelling the
communication through an TLS layer.
Therefore, libvirt opens the TCP connection itself and passes an FD to qemu
using QMP and a POSIX-specific mechanism.

Hence, in order to make the reverse-path work in such cases, qemu needs to
distinguish if the transmitted FD is a socket (reverse-path available)
or not (reverse-path might not be available) and use the corresponding
abstraction.

Signed-off-by: Cristian Klein <address@hidden>
Reviewed-by: Paolo Bonzini <address@hidden>
Reviewed-by: Amit Shah <address@hidden>
Signed-off-by: Amit Shah <address@hidden>


  Commit: a26ba26e214911dc879a23e797d2c269cdb38577
      
https://github.com/qemu/qemu/commit/a26ba26e214911dc879a23e797d2c269cdb38577
  Author: Dr. David Alan Gilbert <address@hidden>
  Date:   2015-01-16 (Fri, 16 Jan 2015)

  Changed paths:
    M migration/migration.c

  Log Message:
  -----------
  migration_cancel: shutdown migration socket

Force shutdown on migration socket on cancel to cause the cancel
to complete even if the socket is blocked on a dead network.

Signed-off-by: Dr. David Alan Gilbert <address@hidden>
Reviewed-by: Paolo Bonzini <address@hidden>
Reviewed-by: Amit Shah <address@hidden>
Signed-off-by: Amit Shah <address@hidden>


  Commit: ea987c2c21d4326bb58ee28f6888fdcf8fbda067
      
https://github.com/qemu/qemu/commit/ea987c2c21d4326bb58ee28f6888fdcf8fbda067
  Author: Paolo Bonzini <address@hidden>
  Date:   2015-01-16 (Fri, 16 Jan 2015)

  Changed paths:
    M include/migration/vmstate.h

  Log Message:
  -----------
  vmstate: type-check sub-arrays

While we cannot check against the type of the full array, we can check
against the type of the fields.

Signed-off-by: Paolo Bonzini <address@hidden>
Signed-off-by: Amit Shah <address@hidden>


  Commit: e68cba36360a2ab5bf0576b66df4d0eb0d822f8d
      
https://github.com/qemu/qemu/commit/e68cba36360a2ab5bf0576b66df4d0eb0d822f8d
  Author: Peter Maydell <address@hidden>
  Date:   2015-01-16 (Fri, 16 Jan 2015)

  Changed paths:
    M arch_init.c
    M docs/xbzrle.txt
    M include/migration/page_cache.h
    M include/migration/qemu-file.h
    M include/migration/vmstate.h
    M include/qemu/sockets.h
    M migration/fd.c
    M migration/migration.c
    M migration/qemu-file-buf.c
    M migration/qemu-file-unix.c
    M migration/qemu-file.c
    M page_cache.c
    M tests/test-vmstate.c

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/amit-migration/tags/mig-2.3-1' into 
staging

A set of patches collected over the holidays.  Mix of optimizations and
fixes.

# gpg: Signature made Fri 16 Jan 2015 07:42:00 GMT using RSA key ID 854083B6
# gpg: Good signature from "Amit Shah <address@hidden>"
# gpg:                 aka "Amit Shah <address@hidden>"
# gpg:                 aka "Amit Shah <address@hidden>"

* remotes/amit-migration/tags/mig-2.3-1:
  vmstate: type-check sub-arrays
  migration_cancel: shutdown migration socket
  Handle bi-directional communication for fd migration
  socket shutdown
  Tests: QEMUSizedBuffer/QEMUBuffer
  QEMUSizedBuffer: only free qsb that qemu_bufopen allocated
  xbzrle: rebuild the cache_is_cached function
  xbzrle: optimize XBZRLE to decrease the cache misses

Signed-off-by: Peter Maydell <address@hidden>


Compare: https://github.com/qemu/qemu/compare/df58887b20fa...e68cba36360a