[Qemu-commits] [qemu/qemu] 34c55a: migration: Create multipage support

qemu-commits
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Qemu-commits] [qemu/qemu] 34c55a: migration: Create multipage support

From:	GitHub
Subject:	[Qemu-commits] [qemu/qemu] 34c55a: migration: Create multipage support
Date:	Thu, 28 Jun 2018 08:28:05 -0700
  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 34c55a94b14323c06791395801aadaf5606761dc
      
https://github.com/qemu/qemu/commit/34c55a94b14323c06791395801aadaf5606761dc
  Author: Juan Quintela <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M migration/ram.c

  Log Message:
  -----------
  migration: Create multipage support

We only create/destry the page list here.  We will use it later.

Signed-off-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 2a26c979b15dfedc1d7f4df7ed7e853d768e2850
      
https://github.com/qemu/qemu/commit/2a26c979b15dfedc1d7f4df7ed7e853d768e2850
  Author: Juan Quintela <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M migration/ram.c

  Log Message:
  -----------
  migration: Create multifd packet

We still don't put anything there.

Signed-off-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>

--
fix magic (dave)
check offset/ramblock  (dave)
s/seq/packet_num/ and make it 64bit


  Commit: 6cde6fbe2b2ad672029ce9737659aa34f581c07a
      
https://github.com/qemu/qemu/commit/6cde6fbe2b2ad672029ce9737659aa34f581c07a
  Author: Juan Quintela <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M migration/migration.c

  Log Message:
  -----------
  migration: Calculate mbps only during transfer time

We used to include in this calculation the setup time, but that can be
quite big in rdma or multifd.

Signed-off-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 0c8f0efdd461b26f64fdf8785c333558315bc297
      
https://github.com/qemu/qemu/commit/0c8f0efdd461b26f64fdf8785c333558315bc297
  Author: Juan Quintela <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M migration/migration.c

  Log Message:
  -----------
  migration: Abstract the number of bytes sent

Right now we use the "position" inside the QEMUFile, but things like
RDMA already do weird things to be able to maintain that counter
right, and multifd will have some similar problems.

Signed-off-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 408ea6ae4c8f94621dff64d69c7fa3f6e84ef2fb
      
https://github.com/qemu/qemu/commit/408ea6ae4c8f94621dff64d69c7fa3f6e84ef2fb
  Author: Juan Quintela <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M migration/ram.c
    M migration/trace-events

  Log Message:
  -----------
  migration: Add multifd traces for start/end thread

We want to know how many pages/packets each channel has sent.  Add
counters for those.

Signed-off-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>

--
sort trace-events (dave)


  Commit: d82628e4bde23aaf50273cc835c8ce28b43590bd
      
https://github.com/qemu/qemu/commit/d82628e4bde23aaf50273cc835c8ce28b43590bd
  Author: Juan Quintela <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M migration/ram.c

  Log Message:
  -----------
  migration: Multifd channels always wait on the sem

Either for quit, sync or packet, we first wake them.

Signed-off-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 0beb5ed3279abf80d0475ca35f48041b02a9851a
      
https://github.com/qemu/qemu/commit/0beb5ed3279abf80d0475ca35f48041b02a9851a
  Author: Juan Quintela <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M migration/ram.c
    M migration/trace-events

  Log Message:
  -----------
  migration: Add block where to send/receive packets

Once there add tracepoints.

Signed-off-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 6df264ac5a7f532adf718b55d5ef713247d857b1
      
https://github.com/qemu/qemu/commit/6df264ac5a7f532adf718b55d5ef713247d857b1
  Author: Juan Quintela <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M migration/ram.c
    M migration/trace-events

  Log Message:
  -----------
  migration: Synchronize multifd threads with main thread

We synchronize all threads each RAM_SAVE_FLAG_EOS.  Bitmap
synchronizations don't happen inside a  ram section, so we are safe
about two channels trying to overwrite the same memory.

Signed-off-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>

--
seq needs to be atomic now, will also be accessed from main thread.
Fix the if (true || ...) leftover
We are back to non-atomics


  Commit: a61c45bd227951cd792f22b00d4e502c01ae8d9b
      
https://github.com/qemu/qemu/commit/a61c45bd227951cd792f22b00d4e502c01ae8d9b
  Author: Juan Quintela <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M hmp.c
    M migration/migration.c
    M qapi/migration.json

  Log Message:
  -----------
  migration: Create multifd_bytes ram_counter

This will include how many bytes they are sent through multifd.

Signed-off-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>


  Commit: b9ee2f7d701457f205a3329ea67b33e0b63a0f43
      
https://github.com/qemu/qemu/commit/b9ee2f7d701457f205a3329ea67b33e0b63a0f43
  Author: Juan Quintela <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M migration/migration.c
    M migration/ram.c

  Log Message:
  -----------
  migration: Create ram_save_multifd_page

The function still don't use multifd, but we have simplified
ram_save_page, xbzrle and RDMA stuff is gone.  We have added a new
counter.

Signed-off-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>

--
Add last_page parameter
Add commets for done and address
Remove multifd field, it is the same than normal pages
Merge next patch, now we send multiple pages at a time
Remove counter for multifd pages, it is identical to normal pages
Use iovec's instead of creating the equivalent.
Clear memory used by pages (dave)
Use g_new0(danp)
define MULTIFD_CONTINUE
now pages member is a pointer
Fix off-by-one in number of pages in one packet
Remove RAM_SAVE_FLAG_MULTIFD_PAGE
s/multifd_pages_t/MultiFDPages_t/
add comment explaining what it means


  Commit: 8b2db7f5fd9566ab9cf22b02b623c51b2075a60e
      
https://github.com/qemu/qemu/commit/8b2db7f5fd9566ab9cf22b02b623c51b2075a60e
  Author: Juan Quintela <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M migration/ram.c

  Log Message:
  -----------
  migration: Start sending messages

Signed-off-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 4d22c148c989f80e4377233bed886e96d85ef645
      
https://github.com/qemu/qemu/commit/4d22c148c989f80e4377233bed886e96d85ef645
  Author: Juan Quintela <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M migration/ram.c

  Log Message:
  -----------
  migration: Wait for blocking IO

We have three conditions here:
- channel fails -> error
- we have to quit: we close the channel and reads fails
- normal read that success, we are in bussiness

So forget the complications of waiting in a semaphore.

Signed-off-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 7a5cc33c4878e998e1e2d7c9092dddf963aeb795
      
https://github.com/qemu/qemu/commit/7a5cc33c4878e998e1e2d7c9092dddf963aeb795
  Author: Juan Quintela <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M migration/ram.c

  Log Message:
  -----------
  migration: Remove not needed semaphore and quit

We know quit with shutdwon in the QIO.

Signed-off-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>
--
Add comment
Use shutdown() instead of unref()


  Commit: 35374cbdff2bf6b6fb7caae5ea8aaa3b0f53f4ca
      
https://github.com/qemu/qemu/commit/35374cbdff2bf6b6fb7caae5ea8aaa3b0f53f4ca
  Author: Juan Quintela <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M migration/ram.c

  Log Message:
  -----------
  migration: Stop sending whole pages through main channel

We have to flush() the QEMUFile because now we sent really few data
through that channel.

Signed-off-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>


  Commit: c136180c901dd84d36577edc9b7f377fb76b832d
      
https://github.com/qemu/qemu/commit/c136180c901dd84d36577edc9b7f377fb76b832d
  Author: David Hildenbrand <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M exec.c
    M include/exec/ram_addr.h
    M migration/postcopy-ram.c
    M migration/postcopy-ram.h
    M migration/ram.c

  Log Message:
  -----------
  postcopy: drop ram_pages parameter from postcopy_ram_incoming_init()

Not needed. Don't expose last_ram_page().

Signed-off-by: David Hildenbrand <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: ca273df3014930f4ecb6d3ba24aa778354634b56
      
https://github.com/qemu/qemu/commit/ca273df3014930f4ecb6d3ba24aa778354634b56
  Author: Daniel P. Berrangé <address@hidden>
  Date:   2018-06-27 (Wed, 27 Jun 2018)

  Changed paths:
    M migration/migration.c

  Log Message:
  -----------
  migration: fix crash in when incoming client channel setup fails

The way we determine if we can start the incoming migration was
changed to use migration_has_all_channels() in:

  commit 428d89084c709e568f9cd301c2f6416a54c53d6d
  Author: Juan Quintela <address@hidden>
  Date:   Mon Jul 24 13:06:25 2017 +0200

    migration: Create migration_has_all_channels

This method in turn calls multifd_recv_all_channels_created()
which is hardcoded to always return 'true' when multifd is
not in use. This is a latent bug...

...activated in a following commit where that return result
ends up acting as the flag to indicate whether it is possible
to start processing the migration:

  commit 36c2f8be2c4eb0003ac77a14910842b7ddd7337e
  Author: Juan Quintela <address@hidden>
  Date:   Wed Mar 7 08:40:52 2018 +0100

    migration: Delay start of migration main routines

This means that if channel initialization fails with normal
migration, it'll never notice and attempt to start the
incoming migration regardless and crash on a NULL pointer.

This can be seen, for example, if a client connects to a server
requiring TLS, but has an invalid x509 certificate:

qemu-system-x86_64: The certificate hasn't got a known issuer
qemu-system-x86_64: migration/migration.c:386: process_incoming_migration_co: 
Assertion `mis->from_src_file' failed.

 #0  0x00007fffebd24f2b in raise () at /lib64/libc.so.6
 #1  0x00007fffebd0f561 in abort () at /lib64/libc.so.6
 #2  0x00007fffebd0f431 in _nl_load_domain.cold.0 () at /lib64/libc.so.6
 #3  0x00007fffebd1d692 in  () at /lib64/libc.so.6
 #4  0x0000555555ad027e in process_incoming_migration_co (opaque=<optimized 
out>) at migration/migration.c:386
 #5  0x0000555555c45e8b in coroutine_trampoline (i0=<optimized out>, 
i1=<optimized out>) at util/coroutine-ucontext.c:116
 #6  0x00007fffebd3a6a0 in __start_context () at /lib64/libc.so.6
 #7  0x0000000000000000 in  ()

To handle the non-multifd case, we check whether mis->from_src_file
is non-NULL. With this in place, the migration server drops the
rejected client and stays around waiting for another, hopefully
valid, client to arrive.

Signed-off-by: Daniel P. Berrangé <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: 4a83bf2f339d4b63ecd5ef48b9816c3b7ee24553
      
https://github.com/qemu/qemu/commit/4a83bf2f339d4b63ecd5ef48b9816c3b7ee24553
  Author: Peter Maydell <address@hidden>
  Date:   2018-06-28 (Thu, 28 Jun 2018)

  Changed paths:
    M exec.c
    M hmp.c
    M include/exec/ram_addr.h
    M migration/migration.c
    M migration/postcopy-ram.c
    M migration/postcopy-ram.h
    M migration/ram.c
    M migration/trace-events
    M qapi/migration.json

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20180627' 
into staging

migration/next for 20180627

# gpg: Signature made Wed 27 Jun 2018 13:53:53 BST
# gpg:                using RSA key F487EF185872D723
# gpg: Good signature from "Juan Quintela <address@hidden>"
# gpg:                 aka "Juan Quintela <address@hidden>"
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03  4B82 F487 EF18 5872 D723

* remotes/juanquintela/tags/migration/20180627:
  migration: fix crash in when incoming client channel setup fails
  postcopy: drop ram_pages parameter from postcopy_ram_incoming_init()
  migration: Stop sending whole pages through main channel
  migration: Remove not needed semaphore and quit
  migration: Wait for blocking IO
  migration: Start sending messages
  migration: Create ram_save_multifd_page
  migration: Create multifd_bytes ram_counter
  migration: Synchronize multifd threads with main thread
  migration: Add block where to send/receive packets
  migration: Multifd channels always wait on the sem
  migration: Add multifd traces for start/end thread
  migration: Abstract the number of bytes sent
  migration: Calculate mbps only during transfer time
  migration: Create multifd packet
  migration: Create multipage support

Signed-off-by: Peter Maydell <address@hidden>


Compare: https://github.com/qemu/qemu/compare/18896679aa98...4a83bf2f339d
      **NOTE:** This service been marked for deprecation: 
https://developer.github.com/changes/2018-04-25-github-services-deprecation/

      Functionality will be removed from GitHub.com on January 31st, 2019.
[Prev in Thread]
Current Thread
[Next in Thread]
[Qemu-commits] [qemu/qemu] 34c55a: migration: Create multipage support, GitHub <=
Prev by Date: [Qemu-commits] [qemu/qemu] c92023: MAINTAINERS: update target-mips maintainers
Next by Date: [Qemu-commits] [qemu/qemu] f18793: compiler: add a sizeof_field() macro
Previous by thread: [Qemu-commits] [qemu/qemu] c92023: MAINTAINERS: update target-mips maintainers
Next by thread: [Qemu-commits] [qemu/qemu] f18793: compiler: add a sizeof_field() macro
Index(es):
- Date
- Thread