[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Qemu-commits] [qemu/qemu] ddec20: multifd: Create page_size fields into
|
From: |
Peter Maydell |
|
Subject: |
[Qemu-commits] [qemu/qemu] ddec20: multifd: Create page_size fields into both MultiFD... |
|
Date: |
Thu, 15 Dec 2022 06:52:40 -0800 |
Branch: refs/heads/staging
Home: https://github.com/qemu/qemu
Commit: ddec20f858800cfe3ff7f56aec486dab0585e8b1
https://github.com/qemu/qemu/commit/ddec20f858800cfe3ff7f56aec486dab0585e8b1
Author: Juan Quintela <quintela@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/multifd-zlib.c
M migration/multifd-zstd.c
M migration/multifd.c
M migration/multifd.h
Log Message:
-----------
multifd: Create page_size fields into both MultiFD{Recv,Send}Params
We were calling qemu_target_page_size() left and right.
Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Leonardo Bras <leobras@redhat.com>
Commit: d6f45eba2bb1dbf48c05adcf3e14040b913d2085
https://github.com/qemu/qemu/commit/d6f45eba2bb1dbf48c05adcf3e14040b913d2085
Author: Juan Quintela <quintela@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/multifd.c
M migration/multifd.h
Log Message:
-----------
multifd: Create page_count fields into both MultiFD{Recv,Send}Params
We were recalculating it left and right. We plan to change that
values on next patches.
Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Leonardo Bras <leobras@redhat.com>
Commit: 26a2606916781dc83d375bda439290d7baa80498
https://github.com/qemu/qemu/commit/26a2606916781dc83d375bda439290d7baa80498
Author: Juan Quintela <quintela@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/ram.c
M migration/ram.h
Log Message:
-----------
migration: Export ram_transferred_ram()
Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: David Edmondson <david.edmondson@oracle.com>
Reviewed-by: Leonardo Bras <leobras@redhat.com>
Commit: a4dbaf8eedbb81060ce6963f118442f7aa0ba61e
https://github.com/qemu/qemu/commit/a4dbaf8eedbb81060ce6963f118442f7aa0ba61e
Author: Juan Quintela <quintela@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/ram.c
M migration/ram.h
Log Message:
-----------
migration: Export ram_release_page()
Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Leonardo Bras <leobras@redhat.com>
Commit: c13221b56faaaf732de95d05a7742c1913363b48
https://github.com/qemu/qemu/commit/c13221b56faaaf732de95d05a7742c1913363b48
Author: Peter Xu <peterx@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/ram.c
Log Message:
-----------
migration: Take bitmap mutex when completing ram migration
Any call to ram_find_and_save_block() needs to take the bitmap mutex. We
used to not take it for most of ram_save_complete() because we thought
we're the only one left using the bitmap, but it's not true after the
preempt full patchset applied, since the return path can be taking it too.
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Commit: 20123ee1ded768a908d96b39b17ea281dfdb0ea6
https://github.com/qemu/qemu/commit/20123ee1ded768a908d96b39b17ea281dfdb0ea6
Author: Peter Xu <peterx@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/ram.c
Log Message:
-----------
migration: Add postcopy_preempt_active()
Add the helper to show that postcopy preempt enabled, meanwhile active.
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Commit: ef5c3d13916f04176ed2d6d8405fb678e50b043f
https://github.com/qemu/qemu/commit/ef5c3d13916f04176ed2d6d8405fb678e50b043f
Author: Peter Xu <peterx@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/ram.c
Log Message:
-----------
migration: Cleanup xbzrle zero page cache update logic
The major change is to replace "!save_page_use_compression()" with
"xbzrle_enabled" to make it clear.
Reasonings:
(1) When compression enabled, "!save_page_use_compression()" is exactly the
same as checking "xbzrle_enabled".
(2) When compression disabled, "!save_page_use_compression()" always return
true. We used to try calling the xbzrle code, but after this change we
won't, and we shouldn't need to.
Since at it, drop the xbzrle_enabled check in xbzrle_cache_zero_page()
because with this change it's not needed anymore.
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Commit: 10661f118048265df33cc30597ecb346c0f092c1
https://github.com/qemu/qemu/commit/10661f118048265df33cc30597ecb346c0f092c1
Author: Peter Xu <peterx@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/ram.c
Log Message:
-----------
migration: Trivial cleanup save_page_header() on same block check
The 2nd check on RAM_SAVE_FLAG_CONTINUE is a bit redundant. Use a boolean
to be clearer.
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Commit: eaa238ab31d9d74ad09100b0f56dfe21f299eb30
https://github.com/qemu/qemu/commit/eaa238ab31d9d74ad09100b0f56dfe21f299eb30
Author: Peter Xu <peterx@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/ram.c
Log Message:
-----------
migration: Remove RAMState.f references in compression code
Removing referencing to RAMState.f in compress_page_with_multi_thread() and
flush_compressed_data().
Compression code by default isn't compatible with having >1 channels (or it
won't currently know which channel to flush the compressed data), so to
make it simple we always flush on the default to_dst_file port until
someone wants to add >1 ports support, as rs->f right now can really
change (after postcopy preempt is introduced).
There should be no functional change at all after patch applied, since as
long as rs->f referenced in compression code, it must be to_dst_file.
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Commit: f3321554ef7e3adc3be6f697185d3bd50d60e30e
https://github.com/qemu/qemu/commit/f3321554ef7e3adc3be6f697185d3bd50d60e30e
Author: Peter Xu <peterx@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/ram.c
Log Message:
-----------
migration: Yield bitmap_mutex properly when sending/sleeping
Don't take the bitmap mutex when sending pages, or when being throttled by
migration_rate_limit() (which is a bit tricky to call it here in ram code,
but seems still helpful).
It prepares for the possibility of concurrently sending pages in >1 threads
using the function ram_save_host_page() because all threads may need the
bitmap_mutex to operate on bitmaps, so that either sendmsg() or any kind of
qemu_sem_wait() blocking for one thread will not block the other from
progressing.
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Commit: 23b7576d7879cdc93e3328e1a9228e8843780012
https://github.com/qemu/qemu/commit/23b7576d7879cdc93e3328e1a9228e8843780012
Author: Peter Xu <peterx@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/migration.c
M migration/multifd.c
M migration/ram.c
M migration/ram.h
Log Message:
-----------
migration: Use atomic ops properly for page accountings
To prepare for thread-safety on page accountings, at least below counters
need to be accessed only atomically, they are:
ram_counters.transferred
ram_counters.duplicate
ram_counters.normal
ram_counters.postcopy_bytes
There are a lot of other counters but they won't be accessed outside
migration thread, then they're still safe to be accessed without atomic
ops.
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Commit: d9e474ea564bc109bc6fc81323ae90a7c9e7f04f
https://github.com/qemu/qemu/commit/d9e474ea564bc109bc6fc81323ae90a7c9e7f04f
Author: Peter Xu <peterx@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/ram.c
Log Message:
-----------
migration: Teach PSS about host page
Migration code has a lot to do with host pages. Teaching PSS core about
the idea of host page helps a lot and makes the code clean. Meanwhile,
this prepares for the future changes that can leverage the new PSS helpers
that this patch introduces to send host page in another thread.
Three more fields are introduced for this:
(1) host_page_sending: this is set to true when QEMU is sending a host
page, false otherwise.
(2) host_page_{start|end}: these point to the start/end of host page
we're sending, and it's only valid when host_page_sending==true.
For example, when we look up the next dirty page on the ramblock, with
host_page_sending==true, we'll not try to look for anything beyond the
current host page boundary. This can be slightly efficient than current
code because currently we'll set pss->page to next dirty bit (which can be
over current host page boundary) and reset it to host page boundary if we
found it goes beyond that.
With above, we can easily make migration_bitmap_find_dirty() self contained
by updating pss->page properly. rs* parameter is removed because it's not
even used in old code.
When sending a host page, we should use the pss helpers like this:
- pss_host_page_prepare(pss): called before sending host page
- pss_within_range(pss): whether we're still working on the cur host page?
- pss_host_page_finish(pss): called after sending a host page
Then we can use ram_save_target_page() to save one small page.
Currently ram_save_host_page() is still the only user. If there'll be
another function to send host page (e.g. in return path thread) in the
future, it should follow the same style.
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Commit: 61717ea9d21c8701f312b647a8ea092c5dfbf8e4
https://github.com/qemu/qemu/commit/61717ea9d21c8701f312b647a8ea092c5dfbf8e4
Author: Peter Xu <peterx@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/ram.c
Log Message:
-----------
migration: Introduce pss_channel
Introduce pss_channel for PageSearchStatus, define it as "the migration
channel to be used to transfer this host page".
We used to have rs->f, which is a mirror to MigrationState.to_dst_file.
After postcopy preempt initial version, rs->f can be dynamically changed
depending on which channel we want to use.
But that later work still doesn't grant full concurrency of sending pages
in e.g. different threads, because rs->f can either be the PRECOPY channel
or POSTCOPY channel. This needs to be per-thread too.
PageSearchStatus is actually a good piece of struct which we can leverage
if we want to have multiple threads sending pages. Sending a single guest
page may not make sense, so we make the granule to be "host page", and in
the PSS structure we allow specify a QEMUFile* to migrate a specific host
page. Then we open the possibility to specify different channels in
different threads with different PSS structures.
The PSS prefix can be slightly misleading here because e.g. for the
upcoming usage of postcopy channel/thread it's not "searching" (or,
scanning) at all but sending the explicit page that was requested. However
since PSS existed for some years keep it as-is until someone complains.
This patch mostly (simply) replace rs->f with pss->pss_channel only. No
functional change intended for this patch yet. But it does prepare to
finally drop rs->f, and make ram_save_guest_page() thread safe.
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Commit: ebd88a49736f4e3af0aa4004ceb0980698659ebe
https://github.com/qemu/qemu/commit/ebd88a49736f4e3af0aa4004ceb0980698659ebe
Author: Peter Xu <peterx@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/ram.c
Log Message:
-----------
migration: Add pss_init()
Helper to init PSS structures.
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Commit: f166876423a4c8ba706e31075e21ae4ff92b4332
https://github.com/qemu/qemu/commit/f166876423a4c8ba706e31075e21ae4ff92b4332
Author: Peter Xu <peterx@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/ram.c
Log Message:
-----------
migration: Make PageSearchStatus part of RAMState
We used to allocate PSS structure on the stack for precopy when sending
pages. Make it static, so as to describe per-channel ram migration status.
Here we declared RAM_CHANNEL_MAX instances, preparing for postcopy to use
it, even though this patch has not yet to start using the 2nd instance.
This should not have any functional change per se, but it already starts to
export PSS information via the RAMState, so that e.g. one PSS channel can
start to reference the other PSS channel.
Always protect PSS access using the same RAMState.bitmap_mutex. We already
do so, so no code change needed, just some comment update. Maybe we should
consider renaming bitmap_mutex some day as it's going to be a more commonly
and big mutex we use for ram states, but just leave it for later.
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Commit: ec6f3ab9f489ecabec2b660887c38b6a95c85e41
https://github.com/qemu/qemu/commit/ec6f3ab9f489ecabec2b660887c38b6a95c85e41
Author: Peter Xu <peterx@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/ram.c
Log Message:
-----------
migration: Move last_sent_block into PageSearchStatus
Since we use PageSearchStatus to represent a channel, it makes perfect
sense to keep last_sent_block (aka, leverage RAM_SAVE_FLAG_CONTINUE) to be
per-channel rather than global because each channel can be sending
different pages on ramblocks.
Hence move it from RAMState into PageSearchStatus.
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Commit: 93589827443e7fbd68ea39e3d70dc7fb1460bb84
https://github.com/qemu/qemu/commit/93589827443e7fbd68ea39e3d70dc7fb1460bb84
Author: Peter Xu <peterx@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/migration.c
M migration/ram.c
Log Message:
-----------
migration: Send requested page directly in rp-return thread
With all the facilities ready, send the requested page directly in the
rp-return thread rather than queuing it in the request queue, if and only
if postcopy preempt is enabled. It can achieve so because it uses separate
channel for sending urgent pages. The only shared data is bitmap and it's
protected by the bitmap_mutex.
Note that since we're moving the ownership of the urgent channel from the
migration thread to rp thread it also means the rp thread is responsible
for managing the qemufile, e.g. properly close it when pausing migration
happens. For this, let migration_release_from_dst_file to cover shutdown
of the urgent channel too, renaming it as migration_release_dst_files() to
better show what it does.
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Commit: b062106d3a06837b8a428ebb7e72f2aeabe0447b
https://github.com/qemu/qemu/commit/b062106d3a06837b8a428ebb7e72f2aeabe0447b
Author: Peter Xu <peterx@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/migration.c
M migration/migration.h
M migration/ram.c
Log Message:
-----------
migration: Remove old preempt code around state maintainance
With the new code to send pages in rp-return thread, there's little help to
keep lots of the old code on maintaining the preempt state in migration
thread, because the new way should always be faster..
Then if we'll always send pages in the rp-return thread anyway, we don't
need those logic to maintain preempt state anymore because now we serialize
things using the mutex directly instead of using those fields.
It's very unfortunate to have those code for a short period, but that's
still one intermediate step that we noticed the next bottleneck on the
migration thread. Now what we can do best is to drop unnecessary code as
long as the new code is stable to reduce the burden. It's actually a good
thing because the new "sending page in rp-return thread" model is (IMHO)
even cleaner and with better performance.
Remove the old code that was responsible for maintaining preempt states, at
the meantime also remove x-postcopy-preempt-break-huge parameter because
with concurrent sender threads we don't really need to break-huge anymore.
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Commit: 7f401b80445e8746202a6d643410ba1b9eeb3cb1
https://github.com/qemu/qemu/commit/7f401b80445e8746202a6d643410ba1b9eeb3cb1
Author: Peter Xu <peterx@redhat.com>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/ram.c
Log Message:
-----------
migration: Drop rs->f
Now with rs->pss we can already cache channels in pss->pss_channels. That
pss_channel contains more infromation than rs->f because it's per-channel.
So rs->f could be replaced by rss->pss[RAM_CHANNEL_PRECOPY].pss_channel,
while rs->f itself is a bit vague now.
Note that vanilla postcopy still send pages via pss[RAM_CHANNEL_PRECOPY],
that's slightly confusing but it reflects the reality.
Then, after the replacement we can safely drop rs->f.
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Commit: 928eac953918cbd9b237d7cb8b937a6fc575d009
https://github.com/qemu/qemu/commit/928eac953918cbd9b237d7cb8b937a6fc575d009
Author: Peter Maydell <peter.maydell@linaro.org>
Date: 2022-12-15 (Thu, 15 Dec 2022)
Changed paths:
M migration/migration.c
M migration/migration.h
M migration/multifd-zlib.c
M migration/multifd-zstd.c
M migration/multifd.c
M migration/multifd.h
M migration/ram.c
M migration/ram.h
Log Message:
-----------
Merge tag 'next-8.0-pull-request' of https://gitlab.com/juan.quintela/qemu
into staging
Migration patches for 8.0
Hi
This are the patches that I had to drop form the last PULL request because they
werent fixes:
- AVX2 is dropped, intel posted a fix, I have to redo it
- Fix for out of order channels is out
Daniel nacked it and I need to redo it
# gpg: Signature made Thu 15 Dec 2022 09:38:29 GMT
# gpg: using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [full]
# gpg: aka "Juan Quintela <quintela@trasno.org>" [full]
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03 4B82 F487 EF18 5872 D723
* tag 'next-8.0-pull-request' of https://gitlab.com/juan.quintela/qemu:
migration: Drop rs->f
migration: Remove old preempt code around state maintainance
migration: Send requested page directly in rp-return thread
migration: Move last_sent_block into PageSearchStatus
migration: Make PageSearchStatus part of RAMState
migration: Add pss_init()
migration: Introduce pss_channel
migration: Teach PSS about host page
migration: Use atomic ops properly for page accountings
migration: Yield bitmap_mutex properly when sending/sleeping
migration: Remove RAMState.f references in compression code
migration: Trivial cleanup save_page_header() on same block check
migration: Cleanup xbzrle zero page cache update logic
migration: Add postcopy_preempt_active()
migration: Take bitmap mutex when completing ram migration
migration: Export ram_release_page()
migration: Export ram_transferred_ram()
multifd: Create page_count fields into both MultiFD{Recv,Send}Params
multifd: Create page_size fields into both MultiFD{Recv,Send}Params
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Compare: https://github.com/qemu/qemu/compare/9c2e982f6887...928eac953918
- [Qemu-commits] [qemu/qemu] ddec20: multifd: Create page_size fields into both MultiFD...,
Peter Maydell <=