Re: [PATCH v4 11/34] migration/ram: Introduce 'fixed-ram' migration capability

[Peter Xu]

On Tue, Feb 20, 2024 at 07:41:15PM -0300, Fabiano Rosas wrote:
> Add a new migration capability 'fixed-ram'.
> 
> The core of the feature is to ensure that each RAM page has a specific
> offset in the resulting migration stream. The reasons why we'd want
> such behavior are:
> 
>  - The resulting file will have a bounded size, since pages which are
>    dirtied multiple times will always go to a fixed location in the
>    file, rather than constantly being added to a sequential
>    stream. This eliminates cases where a VM with, say, 1G of RAM can
>    result in a migration file that's 10s of GBs, provided that the
>    workload constantly redirties memory.
> 
>  - It paves the way to implement O_DIRECT-enabled save/restore of the
>    migration stream as the pages are ensured to be written at aligned
>    offsets.
> 
>  - It allows the usage of multifd so we can write RAM pages to the
>    migration file in parallel.
> 
> For now, enabling the capability has no effect. The next couple of
> patches implement the core functionality.
> 
> Signed-off-by: Fabiano Rosas <farosas@suse.de>
> ---
> - update migration.json to 9.0 and improve wording
> - move docs to a separate file and add use cases information
> ---
>  docs/devel/migration/features.rst  |   1 +
>  docs/devel/migration/fixed-ram.rst | 137 +++++++++++++++++++++++++++++
>  migration/options.c                |  34 +++++++
>  migration/options.h                |   1 +
>  migration/savevm.c                 |   1 +
>  qapi/migration.json                |   6 +-
>  6 files changed, 179 insertions(+), 1 deletion(-)
>  create mode 100644 docs/devel/migration/fixed-ram.rst
> 
> diff --git a/docs/devel/migration/features.rst 
> b/docs/devel/migration/features.rst
> index a9acaf618e..4c708b679a 100644
> --- a/docs/devel/migration/features.rst
> +++ b/docs/devel/migration/features.rst
> @@ -10,3 +10,4 @@ Migration has plenty of features to support different use 
> cases.
>     dirty-limit
>     vfio
>     virtio
> +   fixed-ram
> diff --git a/docs/devel/migration/fixed-ram.rst 
> b/docs/devel/migration/fixed-ram.rst
> new file mode 100644
> index 0000000000..a6c0e5a360
> --- /dev/null
> +++ b/docs/devel/migration/fixed-ram.rst
> @@ -0,0 +1,137 @@
> +Fixed-ram
> +=========
> +
> +Fixed-ram is a new stream format for the RAM section designed to
> +supplement the existing ``file:`` migration and make it compatible
> +with ``multifd``. This enables parallel migration of a guest's RAM to
> +a file.
> +
> +The core of the feature is to ensure that each RAM page has a specific
> +offset in the resulting migration file. This enables the ``multifd``
> +threads to write exclusively to those offsets even if the guest is
> +constantly dirtying pages (i.e. live migration). Another benefit is
> +that the resulting file will have a bounded size, since pages which
> +are dirtied multiple times will always go to a fixed location in the
> +file, rather than constantly being added to a sequential
> +stream. Having the pages at fixed offsets also allows the usage of
> +O_DIRECT for save/restore of the migration stream as the pages are
> +ensured to be written respecting O_DIRECT alignment restrictions.
> +
> +Usage
> +-----
> +
> +On both source and destination, enable the ``multifd`` and
> +``fixed-ram`` capabilities:
> +
> +    ``migrate_set_capability multifd on``
> +
> +    ``migrate_set_capability fixed-ram on``
> +
> +Use a ``file:`` URL for migration:
> +
> +    ``migrate file:/path/to/migration/file``
> +
> +Fixed-ram migration is best done non-live, i.e. by stopping the VM on
> +the source side before migrating.
> +
> +For best performance enable the ``direct-io`` capability as well:
> +
> +    ``migrate_set_capability direct-io on``
> +
> +Use-cases
> +---------
> +
> +The fixed-ram feature was designed for use cases where the migration
> +stream will be directed to a file in the filesystem and not
> +immediately restored on the destination VM [#]_. These could be
> +thought of as snapshots. We can further categorize them into live and
> +non-live.
> +
> +- Non-live snapshot
> +
> +If the use case requires a VM to be stopped before taking a snapshot,
> +that's the ideal scenario for fixed-ram migration. Not having to track
> +dirty pages, the migration will write the RAM pages to the disk as
> +fast as it can.
> +
> +Note: if a snapshot is taken of a running VM, but the VM will be
> +stopped after the snapshot by the admin, then consider stopping it
> +right before the snapshot to take benefit of the performance gains
> +mentioned above.
> +
> +- Live snapshot
> +
> +If the use case requires that the VM keeps running during and after
> +the snapshot operation, then fixed-ram migration can still be used,
> +but will be less performant. Other strategies such as
> +background-snapshot should be evaluated as well. One benefit of
> +fixed-ram in this scenario is portability since background-snapshot
> +depends on async dirty tracking (KVM_GET_DIRTY_LOG) which is not

Background snapshot uses userfaultfd-wp rather than KVM_GET_DIRTY_LOG.  The
statement is still correct though, that userfault is only supported on
Linux in general (wp is one sub-feature, represents "write-protect mode")
so this should help portability, as it removes the dependency on the OS.

> +supported outside of Linux.
> +
> +.. [#] While this same effect could be obtained with the usage of
> +       snapshots or the ``file:`` migration alone, fixed-ram provides
> +       a performance increase for VMs with larger RAM sizes (10s to
> +       100s of GiBs), specially if the VM has been stopped beforehand.
> +
> +RAM section format
> +------------------
> +
> +Instead of having a sequential stream of pages that follow the
> +RAMBlock headers, the dirty pages for a RAMBlock follow its header
> +instead. This ensures that each RAM page has a fixed offset in the
> +resulting migration file.
> +
> +A bitmap is introduced to track which pages have been written in the
> +migration file. Pages are written at a fixed location for every
> +ramblock. Zero pages are ignored as they'd be zero in the destination
> +migration as well.
> +
> +::
> +
> + Without fixed-ram:                  With fixed-ram:
> +
> + ---------------------               --------------------------------
> + | ramblock 1 header |               | ramblock 1 header            |
> + ---------------------               --------------------------------
> + | ramblock 2 header |               | ramblock 1 fixed-ram header  |
> + ---------------------               --------------------------------
> + | ...               |               | padding to next 1MB boundary |
> + ---------------------               | ...                          |
> + | ramblock n header |               --------------------------------
> + ---------------------               | ramblock 1 pages             |
> + | RAM_SAVE_FLAG_EOS |               | ...                          |
> + ---------------------               --------------------------------
> + | stream of pages   |               | ramblock 2 header            |
> + | (iter 1)          |               --------------------------------
> + | ...               |               | ramblock 2 fixed-ram header  |
> + ---------------------               --------------------------------
> + | RAM_SAVE_FLAG_EOS |               | padding to next 1MB boundary |
> + ---------------------               | ...                          |
> + | stream of pages   |               --------------------------------
> + | (iter 2)          |               | ramblock 2 pages             |
> + | ...               |               | ...                          |
> + ---------------------               --------------------------------
> + | ...               |               | ...                          |
> + ---------------------               --------------------------------
> +                                     | RAM_SAVE_FLAG_EOS            |
> +                                     --------------------------------
> +                                     | ...                          |
> +                                     --------------------------------
> +
> + where:
> +  - ramblock header: the generic information for a ramblock, such as
> +    idstr, used_len, etc.
> +
> +  - ramblock fixed-ram header: the information added by this feature:
> +    bitmap of pages written, bitmap size and offset of pages in the
> +    migration file.
> +
> +Restrictions
> +------------
> +
> +Since pages are written to their relative offsets and out of order
> +(due to the memory dirtying patterns), streaming channels such as
> +sockets are not supported. A seekable channel such as a file is
> +required. This can be verified in the QIOChannel by the presence of
> +the QIO_CHANNEL_FEATURE_SEEKABLE.
> diff --git a/migration/options.c b/migration/options.c
> index 3e3e0b93b4..4909e5c72a 100644
> --- a/migration/options.c
> +++ b/migration/options.c
> @@ -204,6 +204,7 @@ Property migration_properties[] = {
>      DEFINE_PROP_MIG_CAP("x-switchover-ack",
>                          MIGRATION_CAPABILITY_SWITCHOVER_ACK),
>      DEFINE_PROP_MIG_CAP("x-dirty-limit", MIGRATION_CAPABILITY_DIRTY_LIMIT),
> +    DEFINE_PROP_MIG_CAP("x-fixed-ram", MIGRATION_CAPABILITY_FIXED_RAM),

Let's directly use "fixed-ram" (or "mapped-ram", or whatever new name we
decide to use, as long as without "x-")?

migration_properties is not documented anywhere, mostly yet for debugging
purpose.  We could have dropped all the "x-"s, IMHO.

>      DEFINE_PROP_END_OF_LIST(),
>  };
>  
> @@ -263,6 +264,13 @@ bool migrate_events(void)
>      return s->capabilities[MIGRATION_CAPABILITY_EVENTS];
>  }
>  
> +bool migrate_fixed_ram(void)
> +{
> +    MigrationState *s = migrate_get_current();
> +
> +    return s->capabilities[MIGRATION_CAPABILITY_FIXED_RAM];
> +}
> +
>  bool migrate_ignore_shared(void)
>  {
>      MigrationState *s = migrate_get_current();
> @@ -645,6 +653,32 @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, 
> Error **errp)
>          }
>      }
>  
> +    if (new_caps[MIGRATION_CAPABILITY_FIXED_RAM]) {
> +        if (new_caps[MIGRATION_CAPABILITY_MULTIFD]) {
> +            error_setg(errp,
> +                       "Fixed-ram migration is incompatible with multifd");
> +            return false;
> +        }
> +
> +        if (new_caps[MIGRATION_CAPABILITY_XBZRLE]) {
> +            error_setg(errp,
> +                       "Fixed-ram migration is incompatible with xbzrle");
> +            return false;
> +        }
> +
> +        if (new_caps[MIGRATION_CAPABILITY_COMPRESS]) {
> +            error_setg(errp,
> +                       "Fixed-ram migration is incompatible with 
> compression");
> +            return false;
> +        }
> +
> +        if (new_caps[MIGRATION_CAPABILITY_POSTCOPY_RAM]) {
> +            error_setg(errp,
> +                       "Fixed-ram migration is incompatible with postcopy 
> ram");
> +            return false;
> +        }
> +    }
> +
>      return true;
>  }
>  
> diff --git a/migration/options.h b/migration/options.h
> index 246c160aee..8680a10b79 100644
> --- a/migration/options.h
> +++ b/migration/options.h
> @@ -31,6 +31,7 @@ bool migrate_compress(void);
>  bool migrate_dirty_bitmaps(void);
>  bool migrate_dirty_limit(void);
>  bool migrate_events(void);
> +bool migrate_fixed_ram(void);
>  bool migrate_ignore_shared(void);
>  bool migrate_late_block_activate(void);
>  bool migrate_multifd(void);
> diff --git a/migration/savevm.c b/migration/savevm.c
> index d612c8a902..4b928dd6bb 100644
> --- a/migration/savevm.c
> +++ b/migration/savevm.c
> @@ -245,6 +245,7 @@ static bool should_validate_capability(int capability)
>      /* Validate only new capabilities to keep compatibility. */
>      switch (capability) {
>      case MIGRATION_CAPABILITY_X_IGNORE_SHARED:
> +    case MIGRATION_CAPABILITY_FIXED_RAM:
>          return true;
>      default:
>          return false;
> diff --git a/qapi/migration.json b/qapi/migration.json
> index 5a565d9b8d..3fce5fe53e 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -531,6 +531,10 @@
>  #     and can result in more stable read performance.  Requires KVM
>  #     with accelerator property "dirty-ring-size" set.  (Since 8.1)
>  #
> +# @fixed-ram: Migrate using fixed offsets in the migration file for
> +#     each RAM page.  Requires a migration URI that supports seeking,
> +#     such as a file.  (since 9.0)
> +#
>  # Features:
>  #
>  # @deprecated: Member @block is deprecated.  Use blockdev-mirror with
> @@ -555,7 +559,7 @@
>             { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] },
>             'validate-uuid', 'background-snapshot',
>             'zero-copy-send', 'postcopy-preempt', 'switchover-ack',
> -           'dirty-limit'] }
> +           'dirty-limit', 'fixed-ram'] }
>  
>  ##
>  # @MigrationCapabilityStatus:
> -- 
> 2.35.3
> 

-- 
Peter Xu