Re: Export checkpoint/bitmap from image on qcow2

[Nir Soffer]

On Thu, Nov 30, 2023 at 4:14 PM João Jandre Paraquetti
<joao@scclouds.com.br> wrote:
>
> Hi, all
>
> I recently started looking into how to create incremental backups using
> Libvirt+Qemu. I have already found that, since Libvirt 7.6.0, we can use
> the virDomainBackupBegin API to create incremental backups of live VMs
> through Libvirt. Using this API we can back up the VM's disks and create
> a checkpoint at the same time. After the first full backup, we can
> create incremental backups referencing the checkpoint that was generated
> on the previous backup process.
>
> As far as I understood Libvirt's documentation, when using this API to
> create a checkpoint, what happens is that Libvirt will create a new
> bitmap on the VM's disk to track the changes that happened from that
> checkpoint on, then, they'll copy what changed between the last bitmap
> and the new one. By doing this, Libvirt is able to create incremental
> backups of disks that are used by running VMs.
>
> My problem is the following though: after I stop the VM, I'm no longer
> able to use Libvirts API as it requires the VM (domain) to be running.
> However, there might still be changes on the disk since the last backup
> I performed. Therefore, to create a backup of the changes made since the
> last backup until the VM was stopped, I have to work directly on the VM
> image using Qemu. I have checked, and the QCOW2 file has the checkpoint
> created with the Libvirt backup process (I checked via the "qemu-img
> info"). Therefore, we have the marker of the last checkpoint generated
> by the last backup process. However, if the VM is stopped, there is no
> clear way to export this last differencial copy of the volume.
>
> I have searched through the documentation about this; however, I haven't
> found any example or explanation on how to manually export a
> differencial copy of the QCOW2 file using the last checkpoint (bitmap)
> created.
>
> I would much appreaciate, if people could point me to the direction on
> how to export differencial copies of a QCOW2 file, using the checkpoints
> (bitmaps) that it has.

Backup is complicated and a lot of hard work. Before you reinvent the
wheel, check
if this project project works for you:
https://github.com/abbbi/virtnbdbackup

If you need to create your own solution of want to understand how backup works
with bitmaps, here is how it works.

The basic idea is - you start an nbd server exporting the dirty
bitmap, so the client can
get the dirty extents using NBD_BLOCK_STATUS and copy the dirty clusters.

To start an nbd server you have 2 options:

- qemu-nbd using the --bitmap option. vdsm code is a good place to
learn how to do this
  (more about this below). This was the best option few years ago, but
now you may want
  the second option.

- qemu-storage-daemon - you configure and start the nbd server in the same
  way libvirt does it - using the qmp commands. libvirt code is
probably the best place
  to learn how to do this. This is probably the best option at this
time, unless you are
  using an old qemu version that does not have working qemu-storage-daemon.

If you have a single qcow2 image with the bitmap, you are ready to
export the image.
But you may have a more interesting image when the same bitmap exists
in multiple
snapshots. You can see all the bitmaps and snapshots using:

    qemu-img info --backing-chain --output json filename

If the bitmap exists on multiple images in the chain, you need to
validate that the bitmap
is valid - it must exist in all images in the chain since the bitmap
was created and must have
the right flags (see vdsm code below for the details).

After validating the bitmap, you need to create a new bitmap with all
the bits in the same bitmap
in all the images in the chain. You have 2 ways to do this:

- with qemu-nbd: create an overlay on top of the image, create a
bitmap in the overlay, and merge
  the bitmaps from the image chain into this bitmap. Then start
qemu-nbd with the overlay. You can
  create and merge bitmaps using `qemu-img bitmap ...` (see vdsm code
below for the details)

- with qemu-storage-daemon: you can do what libvirt does - create a
new temporary non-persistent bitmap
  (in qemu-storage-daemon memory), and merge the other bitmaps in the
chain using qmp commands.
  (see libvirt source or debug logs for the details).

When the backup bitmap is ready, you can start the nbd server and
configure it to export this bitmap.

When backup is done delete the overlay or the temporary backup bitmap.

You can check how vdsm does this using qemu-nbd here:
https://github.com/oVirt/vdsm/blob/0fc22ff0b81d605f10a2bc67309e119b7462b506/lib/vdsm/storage/nbd.py#L97

The steps:

1. Finding the images with the bitmap and validating the bitmap
   
https://github.com/oVirt/vdsm/blob/0fc22ff0b81d605f10a2bc67309e119b7462b506/lib/vdsm/storage/nbd.py#L217

2. Creating overlay with a backup bitmap
   
https://github.com/oVirt/vdsm/blob/0fc22ff0b81d605f10a2bc67309e119b7462b506/lib/vdsm/storage/nbd.py#L184

3. Starting qemu-nbd with exporting the bitmap using systemd-run:
   
https://github.com/oVirt/vdsm/blob/0fc22ff0b81d605f10a2bc67309e119b7462b506/lib/vdsm/storage/nbd.py#L419

Nir