On Thu, Nov 30, 2023 at 4:14 PM João Jandre Paraquetti
<joao@scclouds.com.br> wrote:
Hi, all
I recently started looking into how to create incremental backups using
Libvirt+Qemu. I have already found that, since Libvirt 7.6.0, we can use
the virDomainBackupBegin API to create incremental backups of live VMs
through Libvirt. Using this API we can back up the VM's disks and create
a checkpoint at the same time. After the first full backup, we can
create incremental backups referencing the checkpoint that was generated
on the previous backup process.
As far as I understood Libvirt's documentation, when using this API to
create a checkpoint, what happens is that Libvirt will create a new
bitmap on the VM's disk to track the changes that happened from that
checkpoint on, then, they'll copy what changed between the last bitmap
and the new one. By doing this, Libvirt is able to create incremental
backups of disks that are used by running VMs.
My problem is the following though: after I stop the VM, I'm no longer
able to use Libvirts API as it requires the VM (domain) to be running.
However, there might still be changes on the disk since the last backup
I performed. Therefore, to create a backup of the changes made since the
last backup until the VM was stopped, I have to work directly on the VM
image using Qemu. I have checked, and the QCOW2 file has the checkpoint
created with the Libvirt backup process (I checked via the "qemu-img
info"). Therefore, we have the marker of the last checkpoint generated
by the last backup process. However, if the VM is stopped, there is no
clear way to export this last differencial copy of the volume.
I have searched through the documentation about this; however, I haven't
found any example or explanation on how to manually export a
differencial copy of the QCOW2 file using the last checkpoint (bitmap)
created.
I would much appreaciate, if people could point me to the direction on
how to export differencial copies of a QCOW2 file, using the checkpoints
(bitmaps) that it has.
Backup is complicated and a lot of hard work. Before you reinvent the
wheel, check
if this project project works for you:
https://github.com/abbbi/virtnbdbackup
If you need to create your own solution of want to understand how backup works
with bitmaps, here is how it works.
The basic idea is - you start an nbd server exporting the dirty
bitmap, so the client can
get the dirty extents using NBD_BLOCK_STATUS and copy the dirty clusters.
To start an nbd server you have 2 options:
- qemu-nbd using the --bitmap option. vdsm code is a good place to
learn how to do this
(more about this below). This was the best option few years ago, but
now you may want
the second option.
- qemu-storage-daemon - you configure and start the nbd server in the same
way libvirt does it - using the qmp commands. libvirt code is
probably the best place
to learn how to do this. This is probably the best option at this
time, unless you are
using an old qemu version that does not have working qemu-storage-daemon.
If you have a single qcow2 image with the bitmap, you are ready to
export the image.
But you may have a more interesting image when the same bitmap exists
in multiple
snapshots. You can see all the bitmaps and snapshots using:
qemu-img info --backing-chain --output json filename
If the bitmap exists on multiple images in the chain, you need to
validate that the bitmap
is valid - it must exist in all images in the chain since the bitmap
was created and must have
the right flags (see vdsm code below for the details).
After validating the bitmap, you need to create a new bitmap with all
the bits in the same bitmap
in all the images in the chain. You have 2 ways to do this:
- with qemu-nbd: create an overlay on top of the image, create a
bitmap in the overlay, and merge
the bitmaps from the image chain into this bitmap. Then start
qemu-nbd with the overlay. You can
create and merge bitmaps using `qemu-img bitmap ...` (see vdsm code
below for the details)
- with qemu-storage-daemon: you can do what libvirt does - create a
new temporary non-persistent bitmap
(in qemu-storage-daemon memory), and merge the other bitmaps in the
chain using qmp commands.
(see libvirt source or debug logs for the details).
When the backup bitmap is ready, you can start the nbd server and
configure it to export this bitmap.
When backup is done delete the overlay or the temporary backup bitmap.
You can check how vdsm does this using qemu-nbd here:
https://github.com/oVirt/vdsm/blob/0fc22ff0b81d605f10a2bc67309e119b7462b506/lib/vdsm/storage/nbd.py#L97
The steps:
1. Finding the images with the bitmap and validating the bitmap
https://github.com/oVirt/vdsm/blob/0fc22ff0b81d605f10a2bc67309e119b7462b506/lib/vdsm/storage/nbd.py#L217
2. Creating overlay with a backup bitmap
https://github.com/oVirt/vdsm/blob/0fc22ff0b81d605f10a2bc67309e119b7462b506/lib/vdsm/storage/nbd.py#L184
3. Starting qemu-nbd with exporting the bitmap using systemd-run:
https://github.com/oVirt/vdsm/blob/0fc22ff0b81d605f10a2bc67309e119b7462b506/lib/vdsm/storage/nbd.py#L419
Nir