Re: Question on Compression for Raw Image

[Richard W.M. Jones]

On Tue, Oct 20, 2020 at 09:32:23AM -0500, Eric Blake wrote:
> On 10/20/20 9:22 AM, Wang, Wei W wrote:
> > On Tuesday, October 20, 2020 4:01 PM, Kevin Wolf wrote:
> >> Am 20.10.2020 um 03:31 hat Wang, Wei W geschrieben:
> >>> Hi,
> >>>
> >>> Does anyone know the reason why raw-format.c doesn't have
> >> compression
> >>> support (but qcow has the supported added)?  For example, raw image
> >>> backup with compression, "qemu-img convert -c -O raw origin.img
> >>> dist.img", doesn't work.
> >>
> >> A raw image is by definition a file that contains the exact same sequence 
> >> of
> >> bytes as the guest sees, without any additional information or encoding. If
> >> you compress a raw file, the guest will see compressed data on its hard 
> >> disk
> >> instead of the real data.
> > 
> > Ok, thanks. I'm thinking QEMU could do decompression of the compressed data 
> > in raw.img when guest reads data.
> > 
> >>
> >> Anything you could do to add transparent compression to it would mean that
> >> it's not a raw image any more, but a new image format.
> >>
> > Yes, decompression makes it transparent to the guest. Would you think it's 
> > good to reuse the raw image implementation, just add the 
> > compress/decompress option?
> 
> My recommendation would be implementing a new BDS filter that does
> uncompression.  Then, you could do things like:
> 
> raw -> decompress -> file.xz
> 
> or even
> 
> qcow2 -> decompress -> file.qcow2.xz
> 
> By the way, the notion of filters is already possible in other ways.
> For example, you can point qemu to read from an NBD server, and then use
> nbdkit to do the decompression with its filters:
> 
> nbdkit --filter=xz file file.xz
> raw -> nbd://localhost:10809

nbdkit can already act as a front-end to qemu-nbd, allowing you to use
the rich nbdkit filter ecosystem with qemu block devices.  There are
some examples in the link below, but --filter=xz could also be used
here:

https://libguestfs.org/nbdkit-nbd-plugin.1.html#Use-qemu-nbd-to-open-a-qcow2-file

> Also note that serving a decompressed view of a compressed image tends
> to be a read-only proposition (you really CAN'T write to the image
> without recompressing, but even if recompression has been blocked for
> parallelism, you would end up writing far more of the file after
> recompression than the amount of data written by a guest).

Yup!

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v