Re: [Qemu-discuss] Discard disk image holes for sparse shrink

[Pascal]

it's ok with  discard=unmap  option (even if the disc is thick-provisioned)
: the block used by the  test  file appears (VM side) or is reset (host
side) at zero.

notice that  lsblk  "thinks" that the disc is a rotational disk (eg. not
really SSD) :

lsblk -ndo /dev/sda
1

thanks for explanations !

Le mar. 9 avr. 2019 à 20:38, Narcis Garcia <address@hidden> a écrit :

> I see now the simple:
> -device virtio-scsi-pci
> makes all -disk (media=disk) to be detected by guest as SSD
>
> Additionally, adding "discard=unmap,detect-zeroes=unmap" to disk image
> specification makes host Qemu to discard sparse image holes and recover
> host disk space when guest sends discard signal!
>
> This is my successful test with Debian 9 (stable):
>
> $ dd if=/dev/zero of=ssd.img obs=1M seek=128 count=0
> $ qemu-system-... -device virtio-scsi-pci -drive
> file=ssd.img,media=disk,index=0,discard=unmap,detect-zeroes=unmap
>
> Thank you everyone.
>
>
> El 9/4/19 a les 19:19, Friedrich Oslage ha escrit:
> > 1. You need to set the discard option for the -drive parameter, eg
> > -drive if=none,...,format=raw,discard=unmap since the default is still
> > to simply ignore discards.
> >
> > You may also want to set the detect-zeroes option to unmap, to discard
> > all-zero blocks instead of actually writing them.
> >
> >
> > 2. The disk image you created is thick-provisioned. How is qemu supposed
> > to discard anything in that file? It could turn it into a sparse file or
> > overwrite it with zeroes I suppose, but both options are undesirable.
> >
> > To make use of the discard operation your backing storage has to support
> > it. Either by beeing thin-provisioned, like qcow2 or raw sparse files,
> > or by actually beeing a ssd/nvme disk.
> >
> >
> > 3. Your testing method is flawed. You are using a journaling filesystem
> > to write the data but to read it you access the block device directly .
> > Even if the discard operation was working strings(1) would still show
> > the hello-world string in the journal.
> >
> > You could try something like this:
> >
> > $ echo hello, world > /dev/sda
> > $ strings /dev/sda
> > $ blkdiscard /dev/sda
> > $ strings /dev/sda
> >
> >
> > Regards
> > Friedrich
> >
> > On 4/9/19 4:38 PM, Pascal wrote:
> >> the blocks do not seem to be discarded on a raw format disk: the data
> >> remains on the disk...
> >>
> >> on host :
> >>
> >> dd if=/dev/zero of=/tmp/ssd.disk bs=1M count=128
> >> qemu -drive media=disk,file=linux.disk -device virtio-scsi-pci,id=scsi
> >> -device scsi-hd,drive=hd -drive
> >> if=none,id=hd,file=/tmp/ssd.disk,format=raw
> >>
> >> on Linux VM :
> >>
> >> fdisk /dev/sda
> >>      # create one partition on gpt type...
> >> fdisk -l /dev/sda
> >>      Disk /dev/sda: 128 MiB, 134217728 bytes, 262144 sectors
> >>      Units: sectors of 1 * 512 = 512 bytes
> >>      Sector size (logical/physical): 512 bytes / 512 bytes
> >>      I/O size (minimum/optimal): 512 bytes / 512 bytes
> >>      Disklabel type: gpt
> >>      Disk identifier: B8D98B8E-7790-4043-8F37-D4D8CA218884
> >>      Device         Start    End Sectors  Size Type
> >>      /dev/sda1  2048 262110  260063  127M Linux filesystem
> >> mkfs.ext4 -L ssd /dev/sda1
> >> mount /dev/sda1 /ssd
> >> mount
> >>      /dev/sda1 on /ssd type ext4 (rw,relatime,data=ordered)
> >> echo hello world > /ssd/test
> >> cat /ssd/test
> >>      hello world
> >> sync && sysctl -q vm.drop_caches=3
> >> rm -f /ssd/test
> >> sync && sysctl -q vm.drop_caches=3
> >> fstrim /ssd
> >> umount /ssd
> >> strings /dev/sda1 | grep 'hello world'
> >>      hello world
> >>
> >>
> >> Le mar. 9 avr. 2019 à 08:36, Narcis Garcia <address@hidden> a
> >> écrit :
> >>
> >>> Thank you.
> >>>
> >>> * I use RAW images because of it's easier to offline mount/chroot when
> >>> some issue happens.
> >>> * I call directly qemu instead of libvirt
> >>>
> >>> I'll try to parse these parameters to a qemu-system call.
> >>>
> >>>
> >>> El 8/4/19 a les 22:03, Friedrich Oslage ha escrit:
> >>>> Yes.
> >>>>
> >>>> You'll have to use the virtio-scsi driver, to my knowledge it's the
> >>>> only
> >>>> driver that supports block discards.
> >>>>
> >>>> Quick example:
> >>>>
> >>>> # create a new empty disk
> >>>> $ qemu-img create -f qcow2 disk.qcow2 10G
> >>>>
> >>>> # check size, should only be a few kilobytes
> >>>> $ du -sh disk.qcow2
> >>>>
> >>>> # add it to your vm
> >>>> $ virtsh edit your-vm
> >>>>    <disk type='block' device='disk'>
> >>>>      <driver name='qemu' type='qcow2' discard='unmap'
> >>>> detect_zeroes='unmap'/>
> >>>>      <source dev='/tmp/disk.qcow2'/>
> >>>>      <target dev='sda' bus='scsi'/>
> >>>>      <address type='drive' controller='0' bus='0' target='0'
> unit='0'/>
> >>>>    </disk>
> >>>>    <controller type='scsi' index='0' model='virtio-scsi'>
> >>>>      <driver queues='4'/>
> >>>>    </controller>
> >>>>
> >>>> # in your vm, format and mount it
> >>>> $ mkfs.ext4 /dev/sda
> >>>> $ mount /dev/sda /mnt/discardtest
> >>>>
> >>>> # put some data on it
> >>>> $ dd if=/dev/urandom of=/mnt/discardtest/dummy.bin bs=1M count=1000
> >>>> $ sync
> >>>>
> >>>> # on host, check size...it should be about 1GB
> >>>> $ du -sh disk.qcow2
> >>>>
> >>>> # in vm, delete file and trim it
> >>>> $ rm /mnt/discardtest/dummy.bin
> >>>> $ sync
> >>>> $ fstrim /mnt/discardtest
> >>>>
> >>>> # on host, check size...should only be a few megabytes
> >>>> $ du -sh disk.qcow2
> >>>>
> >>>>
> >>>> It works with other guest OSes as well, for Windows you just use the
> >>>> Optimize-Volume cmdlets instead of fstrim.
> >>>>
> >>>> You also don't have to use qcow2 for backing. Anything that can handle
> >>>> discards will do, including sparse files.
> >>>>
> >>>>
> >>>> My advice, however, would be to use qcow2, since sparse files don't
> >>>> work
> >>>> particularly well. Executing the example above would leave you with
> >>>> about 300MB at the end instead of less than 10MB.
> >>>>
> >>>> Regards
> >>>> Friedrich
> >>>>
> >>>> On 4/5/19 4:04 PM, Narcis Garcia wrote:
> >>>>> No; I want to study the possibility of recovering allocated blocs of
> >>>>> host filesystem when guest filesystem removes its files
> >>>>> (unallocates its
> >>>>> blocks).
> >>>>> Host -> HD or SSD (independent) with sparse-mode image
> >>>>> Guest -> Virtual SSD (to signal discards)
> >>>>>
> >>>>>
> >>>>> El 5/4/19 a les 16:01, Pascal ha escrit:
> >>>>>> hello,
> >>>>>>
> >>>>>> if I understand correctly, you want to study the possibility of
> >>>>>> recovering deleted files from an SSD disk: is that right ?
> >>>>>>
> >>>>>> regards, lacsaP.
> >>>>>>
> >>>>>> Le jeu. 4 avr. 2019 à 08:24, Narcis Garcia <address@hidden
> >>>>>> <mailto:address@hidden>> a écrit :
> >>>>>>
> >>>>>>       Is there a way to specify a disk to be detected as an SSD
> >>>>>> drive?
> >>>>>>
> >>>>>>       Once reached this, I want to look for the possibility to
> >>>>>> recover
> >>>>>> host
> >>>>>>       space when a guest discards disk image blocks, and this
> >>>>>> image is
> >>>>>> RAW
> >>>>>>       format and sparse allocated file.
> >>>>>>
> >>>>>>       Thank you.
> >>>>>>
> >>>
> >
>
>