Re: [Qemu-discuss] Discard disk image holes for sparse shrink

[Narcis Garcia]

I see now the simple:
-device virtio-scsi-pci
makes all -disk (media=disk) to be detected by guest as SSD

Additionally, adding "discard=unmap,detect-zeroes=unmap" to disk image
specification makes host Qemu to discard sparse image holes and recover
host disk space when guest sends discard signal!

This is my successful test with Debian 9 (stable):

$ dd if=/dev/zero of=ssd.img obs=1M seek=128 count=0
$ qemu-system-... -device virtio-scsi-pci -drive
file=ssd.img,media=disk,index=0,discard=unmap,detect-zeroes=unmap

Thank you everyone.


El 9/4/19 a les 19:19, Friedrich Oslage ha escrit:
> 1. You need to set the discard option for the -drive parameter, eg
> -drive if=none,...,format=raw,discard=unmap since the default is still
> to simply ignore discards.
> 
> You may also want to set the detect-zeroes option to unmap, to discard
> all-zero blocks instead of actually writing them.
> 
> 
> 2. The disk image you created is thick-provisioned. How is qemu supposed
> to discard anything in that file? It could turn it into a sparse file or
> overwrite it with zeroes I suppose, but both options are undesirable.
> 
> To make use of the discard operation your backing storage has to support
> it. Either by beeing thin-provisioned, like qcow2 or raw sparse files,
> or by actually beeing a ssd/nvme disk.
> 
> 
> 3. Your testing method is flawed. You are using a journaling filesystem
> to write the data but to read it you access the block device directly .
> Even if the discard operation was working strings(1) would still show
> the hello-world string in the journal.
> 
> You could try something like this:
> 
> $ echo hello, world > /dev/sda
> $ strings /dev/sda
> $ blkdiscard /dev/sda
> $ strings /dev/sda
> 
> 
> Regards
> Friedrich
> 
> On 4/9/19 4:38 PM, Pascal wrote:
>> the blocks do not seem to be discarded on a raw format disk: the data
>> remains on the disk...
>>
>> on host :
>>
>> dd if=/dev/zero of=/tmp/ssd.disk bs=1M count=128
>> qemu -drive media=disk,file=linux.disk -device virtio-scsi-pci,id=scsi
>> -device scsi-hd,drive=hd -drive
>> if=none,id=hd,file=/tmp/ssd.disk,format=raw
>>
>> on Linux VM :
>>
>> fdisk /dev/sda
>>      # create one partition on gpt type...
>> fdisk -l /dev/sda
>>      Disk /dev/sda: 128 MiB, 134217728 bytes, 262144 sectors
>>      Units: sectors of 1 * 512 = 512 bytes
>>      Sector size (logical/physical): 512 bytes / 512 bytes
>>      I/O size (minimum/optimal): 512 bytes / 512 bytes
>>      Disklabel type: gpt
>>      Disk identifier: B8D98B8E-7790-4043-8F37-D4D8CA218884
>>      Device         Start    End Sectors  Size Type
>>      /dev/sda1  2048 262110  260063  127M Linux filesystem
>> mkfs.ext4 -L ssd /dev/sda1
>> mount /dev/sda1 /ssd
>> mount
>>      /dev/sda1 on /ssd type ext4 (rw,relatime,data=ordered)
>> echo hello world > /ssd/test
>> cat /ssd/test
>>      hello world
>> sync && sysctl -q vm.drop_caches=3
>> rm -f /ssd/test
>> sync && sysctl -q vm.drop_caches=3
>> fstrim /ssd
>> umount /ssd
>> strings /dev/sda1 | grep 'hello world'
>>      hello world
>>
>>
>> Le mar. 9 avr. 2019 à 08:36, Narcis Garcia <address@hidden> a
>> écrit :
>>
>>> Thank you.
>>>
>>> * I use RAW images because of it's easier to offline mount/chroot when
>>> some issue happens.
>>> * I call directly qemu instead of libvirt
>>>
>>> I'll try to parse these parameters to a qemu-system call.
>>>
>>>
>>> El 8/4/19 a les 22:03, Friedrich Oslage ha escrit:
>>>> Yes.
>>>>
>>>> You'll have to use the virtio-scsi driver, to my knowledge it's the
>>>> only
>>>> driver that supports block discards.
>>>>
>>>> Quick example:
>>>>
>>>> # create a new empty disk
>>>> $ qemu-img create -f qcow2 disk.qcow2 10G
>>>>
>>>> # check size, should only be a few kilobytes
>>>> $ du -sh disk.qcow2
>>>>
>>>> # add it to your vm
>>>> $ virtsh edit your-vm
>>>>    <disk type='block' device='disk'>
>>>>      <driver name='qemu' type='qcow2' discard='unmap'
>>>> detect_zeroes='unmap'/>
>>>>      <source dev='/tmp/disk.qcow2'/>
>>>>      <target dev='sda' bus='scsi'/>
>>>>      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>>    </disk>
>>>>    <controller type='scsi' index='0' model='virtio-scsi'>
>>>>      <driver queues='4'/>
>>>>    </controller>
>>>>
>>>> # in your vm, format and mount it
>>>> $ mkfs.ext4 /dev/sda
>>>> $ mount /dev/sda /mnt/discardtest
>>>>
>>>> # put some data on it
>>>> $ dd if=/dev/urandom of=/mnt/discardtest/dummy.bin bs=1M count=1000
>>>> $ sync
>>>>
>>>> # on host, check size...it should be about 1GB
>>>> $ du -sh disk.qcow2
>>>>
>>>> # in vm, delete file and trim it
>>>> $ rm /mnt/discardtest/dummy.bin
>>>> $ sync
>>>> $ fstrim /mnt/discardtest
>>>>
>>>> # on host, check size...should only be a few megabytes
>>>> $ du -sh disk.qcow2
>>>>
>>>>
>>>> It works with other guest OSes as well, for Windows you just use the
>>>> Optimize-Volume cmdlets instead of fstrim.
>>>>
>>>> You also don't have to use qcow2 for backing. Anything that can handle
>>>> discards will do, including sparse files.
>>>>
>>>>
>>>> My advice, however, would be to use qcow2, since sparse files don't
>>>> work
>>>> particularly well. Executing the example above would leave you with
>>>> about 300MB at the end instead of less than 10MB.
>>>>
>>>> Regards
>>>> Friedrich
>>>>
>>>> On 4/5/19 4:04 PM, Narcis Garcia wrote:
>>>>> No; I want to study the possibility of recovering allocated blocs of
>>>>> host filesystem when guest filesystem removes its files
>>>>> (unallocates its
>>>>> blocks).
>>>>> Host -> HD or SSD (independent) with sparse-mode image
>>>>> Guest -> Virtual SSD (to signal discards)
>>>>>
>>>>>
>>>>> El 5/4/19 a les 16:01, Pascal ha escrit:
>>>>>> hello,
>>>>>>
>>>>>> if I understand correctly, you want to study the possibility of
>>>>>> recovering deleted files from an SSD disk: is that right ?
>>>>>>
>>>>>> regards, lacsaP.
>>>>>>
>>>>>> Le jeu. 4 avr. 2019 à 08:24, Narcis Garcia <address@hidden
>>>>>> <mailto:address@hidden>> a écrit :
>>>>>>
>>>>>>       Is there a way to specify a disk to be detected as an SSD
>>>>>> drive?
>>>>>>
>>>>>>       Once reached this, I want to look for the possibility to
>>>>>> recover
>>>>>> host
>>>>>>       space when a guest discards disk image blocks, and this
>>>>>> image is
>>>>>> RAW
>>>>>>       format and sparse allocated file.
>>>>>>
>>>>>>       Thank you.
>>>>>>
>>>
>