Re: [Qemu-discuss] Discard disk image holes for sparse shrink

[Narcis Garcia]

Can be this rotational detection be affecting in this case?

https://lists.openvz.org/pipermail/users/2019-April/007564.html
(Devuan 1.0 VM with distro kernel succeeds with lsblk and fstrim)

How can I make guest OS to detect device as no rotational or true SSD?


El 10/4/19 a les 12:47, Narcis Garcia ha escrit:
> Pascal, I don't understand syntax you use for lsblk.
> I see this with:
> $ cat /sys/block/sda/queue/rotational
> 
> 
> El 10/4/19 a les 12:38, Pascal ha escrit:
>> it's ok with  discard=unmap  option (even if the disc is thick-provisioned)
>> : the block used by the  test  file appears (VM side) or is reset (host
>> side) at zero.
>>
>> notice that  lsblk  "thinks" that the disc is a rotational disk (eg. not
>> really SSD) :
>>
>> lsblk -ndo /dev/sda
>> 1
>>
>> thanks for explanations !
>>
>> Le mar. 9 avr. 2019 à 20:38, Narcis Garcia <address@hidden> a écrit :
>>
>>> I see now the simple:
>>> -device virtio-scsi-pci
>>> makes all -disk (media=disk) to be detected by guest as SSD
>>>
>>> Additionally, adding "discard=unmap,detect-zeroes=unmap" to disk image
>>> specification makes host Qemu to discard sparse image holes and recover
>>> host disk space when guest sends discard signal!
>>>
>>> This is my successful test with Debian 9 (stable):
>>>
>>> $ dd if=/dev/zero of=ssd.img obs=1M seek=128 count=0
>>> $ qemu-system-... -device virtio-scsi-pci -drive
>>> file=ssd.img,media=disk,index=0,discard=unmap,detect-zeroes=unmap
>>>
>>> Thank you everyone.
>>>
>>>
>>> El 9/4/19 a les 19:19, Friedrich Oslage ha escrit:
>>>> 1. You need to set the discard option for the -drive parameter, eg
>>>> -drive if=none,...,format=raw,discard=unmap since the default is still
>>>> to simply ignore discards.
>>>>
>>>> You may also want to set the detect-zeroes option to unmap, to discard
>>>> all-zero blocks instead of actually writing them.
>>>>
>>>>
>>>> 2. The disk image you created is thick-provisioned. How is qemu supposed
>>>> to discard anything in that file? It could turn it into a sparse file or
>>>> overwrite it with zeroes I suppose, but both options are undesirable.
>>>>
>>>> To make use of the discard operation your backing storage has to support
>>>> it. Either by beeing thin-provisioned, like qcow2 or raw sparse files,
>>>> or by actually beeing a ssd/nvme disk.
>>>>
>>>>
>>>> 3. Your testing method is flawed. You are using a journaling filesystem
>>>> to write the data but to read it you access the block device directly .
>>>> Even if the discard operation was working strings(1) would still show
>>>> the hello-world string in the journal.
>>>>
>>>> You could try something like this:
>>>>
>>>> $ echo hello, world > /dev/sda
>>>> $ strings /dev/sda
>>>> $ blkdiscard /dev/sda
>>>> $ strings /dev/sda
>>>>
>>>>
>>>> Regards
>>>> Friedrich
>>>>
>>>> On 4/9/19 4:38 PM, Pascal wrote:
>>>>> the blocks do not seem to be discarded on a raw format disk: the data
>>>>> remains on the disk...
>>>>>
>>>>> on host :
>>>>>
>>>>> dd if=/dev/zero of=/tmp/ssd.disk bs=1M count=128
>>>>> qemu -drive media=disk,file=linux.disk -device virtio-scsi-pci,id=scsi
>>>>> -device scsi-hd,drive=hd -drive
>>>>> if=none,id=hd,file=/tmp/ssd.disk,format=raw
>>>>>
>>>>> on Linux VM :
>>>>>
>>>>> fdisk /dev/sda
>>>>>      # create one partition on gpt type...
>>>>> fdisk -l /dev/sda
>>>>>      Disk /dev/sda: 128 MiB, 134217728 bytes, 262144 sectors
>>>>>      Units: sectors of 1 * 512 = 512 bytes
>>>>>      Sector size (logical/physical): 512 bytes / 512 bytes
>>>>>      I/O size (minimum/optimal): 512 bytes / 512 bytes
>>>>>      Disklabel type: gpt
>>>>>      Disk identifier: B8D98B8E-7790-4043-8F37-D4D8CA218884
>>>>>      Device         Start    End Sectors  Size Type
>>>>>      /dev/sda1  2048 262110  260063  127M Linux filesystem
>>>>> mkfs.ext4 -L ssd /dev/sda1
>>>>> mount /dev/sda1 /ssd
>>>>> mount
>>>>>      /dev/sda1 on /ssd type ext4 (rw,relatime,data=ordered)
>>>>> echo hello world > /ssd/test
>>>>> cat /ssd/test
>>>>>      hello world
>>>>> sync && sysctl -q vm.drop_caches=3
>>>>> rm -f /ssd/test
>>>>> sync && sysctl -q vm.drop_caches=3
>>>>> fstrim /ssd
>>>>> umount /ssd
>>>>> strings /dev/sda1 | grep 'hello world'
>>>>>      hello world
>>>>>
>>>>>
>>>>> Le mar. 9 avr. 2019 à 08:36, Narcis Garcia <address@hidden> a
>>>>> écrit :
>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>> * I use RAW images because of it's easier to offline mount/chroot when
>>>>>> some issue happens.
>>>>>> * I call directly qemu instead of libvirt
>>>>>>
>>>>>> I'll try to parse these parameters to a qemu-system call.
>>>>>>
>>>>>>
>>>>>> El 8/4/19 a les 22:03, Friedrich Oslage ha escrit:
>>>>>>> Yes.
>>>>>>>
>>>>>>> You'll have to use the virtio-scsi driver, to my knowledge it's the
>>>>>>> only
>>>>>>> driver that supports block discards.
>>>>>>>
>>>>>>> Quick example:
>>>>>>>
>>>>>>> # create a new empty disk
>>>>>>> $ qemu-img create -f qcow2 disk.qcow2 10G
>>>>>>>
>>>>>>> # check size, should only be a few kilobytes
>>>>>>> $ du -sh disk.qcow2
>>>>>>>
>>>>>>> # add it to your vm
>>>>>>> $ virtsh edit your-vm
>>>>>>>    <disk type='block' device='disk'>
>>>>>>>      <driver name='qemu' type='qcow2' discard='unmap'
>>>>>>> detect_zeroes='unmap'/>
>>>>>>>      <source dev='/tmp/disk.qcow2'/>
>>>>>>>      <target dev='sda' bus='scsi'/>
>>>>>>>      <address type='drive' controller='0' bus='0' target='0'
>>> unit='0'/>
>>>>>>>    </disk>
>>>>>>>    <controller type='scsi' index='0' model='virtio-scsi'>
>>>>>>>      <driver queues='4'/>
>>>>>>>    </controller>
>>>>>>>
>>>>>>> # in your vm, format and mount it
>>>>>>> $ mkfs.ext4 /dev/sda
>>>>>>> $ mount /dev/sda /mnt/discardtest
>>>>>>>
>>>>>>> # put some data on it
>>>>>>> $ dd if=/dev/urandom of=/mnt/discardtest/dummy.bin bs=1M count=1000
>>>>>>> $ sync
>>>>>>>
>>>>>>> # on host, check size...it should be about 1GB
>>>>>>> $ du -sh disk.qcow2
>>>>>>>
>>>>>>> # in vm, delete file and trim it
>>>>>>> $ rm /mnt/discardtest/dummy.bin
>>>>>>> $ sync
>>>>>>> $ fstrim /mnt/discardtest
>>>>>>>
>>>>>>> # on host, check size...should only be a few megabytes
>>>>>>> $ du -sh disk.qcow2
>>>>>>>
>>>>>>>
>>>>>>> It works with other guest OSes as well, for Windows you just use the
>>>>>>> Optimize-Volume cmdlets instead of fstrim.
>>>>>>>
>>>>>>> You also don't have to use qcow2 for backing. Anything that can handle
>>>>>>> discards will do, including sparse files.
>>>>>>>
>>>>>>>
>>>>>>> My advice, however, would be to use qcow2, since sparse files don't
>>>>>>> work
>>>>>>> particularly well. Executing the example above would leave you with
>>>>>>> about 300MB at the end instead of less than 10MB.
>>>>>>>
>>>>>>> Regards
>>>>>>> Friedrich
>>>>>>>
>>>>>>> On 4/5/19 4:04 PM, Narcis Garcia wrote:
>>>>>>>> No; I want to study the possibility of recovering allocated blocs of
>>>>>>>> host filesystem when guest filesystem removes its files
>>>>>>>> (unallocates its
>>>>>>>> blocks).
>>>>>>>> Host -> HD or SSD (independent) with sparse-mode image
>>>>>>>> Guest -> Virtual SSD (to signal discards)
>>>>>>>>
>>>>>>>>
>>>>>>>> El 5/4/19 a les 16:01, Pascal ha escrit:
>>>>>>>>> hello,
>>>>>>>>>
>>>>>>>>> if I understand correctly, you want to study the possibility of
>>>>>>>>> recovering deleted files from an SSD disk: is that right ?
>>>>>>>>>
>>>>>>>>> regards, lacsaP.
>>>>>>>>>
>>>>>>>>> Le jeu. 4 avr. 2019 à 08:24, Narcis Garcia <address@hidden
>>>>>>>>> <mailto:address@hidden>> a écrit :
>>>>>>>>>
>>>>>>>>>       Is there a way to specify a disk to be detected as an SSD
>>>>>>>>> drive?
>>>>>>>>>
>>>>>>>>>       Once reached this, I want to look for the possibility to
>>>>>>>>> recover
>>>>>>>>> host
>>>>>>>>>       space when a guest discards disk image blocks, and this
>>>>>>>>> image is
>>>>>>>>> RAW
>>>>>>>>>       format and sparse allocated file.
>>>>>>>>>
>>>>>>>>>       Thank you.
>>>>>>>>>
>>>>>>
>>>>
>>>
>>>
>