Re: [Qemu-discuss] Discard disk image holes for sparse shrink

[Pascal]

I would tend to say that "discard" is not related to "ssd" while "rotational"
is...

Le mer. 10 avr. 2019 à 14:06, Pascal <address@hidden> a écrit :

> lsblk -ndo ROTA /dev/sda
>
> sorry for the bad copy.
>
> Le mer. 10 avr. 2019 à 12:52, Narcis Garcia <address@hidden> a
> écrit :
>
>> Can be this rotational detection be affecting in this case?
>>
>> https://lists.openvz.org/pipermail/users/2019-April/007564.html
>> (Devuan 1.0 VM with distro kernel succeeds with lsblk and fstrim)
>>
>> How can I make guest OS to detect device as no rotational or true SSD?
>>
>>
>> El 10/4/19 a les 12:47, Narcis Garcia ha escrit:
>> > Pascal, I don't understand syntax you use for lsblk.
>> > I see this with:
>> > $ cat /sys/block/sda/queue/rotational
>> >
>> >
>> > El 10/4/19 a les 12:38, Pascal ha escrit:
>> >> it's ok with  discard=unmap  option (even if the disc is
>> thick-provisioned)
>> >> : the block used by the  test  file appears (VM side) or is reset (host
>> >> side) at zero.
>> >>
>> >> notice that  lsblk  "thinks" that the disc is a rotational disk (eg.
>> not
>> >> really SSD) :
>> >>
>> >> lsblk -ndo /dev/sda
>> >> 1
>> >>
>> >> thanks for explanations !
>> >>
>> >> Le mar. 9 avr. 2019 à 20:38, Narcis Garcia <address@hidden> a
>> écrit :
>> >>
>> >>> I see now the simple:
>> >>> -device virtio-scsi-pci
>> >>> makes all -disk (media=disk) to be detected by guest as SSD
>> >>>
>> >>> Additionally, adding "discard=unmap,detect-zeroes=unmap" to disk image
>> >>> specification makes host Qemu to discard sparse image holes and
>> recover
>> >>> host disk space when guest sends discard signal!
>> >>>
>> >>> This is my successful test with Debian 9 (stable):
>> >>>
>> >>> $ dd if=/dev/zero of=ssd.img obs=1M seek=128 count=0
>> >>> $ qemu-system-... -device virtio-scsi-pci -drive
>> >>> file=ssd.img,media=disk,index=0,discard=unmap,detect-zeroes=unmap
>> >>>
>> >>> Thank you everyone.
>> >>>
>> >>>
>> >>> El 9/4/19 a les 19:19, Friedrich Oslage ha escrit:
>> >>>> 1. You need to set the discard option for the -drive parameter, eg
>> >>>> -drive if=none,...,format=raw,discard=unmap since the default is
>> still
>> >>>> to simply ignore discards.
>> >>>>
>> >>>> You may also want to set the detect-zeroes option to unmap, to
>> discard
>> >>>> all-zero blocks instead of actually writing them.
>> >>>>
>> >>>>
>> >>>> 2. The disk image you created is thick-provisioned. How is qemu
>> supposed
>> >>>> to discard anything in that file? It could turn it into a sparse
>> file or
>> >>>> overwrite it with zeroes I suppose, but both options are undesirable.
>> >>>>
>> >>>> To make use of the discard operation your backing storage has to
>> support
>> >>>> it. Either by beeing thin-provisioned, like qcow2 or raw sparse
>> files,
>> >>>> or by actually beeing a ssd/nvme disk.
>> >>>>
>> >>>>
>> >>>> 3. Your testing method is flawed. You are using a journaling
>> filesystem
>> >>>> to write the data but to read it you access the block device
>> directly .
>> >>>> Even if the discard operation was working strings(1) would still show
>> >>>> the hello-world string in the journal.
>> >>>>
>> >>>> You could try something like this:
>> >>>>
>> >>>> $ echo hello, world > /dev/sda
>> >>>> $ strings /dev/sda
>> >>>> $ blkdiscard /dev/sda
>> >>>> $ strings /dev/sda
>> >>>>
>> >>>>
>> >>>> Regards
>> >>>> Friedrich
>> >>>>
>> >>>> On 4/9/19 4:38 PM, Pascal wrote:
>> >>>>> the blocks do not seem to be discarded on a raw format disk: the
>> data
>> >>>>> remains on the disk...
>> >>>>>
>> >>>>> on host :
>> >>>>>
>> >>>>> dd if=/dev/zero of=/tmp/ssd.disk bs=1M count=128
>> >>>>> qemu -drive media=disk,file=linux.disk -device
>> virtio-scsi-pci,id=scsi
>> >>>>> -device scsi-hd,drive=hd -drive
>> >>>>> if=none,id=hd,file=/tmp/ssd.disk,format=raw
>> >>>>>
>> >>>>> on Linux VM :
>> >>>>>
>> >>>>> fdisk /dev/sda
>> >>>>>      # create one partition on gpt type...
>> >>>>> fdisk -l /dev/sda
>> >>>>>      Disk /dev/sda: 128 MiB, 134217728 bytes, 262144 sectors
>> >>>>>      Units: sectors of 1 * 512 = 512 bytes
>> >>>>>      Sector size (logical/physical): 512 bytes / 512 bytes
>> >>>>>      I/O size (minimum/optimal): 512 bytes / 512 bytes
>> >>>>>      Disklabel type: gpt
>> >>>>>      Disk identifier: B8D98B8E-7790-4043-8F37-D4D8CA218884
>> >>>>>      Device         Start    End Sectors  Size Type
>> >>>>>      /dev/sda1  2048 262110  260063  127M Linux filesystem
>> >>>>> mkfs.ext4 -L ssd /dev/sda1
>> >>>>> mount /dev/sda1 /ssd
>> >>>>> mount
>> >>>>>      /dev/sda1 on /ssd type ext4 (rw,relatime,data=ordered)
>> >>>>> echo hello world > /ssd/test
>> >>>>> cat /ssd/test
>> >>>>>      hello world
>> >>>>> sync && sysctl -q vm.drop_caches=3
>> >>>>> rm -f /ssd/test
>> >>>>> sync && sysctl -q vm.drop_caches=3
>> >>>>> fstrim /ssd
>> >>>>> umount /ssd
>> >>>>> strings /dev/sda1 | grep 'hello world'
>> >>>>>      hello world
>> >>>>>
>> >>>>>
>> >>>>> Le mar. 9 avr. 2019 à 08:36, Narcis Garcia <address@hidden>
>> a
>> >>>>> écrit :
>> >>>>>
>> >>>>>> Thank you.
>> >>>>>>
>> >>>>>> * I use RAW images because of it's easier to offline mount/chroot
>> when
>> >>>>>> some issue happens.
>> >>>>>> * I call directly qemu instead of libvirt
>> >>>>>>
>> >>>>>> I'll try to parse these parameters to a qemu-system call.
>> >>>>>>
>> >>>>>>
>> >>>>>> El 8/4/19 a les 22:03, Friedrich Oslage ha escrit:
>> >>>>>>> Yes.
>> >>>>>>>
>> >>>>>>> You'll have to use the virtio-scsi driver, to my knowledge it's
>> the
>> >>>>>>> only
>> >>>>>>> driver that supports block discards.
>> >>>>>>>
>> >>>>>>> Quick example:
>> >>>>>>>
>> >>>>>>> # create a new empty disk
>> >>>>>>> $ qemu-img create -f qcow2 disk.qcow2 10G
>> >>>>>>>
>> >>>>>>> # check size, should only be a few kilobytes
>> >>>>>>> $ du -sh disk.qcow2
>> >>>>>>>
>> >>>>>>> # add it to your vm
>> >>>>>>> $ virtsh edit your-vm
>> >>>>>>>    <disk type='block' device='disk'>
>> >>>>>>>      <driver name='qemu' type='qcow2' discard='unmap'
>> >>>>>>> detect_zeroes='unmap'/>
>> >>>>>>>      <source dev='/tmp/disk.qcow2'/>
>> >>>>>>>      <target dev='sda' bus='scsi'/>
>> >>>>>>>      <address type='drive' controller='0' bus='0' target='0'
>> >>> unit='0'/>
>> >>>>>>>    </disk>
>> >>>>>>>    <controller type='scsi' index='0' model='virtio-scsi'>
>> >>>>>>>      <driver queues='4'/>
>> >>>>>>>    </controller>
>> >>>>>>>
>> >>>>>>> # in your vm, format and mount it
>> >>>>>>> $ mkfs.ext4 /dev/sda
>> >>>>>>> $ mount /dev/sda /mnt/discardtest
>> >>>>>>>
>> >>>>>>> # put some data on it
>> >>>>>>> $ dd if=/dev/urandom of=/mnt/discardtest/dummy.bin bs=1M
>> count=1000
>> >>>>>>> $ sync
>> >>>>>>>
>> >>>>>>> # on host, check size...it should be about 1GB
>> >>>>>>> $ du -sh disk.qcow2
>> >>>>>>>
>> >>>>>>> # in vm, delete file and trim it
>> >>>>>>> $ rm /mnt/discardtest/dummy.bin
>> >>>>>>> $ sync
>> >>>>>>> $ fstrim /mnt/discardtest
>> >>>>>>>
>> >>>>>>> # on host, check size...should only be a few megabytes
>> >>>>>>> $ du -sh disk.qcow2
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> It works with other guest OSes as well, for Windows you just use
>> the
>> >>>>>>> Optimize-Volume cmdlets instead of fstrim.
>> >>>>>>>
>> >>>>>>> You also don't have to use qcow2 for backing. Anything that can
>> handle
>> >>>>>>> discards will do, including sparse files.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> My advice, however, would be to use qcow2, since sparse files
>> don't
>> >>>>>>> work
>> >>>>>>> particularly well. Executing the example above would leave you
>> with
>> >>>>>>> about 300MB at the end instead of less than 10MB.
>> >>>>>>>
>> >>>>>>> Regards
>> >>>>>>> Friedrich
>> >>>>>>>
>> >>>>>>> On 4/5/19 4:04 PM, Narcis Garcia wrote:
>> >>>>>>>> No; I want to study the possibility of recovering allocated
>> blocs of
>> >>>>>>>> host filesystem when guest filesystem removes its files
>> >>>>>>>> (unallocates its
>> >>>>>>>> blocks).
>> >>>>>>>> Host -> HD or SSD (independent) with sparse-mode image
>> >>>>>>>> Guest -> Virtual SSD (to signal discards)
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> El 5/4/19 a les 16:01, Pascal ha escrit:
>> >>>>>>>>> hello,
>> >>>>>>>>>
>> >>>>>>>>> if I understand correctly, you want to study the possibility of
>> >>>>>>>>> recovering deleted files from an SSD disk: is that right ?
>> >>>>>>>>>
>> >>>>>>>>> regards, lacsaP.
>> >>>>>>>>>
>> >>>>>>>>> Le jeu. 4 avr. 2019 à 08:24, Narcis Garcia <
>> address@hidden
>> >>>>>>>>> <mailto:address@hidden>> a écrit :
>> >>>>>>>>>
>> >>>>>>>>>       Is there a way to specify a disk to be detected as an SSD
>> >>>>>>>>> drive?
>> >>>>>>>>>
>> >>>>>>>>>       Once reached this, I want to look for the possibility to
>> >>>>>>>>> recover
>> >>>>>>>>> host
>> >>>>>>>>>       space when a guest discards disk image blocks, and this
>> >>>>>>>>> image is
>> >>>>>>>>> RAW
>> >>>>>>>>>       format and sparse allocated file.
>> >>>>>>>>>
>> >>>>>>>>>       Thank you.
>> >>>>>>>>>
>> >>>>>>
>> >>>>
>> >>>
>> >>>
>> >
>>
>>