Re: Freebsd 12 strange disk behaviour on QEMU machine

[Jakob Bohm]

Using Linux Guests on older qemu versions, I have seen some bad disk speed effects of snapshot mechanisms, but not your particular scenario:

1. When creating a snapshot and until all snapshots are deleted, all disk writes are redirected to a delta storage area that keeps track of which virtual disk blocks are different from the snapshot contents and what that different block contents is.  Read access thus gets overhead from doing the lookup and occasional redirected read.  Write access gets much more overhead from adding delta entries and redirecting the write.  This is inherent in the snapshot mechanism.

2. In snapshot mode I have seen general I/O (including network and/or basic CPU stuff) getting slowed down so hard that the outside cannot even ping the virtual machines, indicating that someone in the qemu team made the mistake of doing the disk snapshot overhead in the same execution thread as basic emulation tasks.  Later qemu developers get so focused on spitting out replies to rookies that they fail to realize the fundamental problem.

3. I have not seen slowdowns happen after snapshot removal, but cannot rule out that someone made further mistakes in I/O handling.

On 2023-09-20 15:40, Roland Giesler wrote:

Has anyone here ever has a similar occurrence?  It's quite perplexing and I'm surprised that I can't find anything about this on the web.

On 2023/09/18 16:03, Roland Giesler wrote:

We have a FreeBSD machine running an IRIS poller that behaves really strange. 

We had FreeBSD 12.3 at first, then changed to 12.4.  Same result.

We also changed from BIOS to UEFI.  Same result.

The storage is ceph RBD on NVMe SSD drives, so it's fast.

The symptoms are as follows:

After installation of the OS and the script (most perl) that IRIS uses, the service run perflectly.  I then take a snapshot.  If the machine is restarted for whatever else reason everything comes up, but the FreeBSD processes go into a "D state", meaning they are waiting for the disk.

This happens very time.  It's then not possible to fix the machine, although all seems quite normal and not config changes took place that we can detect.

If I roll back the snapshot, things are normal for a while put soon processes go into D state again.

I'm not sure that this is a qemu problem, but we have to start somewhere. 

Finally, I not have the VM running an a local volume (non ceph, standalone SSD) and there's no problem.  This is actually an installation that was done on ceph and then the disk volume was moved to the current location.

Enjoy

Jakob
-- 
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded