Re: [PULL 15/18] qapi: introduce x-query-ramblock QMP command

[Claudio Fontana]

On 6/30/22 12:20, Daniel P. Berrangé wrote:
> On Thu, Jun 30, 2022 at 12:14:36PM +0200, Claudio Fontana wrote:
>> On 6/9/22 14:52, Dr. David Alan Gilbert wrote:
>>> * Daniel P. Berrangé (berrange@redhat.com) wrote:
>>>> On Thu, Jun 09, 2022 at 12:07:31PM +0200, Claudio Fontana wrote:
>>>>> Hello all,
>>>>>
>>>>> it would be really good to be able to rely on this command or something 
>>>>> similar,
>>>>> to be able to know the approximate size of a migration before starting it.
>>>>>
>>>>> in QEMU ram_bytes_total() returns what I would like to have,
>>>>> but there is currently no QMP way to get it without starting a migration,
>>>>> which when trying to optimize it/size it is just about too late.
>>>>
>>>> Aside from the main VM RAM, what other RAM blocks are likely to have
>>>> a size large enough to be of consequence to the live migration
>>>> data copy, and whose size is not already known to the mgmt app from
>>>> the guest config choices it made ? VGA RAM could be a few 100MB I
>>>> guess, but the mgmt app knows about that. I've always assumed everything
>>>> else is just noise in comparison to the main RAM region.
>>>>
>>>> Still I wonder how useful this is as its just a static figure, and the
>>>> problems with migration transfer are the bulking up of data when the
>>>> VM is repeatedly dirtying stuff at a high rate.
>>>>
>>>>> Do you think x-query-ramblock could be promoted to non-experimental?
>>>>
>>>> It would have to be re-written, as this current impl is just emitting
>>>> a huge printf formatted string. To be considered supportable, the data
>>>> would have to be formally modelled in QAPI instead.
>>>>
>>>> IOW, it would be a case of introducing a new command that emits formal
>>>> data, convertintg 'info ramblock' to use that, and then deprecating this 
>>>> x-query-ramblock.
>>>>
>>>>> Should another one be made available instead, like :
>>>>> query-ram-bytes-total ?
>>>>
>>>> That would be simpler if you're just wanting it to give a single
>>>> figure.
>>>
>>> Is this what qmp_query_memory_size_summary does?
>>
>> No, I am not looking at something returning the machine->ram_size,
>> but rather how many bytes are actually used in each RAMBlock, in order to 
>> estimate the transfer size of a guest to disk.
>>
>> This would be the return value of something like 
>> migration/ram.c::ram_bytes_total().
>>
>> The main guest RAM total size is in most cases an overestimation of the 
>> actual bytes required to be transferred.
>>
>> If there was such a feature that just returns ram_bytes_total via QMP,
>> by knowing the size in bytes before the transfer, we can prealloc the space 
>> on disk, which would improve the performance of this series:
>>
>> https://patchew.org/Libvirt/20220607091936.7948-1-cfontana@suse.de/
>>
>> The interleaved format I posted there works just fine to migrate a suspended 
>> VM to disk (virsh save) from multifd channels to a single file,
>> but still incurs in a 4-5% performance penalty compared with the multiple 
>> files approach,
>> that is apparently due to multiple threads competing on acquiring locks to 
>> adjust the file size (on XFS).
>>
>> Doing a fallocate() would likely remove this performance decrease compared 
>> with multifd to multiple files,
>> but requires knowing beforehand the approximate size of the transfer, and as 
>> mentioned mnachine->ram_size is just overkill in practice and risks erroring 
>> out if not enough space is available.
>>
>> Feedback on the interleaved format I posted there is welcome as well,
> 
> I still believe that libvirt is the wrong place to be implementing any
> of this logic. It all belongs in QEMU, because QEMU is the place which
> holds all the information needed to do an optimal job, and libvirt does
> not, as this request for extra QMP features shows.
> 
> With regards,
> Daniel

Hi Daniel,

I know your position about the implementation in libvirt vs a potential 
(non-existing for now) QEMU implementation.

The implementation in QEMU seems to me to require more investment due to the 
need of a new migration target protocol to be defined carefully (possibly 
"disk://")
and the need to alter and test all migrated devices participating in the 
creating the migration stream.

I don't think this request for an QMP feature shows anything really.

Knowing the _actual_ size of a migration stream before deciding to migrate is I 
think a pretty useful feature in itself I would think,
including for libvirt and higher level components in the stack. The lack of the 
feature just shows, well, the lack of this feature.

Regarding my prototype I pointed at that happens to use libvirt, I wonder if 
you or anyone have any feedback on the actual format of the VM saved in 
parallel to disk,
regardless of which component writes it, libvirt or QEMU, and I wonder also if 
there is any feedback on the O_DIRECT -friendly I/O API, which are both things 
we can make progress on also regardless of the libvirt vs QEMU implementation 
of the parallel migration to disk.

Thanks,

Claudio