Re: [PATCH for-5.0 v2 0/3] benchmark util

[Vladimir Sementsov-Ogievskiy]

27.11.2019 22:58, Aleksandar Markovic wrote:
> On Tue, Nov 26, 2019 at 4:49 PM Vladimir Sementsov-Ogievskiy
> <address@hidden> wrote:
>>
>> Hi all!
>>
>> Here is simple benchmarking utility, to generate performance
>> comparison tables, like the following:
>>
>> ----------  -------------  -------------  -------------
>>              backup-1       backup-2       mirror
>> ssd -> ssd  0.43 +- 0.00   4.48 +- 0.06   4.38 +- 0.02
>> ssd -> hdd  10.60 +- 0.08  10.69 +- 0.18  10.57 +- 0.05
>> ssd -> nbd  33.81 +- 0.37  10.67 +- 0.17  10.07 +- 0.07
>> ----------  -------------  -------------  -------------
>>
>> This is a v2, as v1 was inside
>>   "[RFC 00/24] backup performance: block_status + async"
>>
>> I'll use this benchmark in other series, hope someone
>> will like it.
>>
> 
> Vladimir,
> 
> I really like this idea, even though I am interested in benchmarks of
> different nature. Certainly a beautiful and handy tool for doing
> detection of performance regressions (and also confirmation of
> performance optimizations).

Hi Aleksandar!

Glad you are interested!

> 
> Did you run the tool on the previous QEMU versions, to detect change
> in numbers between QEMU versions? Do you have the results of the same
> benchmark for QEMU 2.12, 3.0, 4.0,... ?

No, I don't really interested in such comparisons, my goal is to measure
backup performance in context of my backup/block-copy series.

But such statistics may be simply generated, just make several git work-trees
per release you interested in, fill test_envs variable (together with other
variables) in python/bench-example.py (patch 03), and run it.

> 
> What are units used in the table? Seconds, minutes? Hopefully, not hours?

Seconds of course, I'm not so patient :)
Also, if you are interested, 1000M raw images filled with ones used to make
this benchmark table. However, it's in cover-letter only as example.

backup-1 and mirror are upstream. backup-2 is our downstream. nbd gain of
backup-2 is related to async requests. ssd gain for backup-1 is related to
copy offloading (wow, it works!).

> 
> Yours,
> Aleksandar
> 
>> Vladimir Sementsov-Ogievskiy (3):
>>    python: add simplebench.py
>>    python: add qemu/bench_block_job.py
>>    python: add example usage of simplebench
>>
>>   python/bench-example.py        |  80 +++++++++++++++++++++
>>   python/qemu/bench_block_job.py | 115 +++++++++++++++++++++++++++++
>>   python/simplebench.py          | 128 +++++++++++++++++++++++++++++++++
>>   3 files changed, 323 insertions(+)
>>   create mode 100644 python/bench-example.py
>>   create mode 100755 python/qemu/bench_block_job.py
>>   create mode 100644 python/simplebench.py
>>
>> --
>> 2.18.0
>>
>>


-- 
Best regards,
Vladimir