Re: [GSoC/Outreachy QEMU project proposal] Measure and Analyze QEMU Performance

[Stefan Hajnoczi]

On Sun, Jan 26, 2020 at 05:50:24PM +0100, Aleksandar Markovic wrote:
> On Wed, Jan 22, 2020 at 12:28 PM Stefan Hajnoczi <address@hidden> wrote:
> >
> > On Tue, Jan 21, 2020 at 03:07:53PM +0100, Aleksandar Markovic wrote:
> > > On Mon, Jan 20, 2020 at 3:51 PM Stefan Hajnoczi <address@hidden> wrote:
> > > >
> > > > On Sat, Jan 18, 2020 at 03:08:37PM +0100, Aleksandar Markovic wrote:
> > > > > 3) The community will be given all devised performance measurement 
> > > > > methods in the form of easily reproducible step-by-step setup and 
> > > > > execution procedures.
> > > >
> > > > Tracking performance is a good idea and something that has not been done
> > > > upstream yet.
> > >
> > > Thanks for the interest, Stefan!
> > >
> > > >  A few questions:
> > > >
> > > >  * Will benchmarks be run automatically (e.g. nightly or weekly) on
> > > >    someone's hardware or does every TCG architecture maintainer need to
> > > >    run them manually for themselves?
> > >
> > > If the community wants it, definitely yes. Once the methodology is
> > > developed, it should be straightforward to setup nightly and/or weekly
> > > benchmarks - that could definitely include sending mails with reports
> > > to the entire list or just individuals or subgroups. The recipient
> > > choice is just a matter or having decent criteria about
> > > appropriateness of information within the message (e.g. not to flood
> > > the list with the data most people are not really interested).
> > >
> > > For linux-user tests, they are typically very quick, and nightly tests
> > > are quite feasible to run. On someone hardware, of course, and
> > > consistently always on the same hardware, if possible. If it makes
> > > sense, one could setup multiple test beds with a variety of hardware
> > > setups.
> > >
> > > For system mode tests, I knoe they are much more difficult to
> > > automate, and, on top of that, there could be greater risk of
> > > hangs/crashes Also, considering the number of machines we support,
> > > those tests could consume much more time - perhaps even one day would
> > > not be sufficient, if we have many machines and boot/shutdown
> > > variants. For these reason, perhaps weekly executions would be more
> > > appropriate for them, and, in general, given greater complexity, the
> > > expectation from system-mode performance tests should be better kept
> > > quite low for now.
> > >
> > > >  * Where will the benchmark result history be stored?
> > > >
> > >
> > > If emailing is set up, the results could be reconstructed from emails.
> > > But, yes, it would be better if the result history is kept somewhere
> > > on an internet-connected file server
> >
> > Thanks.  I don't want to overcomplicate this project.  The main thing is
> > to identify the stakeholders (TCG target maintainers?) and make sure
> > they are happy.
> >
> 
> Yes, Stefan, TCG target maintainers would be the main stakeholders. To
> some extent, various Machine maintainers would also be stakeholders,
> but they will most likely come back to TCG target maintainers looking
> for solution. In a literal sense, a number of maintainers were
> initially going to be very unhappy seeing the results (for example,
> seeing that the machine or entire target performs poorly compared to
> similar machines/targets), but after a while they should and will
> become happy realizing the problem was identified, and the culprit is
> at least approximately determined.
> 
> I intentionally wanted to keep the project description simple in order
> to be realistic and not develop high expectation among any of us. And
> if the student proved to be capable, it will be very easy to add some
> more useful tasks for him in this area, to be included in his/hers
> GSoC/Outreachy activities.
> 
> He had just today one case of performance degradation identified manually:
> 
> https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg06326.html
> 
> This project aims to do these kind of things easier, and possibly in
> an automated way. Howard did this by manual measurements for one
> particular setup, but this project will cover much much more.
> 
> Thanks, Stefan, again for your interest - and everything else!

Please go ahead and add this project idea to the wiki:
https://wiki.qemu.org/Google_Summer_of_Code_2020#How_to_add_a_project_idea

Stefan

signature.asc
Description: PGP signature