Re: An Emacs benchmarking suite

[Stefan Monnier]

>> So, something like time to create a big list (for CPU), and time to read
>> a defined file (for IO). Then you could say "this file should parse in
>> 1x IO base-line + 10x CPU base-line.
>
> You assume a linear scalability, but that is not necessarily so.
> The ratio between performance indices of different codes could vary
> depending on the build option and the underlying OS.

Another issue is that performance measuring is notoriously difficult
(even on an otherwise idle machine, let alone on some server that has
other tasks running at the same time).  So you might be able to catch
the "10x slower" case easily with a fairly high confidence that the
problem is indeed that the code got slower, but if you want to catch the
"20% slowdown" with any kind of confidence (without being drowned in
false positives), you'll need either a very tight control on the test
runs, or a good statistical analysis.


        Stefan