Re: our benchmark-suite

[Neil Jerram]

address@hidden (Ludovic Courtès) writes:

>> My proposal is to rebase the iteration count in 0-reference.bm to run
>> for 0.5s on some modern machine, and adjust all benchmarks to match,
>> removing those benchmarks that do not measure anything useful.
>
> Sounds good.  However, adjusting iteration counts of the benchmarks
> themselves should be done rarely, as it breaks performance tracking like
> <http://ossau.homelinux.net/~neil/bm_master_i.html>.
>
>> Finally we should perhaps enable automatic scaling of the iteration
>> count.  What do folks think about that?
>>
>> On the positive side, all of our benchmarks are very clear that they are
>> a time per number of iterations, and so this change should not affect
>> users that measure time per iteration.
>
> If the reported time is divided by the global iteration count, then
> automatic scaling of the global iteration count would be good, yes.

For http://ossau.homelinux.net/~neil I do still have all of the raw data
including iteration counts, so I could easily implement dividing by the
iteration count, and hence allow for future iteration count changes.

Is there any downside from doing that?  (I don't think so.)

Regards,
        Neil