Re: our benchmark-suite

[Ludovic Courtès]

Hi Andy!

Andy Wingo <address@hidden> skribis:

> For what it's worth, the current overhead of the benchmark appears to be
> about 35 microseconds per iteration, on my laptop.  If we inline the
> iteration into the benchmark itself, rather than calling a thunk
> repeatedly, we can bring that down to around 13 microseconds.

There are a few benchmarks doing it already.  See, for instance,
‘repeat’ in ‘arithmetic.bm’.

> So, those are the problems: benchmarks running for inappropriate,
> inconsistent durations;

I don’t really see such a problem.  It doesn’t matter to me if
‘arithmetic.bm’ takes 2mn while ‘vlists.bm’ takes 40s, since I’m not
comparing them.

> inappropriate benchmarks;

I agree that things like ‘if.bm’ are not very relevant now.  But there
are also appropriate benchmarks, and benchmarks are always better than
wild guess.  ;-)

> and benchmarks being optimized out.

That should be fixed.

> My proposal is to rebase the iteration count in 0-reference.bm to run
> for 0.5s on some modern machine, and adjust all benchmarks to match,
> removing those benchmarks that do not measure anything useful.

Sounds good.  However, adjusting iteration counts of the benchmarks
themselves should be done rarely, as it breaks performance tracking like
<http://ossau.homelinux.net/~neil/bm_master_i.html>.

> Finally we should perhaps enable automatic scaling of the iteration
> count.  What do folks think about that?
>
> On the positive side, all of our benchmarks are very clear that they are
> a time per number of iterations, and so this change should not affect
> users that measure time per iteration.

If the reported time is divided by the global iteration count, then
automatic scaling of the global iteration count would be good, yes.

Thanks,
Ludo’.