our benchmark-suite

[Andy Wingo]

Hi,

I was going to try to optimize vhash-assoc, but I wanted a good
benchmark first, so I started to look at our benchmark suite.  We have
some issues to deal with.

For those of you who are not familiar with the benchmark suite, we have
a bunch of benchmarks in benchmark-suite/benchmarks/: those files that
end in ".bm".  The format of a .bm file is like our .test files, except
that instead of `pass-if' and the like, we have `benchmark'.  You run
benchmarks via ./benchmark-guile in the $top_builddir.

The benchmarking framework tries to be appropriate for microbenchmarks,
as the `benchmark' form includes a suggested number of iterations.
Ideally when you create a benchmark, you give it a number of iterations
that makes it run approximately as long as the other benchmarks.

When the benchmarking suite was first made, 10 years ago, there was an
empty "reference" benchmark that was created to run for approximately 1
second.  Currently it runs in 0.012 seconds.  This is one problem: the
overall suite has old iteration counts.  There is a facility for scaling
the iteration counts of the suite as a whole, but it is unused.

Another problem is that the actual runtime of the various benchmarks
varies quite a lot, from 3.3 seconds for assoc (srfi-1), to 0.012 for
if.bm.

Short runtimes magnify imprecisions in measurement.  It used to be that
the measurement function was "times", but I just changed that to the
higher-precision get-internal-real-time / get-internal-run-time.  Still,
though, there is nothing you can do for a benchmark that runs in a few
milliseconds or less.

Another big problem is that some effect-free microbenchmarks optimize
away.  For example, the computations in arithmetic.bm fold entirely.
The same goes for if.bm.  These benchmarks do not measure anything
useful.

The benchmarking suite attempts to compensate for the overhead of the
test by providing for "core time": the time taken to run a benchmark,
minus the time taken to run an empty benchmark with the same number of
iterations.  The benchmark itself is compiled as a thunk, and the
framework calls the thunk repeatedly.  In theory this sounds good.  In
practice however, for high-iteration microbenchmarks, the overhead of
the thunk call outweighs any micro-benchmark being called.

For what it's worth, the current overhead of the benchmark appears to be
about 35 microseconds per iteration, on my laptop.  If we inline the
iteration into the benchmark itself, rather than calling a thunk
repeatedly, we can bring that down to around 13 microseconds.  However
it's probably best to leave it as it is, because if we inline the loop,
it's liable to be optimized out.

So, those are the problems: benchmarks running for inappropriate,
inconsistent durations; inappropriate benchmarks; and benchmarks being
optimized out.

My proposal is to rebase the iteration count in 0-reference.bm to run
for 0.5s on some modern machine, and adjust all benchmarks to match,
removing those benchmarks that do not measure anything useful.  Finally
we should perhaps enable automatic scaling of the iteration count.  What
do folks think about that?

On the positive side, all of our benchmarks are very clear that they are
a time per number of iterations, and so this change should not affect
users that measure time per iteration.

Regards,

Andy
-- 
http://wingolog.org/