Re: Benchmarking Octave 322

[Francesco Potorti`]

>There is the benchmark package at
>http://octave.sourceforge.net/benchmark/index.html
>
>and also some codes all over the web (mostly sadly out-of-date).

In the past I used to maintain a benchmark.m, which is still available
at <http://fly.isti.cnr.it/pub/software/octave/benchmark.m>.  It still
works on Octave 3.2.2 and claims to compute the performance relative to
Octave 2 on a Pentium II at 350 MHz with Debian's lapack3 and refblas3
as of November 2005.  Its main asset is that it computes a 95%
confidence interval for the results, something that as far as I can see
is missing from your code.

I even used to publish results sent from people on
<http://fly.isti.cnr.it/pub/software/octave/bm_results>, back when the
reference was Sun Sparc, about 100 times slower than the current Pentium
II reference.  I stopped updating the results when compiling Octave
became so much dependent on libraries and compiling options that
producing a single number for a given hardware would have been
misleading, and maintaining a significant list would have been too
complex.

I often thought about working to merge it with your code, but never
found the time, and I find myself adding things in front of this on my
wishlist.  So I am writing this mail, in the hope that you, or someone
else, possiby with my help, can work towards a better and easy-to-use
benchmark.  It would also be nice to have a page on the Octave wiki with
some performance results.

For a reference, here is what I see using my benchmark on an AMD Athlon
64 X2 3800+ running Debian's Octave 3.2.2:

octave> benchmark
Octave benchmark version bm 2.3
Speed of octave 3.2.2 on x86_64-pc-linux-gnu
  Pentium II 350 MHz has score 1.0, higher numbers are better
Matrix inversion (LAPACK)         9.05 +/- 0.3% (154 runs)
Schur decomposition (LAPACK)     23.88 +/- 0.2% (126 runs)
Differential equation (LSODE)    22.83 +/- 0.3% (546 runs)
Fourier transforms (FFTPACK)      7.43 +/- 0.5% (196 runs)
for loop                         34.76 +/- 1.0% (658 runs)
-- Performance index (bm 2.3): 17

It is surprising to me to see that such a modern processor is less than
ten times faster on some tests than a Pentium II!

-- 
Francesco Potortì (ricercatore)        Voice: +39 050 315 3058 (op.2111)
ISTI - Area della ricerca CNR          Fax:   +39 050 315 2040
via G. Moruzzi 1, I-56124 Pisa         Email: address@hidden
(entrance 20, 1st floor, room C71)     Web:   http://fly.isti.cnr.it/