Re: MATLAB / Scilab / Octave benchmarks

[Andreas Weber]

Am 24.07.2016 um 09:14 schrieb roland65:
> I did some benchmarks to compare the run times of MATLAB, Scilab and Octave
> on 50 programs in 4 benchmark sets.
> The whole thing is is free and is available  here
> <http://roland65.free.fr/benchmarks>  .
> I'll be glad to discuss these results with you.
Thank you for sharing. I think the table in the pdf could need some
color. For example apply a colormap so the best one is green and the bad
values are red or something like that.

One cause that Octave is slower than others is that the code makes
extensive use of for loops and the JIT compiler currently is only able
to optimize simple loops for example the 1000x1000 loop in your benchmark.

If you use the profiler for the "Standard PSO" you'll see that the most
time is spent inside your rosenbrock function. The comment says it's
vectorized but it's not. As soon as you replace roenbrock with a
vectorized version
(https://sourceforge.net/p/octave/optim/ci/default/tree/inst/rosenbrock.m)
I get

Optimization of Rosenbrock function in dimension 30
Method : Particle Swarm with 40 particles

=> Iteration     1 - gbest = 7.57694e+06
....
Best minimum : gbest = 6.4188e+06
Run time : 2.80419 sec.

So you're mostly benchmarking for loops (also standard_pso.m uses them).


I've also looked at your FFT test where you are implementing your own
fft as recursive call which no one should ever do.
If you call Octaves "fft" which uses fftw you'll get:

octave:> x = rand(2^15,1); tic; y=fft(x); toc
Elapsed time is 0.00282216 seconds.

3ms compared to 4000ms.
Again, you're not benchmarking fft here but recursive function calls.

-- Andy