[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: octave benchmark test
|
From: |
Michael Martin |
|
Subject: |
Re: octave benchmark test |
|
Date: |
Mon, 8 Mar 2004 11:58:30 -0600 |
Let me say at the start that I am a big fan of octave. I use both
octave and matlab, and I prefer octave for the most part. It is less
capricious and cantankerous than matlab for the most part. Also, with
matlab, I had better be able to see the license server or I am out of
luck. That is not always possible and there are times when I am in a
mission critical situation and cannot use matlab because of this.
There are places, however, where matlab simply outruns octave by a big
margin to the point where octave is nearly unusable for a given task.
The run time difference may be as much as 100x. For one type of
analysis, matlab's response takes a minute and while octave's is two
hours. Even as a die hard octave fan, I will think twice before I run
those sorts of analyses in octave as opposed to matlab.
Given that some benchmarks have been posted, I will take the liberty of
posting some of my results too. I hope it will generate a profitable
discussion.
The following is the result for the total time of the Octave2 tests on
a 2GHz G5 under OS X using both matlab (6.5.1) and octave (2.1.53). (I
have posted all the results at the end in case they are of interest)
Octave: Total time for all 15 tests_________________________ (sec):
15.53
Matlab: Total time for all 15 tests_________________________ (sec):
5.0555
As you can see, about a 3x difference in the total time. The big
offender is the sort, 7.348 for octave and 0.41325 for matlab. The next
largest is the Toeplitz matrix, 1.5 for octave and 0.012 for matlab.
The third largest difference is the eigenvalues of a 320x320 matrix,
1.144 for octave and 0.43039 for matlab. The forth largest is
Escoufier's method, 1.713 for octave and 0.96309 for matlab. All told
that accounts for almost 10 seconds of the 10.5 second difference.
Now some of the differences are quite likely due to algorithmic
differences. The difference in sorting is certainly likely algorithmic,
though given that my other numbers are slightly better than the times
in Paul Thomas' post, I have to wonder why my sort is quite so much
worse than his.
Another observation is that this test, Octave2, is mostly testing
algorithmic differences between matlab and octave. The individual test
cases generally execute a particular function or set of functions
repeatedly with matrices being operated on. The major portion of the
execution time would be expected to be in the function call(s). It has
been my experience that octave for the most part does as well as
matlab, and in a few cases (depending upon the edition of matlab) much
better when one gets down to this level, namely function calls. Where I
see problems in octave is looping & array functions. Let me add some
results from my own benchmarks. I ran this benchmark on matlab and
octave. The times for each test are reported and then matlab's speed
relative to octave is posted below the test result.
Float test
octave: a = 2001.000000, err = 0.000000, time = 7.752197
matlab: a = 2001.000000, err = 0.000000, time = 0.180680
speedX: 42.906
Sqrt test
octave: time = 0.114017
matlab: time = 0.184528
speedX: 0.61788
Log test
octave: time = 0.150121
matlab: time = 0.158714
speedX: 0.94586
Exp test
octave: time = 0.087117
matlab: time = 0.021750
speedX: 4.0054
Atan test
octave: time = 0.113081
matlab: time = 0.045912
speedX: 2.4630
Tan test
octave: time = 0.166375
matlab: time = 0.221086
speedX: 0.75254
Matrix test
octave: time = 0.001770
matlab: time = 0.001488
speedX: 1.1895
Sieve test using arrays
octave: time = 0.020089
matlab: time = 0.001444
speedX: 13.912
Sieve test using loops
octave: time = 0.092604
matlab: time = 0.001667
speedX: 55.551
The float test is a rather interesting result. The code for it is as
follows:
%% Float test ---------------------------------------------------
fprintf('Float test\n');
tic
for j = 1:100
a = 1;
for i = 1:2000
a = tan(atan(exp(log(sqrt(a*a))))) + 1;
end
end
elapsedTime = toc;
fprintf ('a = %f, err = %f, time = %f\n\n', a, a - 2001, elapsedTime);
As you can see, there is nothing remarkable about this test, other than
it runs some 42x slower on octave than matlab. I use it all the time to
test numerical stability of transcendental functions on whatever
computer I am running on (though usually in C).
It is the individual transcendental function tests that may be
revealing the answer, at least in part. Each of those test takes a
100x100 matrix and performs a given transcendental function on each
element of the matrix (e.g. result = log(matrix)). On the sqrt, log,
and tan tests, octave comes out ahead of matlab. But on the exp octave
is 4x slower and on the atan, octave is 2.5x slower. Algorithmic
certainly, but if one deals with these transcendental functions a lot,
then the slow down can be crushing, as was observed in the float test.
The other observation is the Sieve tests. This is the classic sieve of
Eratosthenes. The first method uses array operators to implement the
algorithm, the second method uses loops, much like one would do in a C
program. There is a very large slowdown with the arrays (13X), and then
the looping version takes a 55x hit over matlab.
If there are folks working on speed improvements, I would suggest that
you have parity with matlab as far as the built-in functions go for the
most part, but arrays and looping definitely need work.
[Yes, I know that conversion of some of these sorts of algorithms to C
& .oct files will improve speed considerably, but I do not always have
that option for a whole host of reasons.]
----------------------------------------------------------------------
Michael W. Martin Phone: (281) 333-2177
Draper Laboratory FAX: (281) 333-5276
2200 Space Park Dr. EMail: address@hidden
Houston, TX 77058 WWW: http://www.jsc.draper.com/
USA Mail Code: EG/Draper
----------------------------------------------------------------------
Octave Benchmark 2
==================
Number of times each test is run__________________________: 3
---------------------
Creation, transp., deformation of a 1500x1500 matrix (sec):
1.251 [0.48235]
800x800 normal distributed random matrix ^1000______ (sec):
0.1284 [0.21914]
Sorting of 2,000,000 random values__________________ (sec):
7.348 [0.41325]
700x700 cross-product matrix (b = a' * a)___________ (sec):
0.1312 [0.19244]
Linear regression over a 600x600 matrix (c = a \ b') (sec):
0.09014 [0.10584]
------------------------------------------------------
Trimmed geom. mean (2 extremes eliminated): 0.2762
II. Matrix functions
--------------------
FFT over 800,000 random values______________________ (sec):
0.2844 [0.28988]
Eigenvalues of a 320x320 random matrix______________ (sec):
1.144 [0.43039]
Determinant of a 650x650 random matrix______________ (sec):
0.1085 [0.095085]
Cholesky decomposition of a 900x900 matrix__________ (sec):
0.1051 [0.13952]
Inverse of a 400x400 random matrix__________________ (sec):
0.06585 [0.073918]
------------------------------------------------------
Trimmed geom. mean (2 extremes eliminated): 0.148
III. Programmation
------------------
750,000 Fibonacci numbers calculation (vector calc)_ (sec):
0.5173 [0.77387]
Creation of a 2250x2250 Hilbert matrix (matrix calc) (sec):
0.7473 [0.6179]
Grand common divisors of 70,000 pairs (recursion)___ (sec):
0.3986 [0.24719]
Creation of a 220x220 Toeplitz matrix (loops)_______ (sec):
1.5 [0.011642]
Escoufier's method on a 37x37 matrix (mixed)________ (sec):
1.713 [0.96309]
------------------------------------------------------
Trimmed geom. mean (2 extremes eliminated): 0.8338
Total time for all 15 tests_________________________ (sec):
15.53 [5.0555]
Overall mean (sum of I, II and III trimmed means/3)_ (sec): 0.3243
--- End of test ---
My speed.m file:
%% Speed tests
%% Float test ---------------------------------------------------
fprintf('Float test\n');
tic
for j = 1:100
a = 1;
for i = 1:2000
a = tan(atan(exp(log(sqrt(a*a))))) + 1;
end
end
elapsedTime = toc;
fprintf ('a = %f, err = %f, time = %f\n\n', a, a - 2001, elapsedTime);
%% Needed for following ----------------------------------------
mat = reshape(1:10000, 100,100);
%% sqrt test ---------------------------------------------------
fprintf('Sqrt test\n');
tic
for j = 1:100
mata = sin(mat);
end
elapsedTime = toc;
fprintf ('time = %f\n\n', elapsedTime);
%% log test ---------------------------------------------------
fprintf('Log test\n');
tic
for j = 1:100
mata = log(mat);
end
elapsedTime = toc;
fprintf ('time = %f\n\n', elapsedTime);
%% exp test ---------------------------------------------------
fprintf('Exp test\n');
tic
for j = 1:100
mata = exp(mat);
end
elapsedTime = toc;
fprintf ('time = %f\n\n', elapsedTime);
%% Atan test ---------------------------------------------------
fprintf('Atan test\n');
tic
for j = 1:100
mata = atan(mat);
end
elapsedTime = toc;
fprintf ('time = %f\n\n', elapsedTime);
%% Tan test ---------------------------------------------------
fprintf('Tan test\n');
tic
for j = 1:100
mata = tan(mat);
end
elapsedTime = toc;
fprintf ('time = %f\n\n', elapsedTime);
%% Matrix test ---------------------------------------------------
fprintf('Matrix test\n');
tic
res = mat*mat;
elapsedTime = toc;
fprintf ('time = %f\n\n', elapsedTime);
%% sieve test ---------------------------------------------------
fprintf('Sieve test using arrays\n');
tic
xlen = 2000;
x = linspace(1, xlen, xlen);
for i = 2:xlen/2
if x(i) ~= 0
x(2*i:i:xlen) = 0;
end
end
elapsedTime = toc;
fprintf ('time = %f\n\n', elapsedTime);
fprintf('Sieve test using loops\n');
tic
x = linspace(1, xlen, xlen);
for i = 2:xlen/2
if x(i) ~= 0
for j = 2*i:i:xlen
x(j) = 0;
end
end
end
elapsedTime = toc;
fprintf ('time = %f\n\n', elapsedTime);
-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.
Octave's home on the web: http://www.octave.org
How to fund new projects: http://www.octave.org/funding.html
Subscription information: http://www.octave.org/archive.html
-------------------------------------------------------------