Re: octave benchmark test

[David Bateman]

According to Michael Martin <address@hidden> (on 03/09/04):
> 
> On Mar 9, 2004, at 3:44 AM, David Bateman wrote:
> 
> >1) Upgrade to 2.1.56 from 2.1.53 due to the bugs in 2.1.53 and some 
> >speed-up
> 
>       I will do that if I get a chance. My installation is via fink, which 
> is generally clean and easy, if not always the latest ad greatest. 
> Since I am in a production mode, I would prefer not to live on the 
> bleeding edge. Fink is quite acceptable to me. Mac OSX has ATLAS & BLAS 
> built-in via its Frameworks, and my octave/fink does indeed use them. I 
> would say that my results show this given the similarity between my 
> numbers for the matrix operations and Paul Thomas' numbers. Though we 
> are running on different architectures, the same sorts of numbers are 
> seen in both or our octave and matlab results. fftw libs is also 
> required by fink, by the way.

Humm, as for bleeding-edge 2.1.53 is as bleeding as it gets. Since then
has mostly be bug fixes. Try 

octave:1> a(:,:,1) = ones(2,2);
octave:2> a(:,:,2) = 2*ones(2,2)
a =

ans(:,:,1) =

  1  1
  1  1

ans(:,:,2) =

  2  2
  2  2


and see what your answer is. There are lots of other fixes, though.

octave as of 2.1.54 needs FFTW3 and not FFTW2. So the presence of FFTW
on your system is not a guarantee that it can be used.

> 
>       I have experimented over the last several years with compiler 
> settings, compilers etc. I have run different level of both octave and 
> matlab, and on different platforms, etc. The results have shown pretty 
> much the same trends wrt looping. I doubt an upgrade will change the 
> results significantly.

Loop performance is related to JIT. Vectorize ALL loops... If you can
avoid the loop, the the code is a strong candidate for conversion to
an oct-file. In this manor I have virtually no performance difference
between matlab/octave.

> >2) Please install the latest version of octave-forge where the sort 
> >and randn
> >   codes are sped-up, which you are clearly missing.
> 
>       Yes, I will try and do that. I am sure the discrepancy in the sort 
> times was due to different codes being run. Again, that is consistent 
> with Paul Thomas' numbers and mine. Ours match pretty closely EXCEPT 
> for the sort. Ergo different codes are likely the source of the 
> difference.
> 
> Taking the next two out of order....
> 
> >Vectorize code wherever possible!!!!! Octave doesn't have JIT which is
> >where Matlab wins on this case. The above is a null example, since the
> >core of the algorithm results in a null operation as
> >'tan(atan(exp(log(sqrt(a*a))))) = a', so a vectorized version is
> >
> >a = 2001;
> 
>       You have missed the point of the test. If the test had been to 
> calculate 2001 as quickly as possible, then I would certainly agree! 
> :-) However, the test was not to see how quickly one can calculate 
> 2001, but to test the speed and numerical stability of ocatve for the 
> common transcendental functions. Since 2001 and is a nice 
> human-readable number, it is easy too subtract it from the result get a 
> feel for the error. The repeated use of the transcendental functions 
> and inverses will usually quickly show if there are any serious 
> problems with them.
> 
>       I always run this test whenever I port code to a new platform or 
> whenever a platform has seen a significant OS or compiler update. 
> Usually I use my standard C program and Fortran program that I have had 
> for what must be close to 20 years now. However, when I am working with 
> a package such as octave or matlab, I will code it into the package's 
> langauge and test it there. In the past this test has revealed 
> significant problems with the code of transcendental functions on more 
> than one platform/compiler/package. All of my applications make 
> extensive use of transcendental functions and any significant errors in 
> them will lead to erroneous results. Call me paranoid :-), but since my 
> algorithms ends up in manned spacecraft, I would rather know **before** 
> flight that my codes have problems in them as opposed to **in** flight, 
> at which time it would invariably be too late. :-)
> 
>       Now octave certainly passed the numerical stability test, so that is 
> not an issue. However, the test did reveal that for whatever reasons, 
> this particular piece of code does not run as fast in octave as opposed 
> to matlab. My intent on this test was simply to inform the community. 
> Perhaps someone is currently working on some octave code that may find 
> the data of use.

I didn't miss the point as such..... We all know that Matlab is faster in
loops by many orders of magnitude. However, there are no reason to use loops
in virtually all cases.

As a test of numerical stability, sure your test is fine. But its not a 
benchmark and this type of abuse of loops should be avoided at all costs
(even in Matlab)

> >4) For the 100x case please vectorize your code....
> 
>       Regrettably, this is not possible in the particular case I 
>       mentioned. Not all codes can be vectorized as much as one might like. 
> There 
> a great number of different reasons for this. Let me touch upon just two 
> of them; namely algorithmic similarity, and heuristic type codes.
> 
>       In the case of algorithmic similarity, if one has a code that 
>       produces a result independently of other systems, then one does indeed 
> have 
> great flexibility in coding style, etc. However, if one has a code or 
> algorithm that one is testing and that will be used in the "real 
> world", then it is best to keep the code as reasonably close to what 
> will run in the "real world". In other words, if the goal of the 
> analysis is to ensure the algorithm will behave properly under a wide 
> range of input scenarios in the real world, then the algorithm should 
> be kept as close as possible to the actual algorithm as it will be used 
> in the real world. Only then can one have confidence that the algorithm 
> will behave the same in the real world as it is behaving in the 
> laboratory.
> 
>       In the case of heuristic type codes, at least the ones I have, there 
> are no linear/matrix formulations of the problem. Yes, vectors are 
> involved. Vectors are associated with the control system effectors and 
> vector/matrix calculations are taking place, so at that level the code 
> can be vectorized. However, these vectors are searched for suitability 
> to satisfy the control system requirements. The problem is 
> combinatorial and cannot really be vectorized. Not only does this 
> control system algorithm not lend itself very well to vectorization, it 
> itself is used in a non-vectorized fashion. A great number of cases 
> with varying control system requirements are run in a non-linear 
> fashion in order to determine control system performance and robustness 
> at extrema.
> 
>       Again, my intent on mentioning looping & arrays (where the 
> inefficiency seems to lie),  was simply to inform the community. 
> Perhaps someone is currently working on some octave code that may find 
> the observations of use. Given how well octave does with most of the 
> common vector and matrix functions, I would propose that if folks are 
> looking at speed improvement strategies for the octave code, then there 
> is much gold to mine here with looping & arrays. More so, I would say 
> then further improvements to matrix & vector functions, since for the 
> most part they are on par with matlab, if not better.

Yeah, I have some viterbi codes that also couldn't be vectorized. In this
case, they should be implemented as oct-files....

The improvement in loops can only come from a complete rewrite of the octave
parser, trying to get it to predict the types of arguments early and spit
out byte-code for stuff it runs several times. Much like what matlab does.
However, the amount of work involved in this is overwhelming and you won't
see anything soon on this front. 

Regards
David

-- 
David Bateman                                address@hidden
Motorola CRM                                 +33 1 69 35 48 04 (Ph) 
Parc Les Algorithmes, Commune de St Aubin    +33 1 69 35 77 01 (Fax) 
91193 Gif-Sur-Yvette FRANCE

The information contained in this communication has been classified as: 

[x] General Business Information 
[ ] Motorola Internal Use Only 
[ ] Motorola Confidential Proprietary



-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.

Octave's home on the web:  http://www.octave.org
How to fund new projects:  http://www.octave.org/funding.html
Subscription information:  http://www.octave.org/archive.html
-------------------------------------------------------------