Re: [Gcl-devel] Re: Boyer benchmark results

[Camm Maguire]

Greetings!

Eric Marsden <address@hidden> writes:

> I apologize for coming in late in this discussion, but I don't read
> this list very regularly.
> 

No problem, and thanks for your contributions (to the discussions, gcl
bug reports, and cl-bench of course!)

> I am the maintainer of the cl-bench suite that has been discussed in
> this thread. This aims to provide reasonably objective performance
> comparisons between ANSI-compliant CL implementations, but to a
> certain extent accepts workarounds for ANSI compliance bugs in
> different implementations. Until recently GCL had a serious bug in
> EVAL-WHEN handling that prevented cl-bench from running, but Camm just
> fixed it (thanks!), inspiring me to retry the suite with GCL.
> 
> A few bug reports later, I have obtained the following provisional
> results. These are run on a dual-processor 1.1GHz i686 machine running
> GNU/Linux; the same optimization qualities are used for all
> implementations: (speed 3) (safety 0). They were not carried out in
> proper (zero load) benchmark conditions, so please do not quote them.
> 
> ,----
> | Benchmark                 Reference     GCL    SBCL    CLISP   
> | ------------------------------------------------------------
> | COMPILER                 [      3.70]  -1.00   1.77    0.77   
> | LOAD-FASL                [      0.36]  -1.00   2.22    2.22   
> | SUM-PERMUTATIONS         [      4.30]   1.18   1.55    1.59   
> | WALK-LIST/SEQ            [      0.19]   1.89   0.37    0.53   
> | WALK-LIST/MESS           [      0.19]  13.68   0.32    1.84   
> | BOYER                    [      8.73]   0.18   1.13   11.42   
> | BROWSE                   [      0.81]   4.44   1.44    2.69   
> | DDERIV                   [      0.53]   3.89   1.77   12.49   
> | DERIV                    [      0.61]   3.59   1.82   10.93   
> | DESTRUCTIVE              [      0.82]   3.49   1.23    4.05   
> | DIV2-TEST-1              [      1.07]   3.87   1.67    8.92   
> | DIV2-TEST-2              [      1.54]   2.96   1.45    7.09   
> | FFT                      [      0.14]  30.21   0.79   52.36   
> | FRPOLY/FIXNUM            [      0.83]   2.41   1.18    9.22   
> | FRPOLY/BIGNUM            [      1.09]   0.83   1.06    4.70   
> | FRPOLY/FLOAT             [      1.29]   1.40   1.21    7.97   
> | PUZZLE                   [      0.51]   5.43   1.25   34.41   
> | TAK                      [      0.71]   1.55   0.94   10.89   
> | CTAK                     [      0.71]   2.65   1.11    7.34   
> | TRTAK                    [      0.67]   1.64   1.06   11.45   
> | TAKL                     [      1.18]   0.68   0.75    9.64   
> | STAK                     [      0.85]   1.02   1.69    7.64   
> | FPRINT/UGLY              [      1.46]   1.23   1.75    1.47   
> | FPRINT/PRETTY            [      5.18]   0.34   1.34    1.23   
> | TRAVERSE                 [      1.51]   3.16   2.15   12.52   
> | TRIANGLE                 [      1.30]   2.35   1.12   25.59   
> | RICHARDS                 [      0.95]   1.24   1.24   23.87   
> | FACTORIAL                [      0.68]   1.69   1.24    6.84   
> | FIB                      [      0.82]   4.41   1.01    5.40   
> | FIB-RATIO                [      0.66]   9.59   1.08    0.09   
> | ACKERMANN                [     10.38]  17.32   1.04   11.07   
> | MANDELBROT/COMPLEX       [      1.44]   1.44   1.26    4.62   
> | MANDELBROT/DFLOAT        [      0.12]  10.08   0.50   38.42   
> | MRG32K3A                 [      1.28]   4.39   1.01   64.30   
> | CRC40                    [     33.12]   7.14   1.28    7.70   
> | BIGNUM/ELEM-100-1000     [      0.97]   0.72   1.05    0.10   
> | BIGNUM/ELEM-1000-100     [      4.95]   0.40   1.01    0.05   
> | BIGNUM/ELEM-10000-1      [      5.42]   0.32   0.97    0.04   
> | BIGNUM/PARI-100-10       [      1.19]   0.03   0.98    0.03   
> | BIGNUM/PARI-200-5        [     14.19]   0.01   0.96    0.01   
> | PI-DECIMAL/SMALL         [     47.36]   4.72   0.97    0.09   
> | PI-DECIMAL/BIG           [     95.08]   3.50   0.96    0.02   
> | PI-ATAN                  [      2.42]   3.75   1.14    3.06   
> | PI-RATIOS                [      8.92]   2.41   1.02    0.12   
> | HASH-STRINGS             [      0.77] 165.82   1.04    3.09   
> | HASH-INTEGERS            [      1.66]   0.82   1.63    2.55   
> | SLURP-LINES              [      5.10]   1.12   1.07    1.65   
> | BOEHM-GC                 [      2.52]   3.38   1.62   14.95   
> | DEFLATE-FILE             [      1.29]   1.74   0.97    5.40   
> | 1D-ARRAYS                [      0.20]   0.00   1.35    7.75   
> | 2D-ARRAYS                [      1.77]   0.05   0.51   21.36   
> | 3D-ARRAYS                [      4.83]   0.04   0.60   16.59   
> | BITVECTORS               [      1.74]   0.01   1.32   13.77   
> | BENCH-STRINGS            [      4.60]   0.00   0.24    0.72   
> | fill-strings/adjustable  [     47.71]   0.00   0.94    2.02   
> | STRING-CONCAT            [    146.96]   0.00   0.42    6.59   
> | SEARCH-SEQUENCE          [      4.94]   0.00   0.11    3.75   
> | CLOS/defclass            [      6.55]   0.23   0.73    0.11   
> | CLOS/defmethod           [     12.99]   0.24   1.07    0.01   
> | CLOS/instantiate         [     18.00]   0.31   0.95    0.64   
> | CLOS/simple-instantiate  [      0.45]  98.27   1.33   13.24   
> | CLOS/methodcalls         [      2.89]   1.83   1.66    5.25   
> | CLOS/method+after        [      9.74]   1.15   1.48    0.67   
> | CLOS/complex-methods     [      0.14]   8.00   7.50   -1.00   
> | EQL-SPECIALIZED-FIB      [      0.67]   2.45   1.06    4.28  
> | Reference time in first column is in seconds; other columns are relative
> | Reference implementation: CMU Common Lisp 18e+, level-1 built 2004-06-11 on 
> melbourne
> | Impl GCL: Kyoto Common Lisp GCL 2.7.0
> | Impl SBCL : SBCL 0.8.10.19
> | Impl CLISP: CLISP 2.32 (2003-12-29) (built 3281883514) (memory 3281883691)
> | === Test machine ===
> |    Machine-type: X86
> |    Machine-version: X86
> | Linux melbourne 2.4.17 #2 SMP Wed Feb 6 11:35:03 CET 2002 i686 GNU/Linux
> `----
> 
> Note that the numbers in all columns except for the first are
> relative, so for instance 1.18 means that GCL 2.7.0 with --enable-ansi
> from anoncvs is 18% slower on the SUM-PERMUATIONS test than
> CMUCL-19a-pre. A value of -1 means that the test was not run for that
> implementation (probably due to a bug in that implementation).
> 
> It is interesting to note that GCL is a lot faster than CMUCL on
> certain tests such as BOYER, the array tests and the bignum tests; I
> suspect that GCC/GCL is "cheating" on some of the array tests by
> deleting unused code. Note however that it is hugely slower than CMUCL
> on many other tests, such as FFT and ACKERMANN. 

In fairness to cmucl, I am suspicious of the magnitude of some of
these supposed GCL performance gains.  Are you sure the tests are
running to completion?

> 
> I strongly suggest that people interested in performance comparisons
> use cl-bench as a basis for their measurements, for a number of
> reasons:
> 

I am very supportive of the idea of using cl-bench as a major tool for
gcl benchmarking.

>    - it includes all the Gabriel benchmarks, but with a few bugfixes
>      and with some rewriting to more closely ressemble typical CL
>      programming style.
> 

Haven't checked here with cl-bench, but I've discovered independently,
and its also recently appeared on comp.lang.lisp, that some versions
of boyer floating around are including the setup in the benchmark --
even worse, the size of the problem increases with each object file
load.  You might want to check the code you are using vs. the
differences discussed in the earlier gcl boyer benchmarking thread.

Apart from this, I hope someone is going to resize the gabriel tests
for modern machines, eliminating the need to loop over tiny tests many
times. 

Also, what about the license?  Is all of this code DFSG free?

>    - I have attempted to include more modern CL benchmarks, such as
>      the CLOS code, the bignum processing code, array and hashtable
>      tests etc. These should give greater coverage of a CL
>      implementation's performance characteristics.
> 
>    - the benchmark methodology attempts to be as fair as possible
>      between implementations; for example I consider Camm's prelinking
>      FTYPE declarations for GCL to be cheating. Other factors to
>      consider are running implementations with the same heap size, and
>      disabling "development-only" settings such as collecting XREF
>      information.
> 

I agree about being fair, but I disagree about function prototype
proclamations being 'cheating'.  The results I posted proclaimed these
prototypes for all lisps tested.  Beyond this, these proclamations
were generated by the compiler, not by a human.  The reason it is not
simply bundled into compile-file is that the facility has the ability
to make use of function proclamation information across many files --
a sort of 'system cross-reference'.  But if it would make you feel
more comfortable, it is probably a good idea in any case to simply
have gcl's compile file run an extra pass and generate these
proclamations for the file in question (alone) automatically with the
same ansi compile-file interface.  This would achieve the same result
I believe in these tests which are encapsulated within separate files.

> The version of cl-bench that is currently on my web site does not run
> in GCL, but I shall shortly make a new release that does work. 
> 

Thanks for your work!

Take care,

> -- 
> Eric Marsden                          <URL:http://www.laas.fr/~emarsden/>
> 
> 
> 
> _______________________________________________
> Gcl-devel mailing list
> address@hidden
> http://lists.gnu.org/mailman/listinfo/gcl-devel
> 
> 

-- 
Camm Maguire                                            address@hidden
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah