Re: [Gcl-devel] Boyer benchmark results

[Camm Maguire]

Greetings, and thanks for this!

Peter Graves <address@hidden> writes:

> Hi,
> 
> I've done another run of Boyer benchmarks, using the approach Camm
> suggested. I did make a minor change to sys-proclaim.lisp:
> 
>     #-(or cmu clisp)
>     (IN-PACKAGE "USER")
>     #+(or cmu clisp)
>     (IN-PACKAGE "CL-USER")
> 
> I also changed the invocation of cmucl to "/usr/bin/lisp" (instead of
> just "lisp") to avoid picking up a newer version that I have installed
> in /usr/local/bin.
> 
> Here are the results:
> 
>     Desired=Unknown/Install/Remove/Purge/Hold
>     | Status=Not/Installed/Config-files/Unpacked/Failed-config/Half-installed
>     |/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: 
> uppercase=bad)
>     ||/ Name           Version        Description
>     
> +++-==============-==============-============================================
>     ii  gcl            2.6.1-40       GNU Common Lisp compiler
>     ii  gclcvs         2.7.0-24       GNU Common Lisp compiler, CVS snapshot
>     ii  cmucl          18e-10         The CMUCL lisp compiler and development 
> system
>     ii  clisp          2.33.2-1       GNU CLISP, a Common Lisp implementation
>     result gcl  "boyer.lisp" T 2.8999999999999999
>     result gcl  "boyer.lisp" NIL 3.23
>     result gcl  "boyer1.cl2" T 2.4500000000000002
>     result gcl  "boyer1.cl2" NIL 2.8700000000000001
>     result gcl t "boyer.lisp" T 2.9900000000000002
>     result gcl t "boyer.lisp" NIL 3.29
>     result gcl t "boyer1.cl2" T 2.5600000000000001
>     result gcl t "boyer1.cl2" NIL 2.96
>     result gclcvs  "boyer.lisp" T 2.8799999999999999
>     result gclcvs  "boyer.lisp" NIL 3.21
>     result gclcvs  "boyer1.cl2" T 2.4399999999999999
>     result gclcvs  "boyer1.cl2" NIL 2.8500000000000001
>     result gclcvs t "boyer.lisp" T 3.0
>     result gclcvs t "boyer.lisp" NIL 3.3199999999999998
>     result gclcvs t "boyer1.cl2" T 2.5899999999999999
>     result gclcvs t "boyer1.cl2" NIL 2.98
>     result /usr/bin/lisp  "boyer.lisp" T 2.9
>     result /usr/bin/lisp  "boyer.lisp" NIL 2.86
>     result /usr/bin/lisp  "boyer1.cl2" T 2.59
>     result /usr/bin/lisp  "boyer1.cl2" NIL 2.61
>     result clisp  "boyer.lisp" T 17.699308
>     result clisp  "boyer.lisp" NIL 17.68931
>     result clisp  "boyer1.cl2" T 15.440652
>     result clisp  "boyer1.cl2" NIL 15.425655
> 

These results still differ in interesting ways from mine, which might
point out avenues for future improvement.

Paul has separately written speculating on cache.  The only logic I
can conceive of being behind these numbers is if 1) gcl's assembler
output via gcc is faster in cache then that output by cmucl, 2) cmucl
either makes use of a superior copying gc algorithm for the cons
storage (improving locality), or makes use of explicit prefetch
instructions for much better memory access times than gcl, at least
for cons storage, and 3) clisp also does something like 2), but not as
well as cmucl.  This 'theory' would also suppose that cpu and memory
access are roughly 50-50 contributors to this test.

I'd be appreciative if someone could confirm/refute my speculations.
In any case, the end user is likely to see results more favorable to
cmucl than that which I've posted depending on their machine.  The
best course would appear to be to select a representative machine which
reverses these strengths/weaknesses to the extent possible, rerun the
gabriel suite, and report both in the release notes.  Thoughts?


> Using sys-proclaim.lisp and boyer1.cl2, gcl in non-ANSI mode is indeed
> a bit faster than cmucl (2.45 vs 2.59).
> 

Christophe is right that in this case the conclusion is apples and
oranges, not so much because of the sys-proclaims, IMHO, but rather
that the difference is within the effect of the larger ansi image size
on the result, making it only fair to use the ansi result when
comparing with cmucl.  I don't know what 'block' mode in cmucl is, but
I wanted to restate that the sys-proclaim.lisp I sent for convenience
is output by the compiler, not a human.  It is in effect an optional
extra optimizing compiler pass, and its effect should therefore be
comparable to other optimizing instructions given to a compiler, but
not to code tuning by hand.

Take care,

> This is on an Athlon XP 2100+ system (512 MB), running Linux 2.6.0.
> 
> -Peter
> 
> 
> 

-- 
Camm Maguire                                            address@hidden
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah