Re: [lmi] Benchmarking: gcc-8 beats gcc-10 soundly?

[Vadim Zeitlin]

On Sat, 19 Sep 2020 15:15:48 +0000 Greg Chicares <gchicares@sbcglobal.net> 
wrote:

GC> It looks like gcc-10 gives us slower lmi binaries. Picking
GC> the third '--selftest' scenario as an index of performance
GC> (results in microseconds--less is better):
GC> 
GC>      gcc-10   gcc-8  ratio
GC>      ------   -----  -----
GC>      102659   84947   1.21  32-bit
GC>       50121   37410   1.34  64-bit
GC> 
GC> The fourth scenario is even worse:
GC> 
GC>       33250   20654   1.61  32-bit
GC>       24616   13009   1.89  64-bit
GC> 
GC> Comparing to the benchmarks here:
GC>   
https://openbenchmarking.org/result/1912179-HU-COREI759632&obr_sgm=y&obr_vb=y&obr_sgm=y&obr_swl=y&obr_vb=y
GC> lmi fares worse than the worst phoronix scenarios, i.e.,
GC> "libgav1" and "function objects".
GC> 
GC> Vadim--Does this seem so astonishing that it can't be
GC> true?

 I've already seen performance regressions in newer g++ versions, but I
don't think I've seen anything nearly like 89% slowdown, so it's indeed
very astonishing. But I have trouble seeing how could it be not true, if
you consistently obtain such results. And you're not the only one, see e.g.
this bug report https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96337

 Unfortunately there is no clear conclusion there, as gcc developers can't
reproduce the problem. They do say that -O2 has been changed in 10.x, so it
could be worth using -O3 with it and see if it helps. Should I/we do it or
will you test this yourself?


 BTW, while looking at this, I had another question: could we add the
standard deviation or at least min/max time output to AliquotTimer? It
might show that the code has indeed become uniformly slower, or that there
are just more outliers. I don't know which scenario would actually be
worse, but it looks like it would be useful to know which one are we
dealing with. If you think this could be useful, I could make a patch
adding this, of course -- just please let me know.

 Thanks,
VZ

pgpVUTgQagYxh.pgp
Description: PGP signature