Re: [lmi] Benchmarking: gcc-8 beats gcc-10 soundly?

[Vadim Zeitlin]

On Sat, 19 Sep 2020 20:37:59 +0000 Greg Chicares <gchicares@sbcglobal.net> 
wrote:

GC> On 2020-09-19 15:48, Vadim Zeitlin wrote:
GC> > On Sat, 19 Sep 2020 15:15:48 +0000 Greg Chicares 
<gchicares@sbcglobal.net> wrote:
GC> > 
GC> > GC> It looks like gcc-10 gives us slower lmi binaries. Picking
GC> > GC> the third '--selftest' scenario as an index of performance
GC> > GC> (results in microseconds--less is better):
GC> > GC> 
GC> > GC>      gcc-10   gcc-8  ratio
GC> > GC>      ------   -----  -----
GC> > GC>      102659   84947   1.21  32-bit
GC> > GC>       50121   37410   1.34  64-bit
GC> > GC> 
GC> > GC> The fourth scenario is even worse:
GC> > GC> 
GC> > GC>       33250   20654   1.61  32-bit
GC> > GC>       24616   13009   1.89  64-bit
GC> 
GC> With -O3, the 64-bit build performs thus on those two scenarios:
GC>   naic, ee prem solve : 5.001e-02 s mean;      49710 us least of  20 runs
GC>   finra, no solve     : 2.483e-02 s mean;      24580 us least of  41 runs
GC> Thus, the -O3 to -O2 speed ratio is
GC>   49710 / 50121 = .992
GC>   24580 / 24616 = .999
GC> which isn't work the extra build time (82.89 vs 72.76 seconds).

 OK, it wouldn't be interesting if it were as simple as that...

GC> >  I've already seen performance regressions in newer g++ versions, but I
GC> > don't think I've seen anything nearly like 89% slowdown, so it's indeed
GC> > very astonishing. But I have trouble seeing how could it be not true, if
GC> > you consistently obtain such results. And you're not the only one, see 
e.g.
GC> > this bug report https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96337
[...]
GC> >  Unfortunately there is no clear conclusion there, as gcc developers can't
GC> > reproduce the problem.
GC> 
GC> It seems really strange that they would say that. I guess
GC> phoronix is just one guy, but he seems to be a serious person
GC> with a serious audience.

 I think they must have discovered some problem in their tests, which is
not described in Bugzilla comments. As you might have noticed, the bug just
got closed, probably because I reminded them about its existence when I
subscribed to it (which generated a notification email), but they do ask to
open a new bug if we can provide a test case showing the regression.

GC> > They do say that -O2 has been changed in 10.x, so it
GC> > could be worth using -O3 with it and see if it helps. Should I/we do it or
GC> > will you test this yourself?
GC> 
GC> We seem to have a test case that should be reproducible,
GC> though it's far from ideally minimal.

 Yes, I don't think it would be such a great idea to submit entire lmi to
gcc Bugzilla. So I think we should do the following:

1. Check if we can see the regression under Linux too because:
 (a) I think gcc developers would pay more attention to something which
     is not MSW/MinGW-specific.
 (b) We could use perf under Linux and maybe find some smoking gun without
     doing anything else and, if all else fails, they did say that we could
     submit perf results even if we can't reduce the test case.

2. Spend some time trying to produce a reasonably small test case.

 Would you approve of this plan?
VZ

pgp9k65jYDZaE.pgp
Description: PGP signature