Re: [lmi] Benchmarking: gcc-8 beats gcc-10 soundly?

[Greg Chicares]

On 2020-09-19 15:48, Vadim Zeitlin wrote:
> On Sat, 19 Sep 2020 15:15:48 +0000 Greg Chicares <gchicares@sbcglobal.net> 
> wrote:
> 
> GC> It looks like gcc-10 gives us slower lmi binaries. Picking
> GC> the third '--selftest' scenario as an index of performance
> GC> (results in microseconds--less is better):
> GC> 
> GC>      gcc-10   gcc-8  ratio
> GC>      ------   -----  -----
> GC>      102659   84947   1.21  32-bit
> GC>       50121   37410   1.34  64-bit
> GC> 
> GC> The fourth scenario is even worse:
> GC> 
> GC>       33250   20654   1.61  32-bit
> GC>       24616   13009   1.89  64-bit
> GC> 
> GC> Comparing to the benchmarks here:
> GC>   
> https://openbenchmarking.org/result/1912179-HU-COREI759632&obr_sgm=y&obr_vb=y&obr_sgm=y&obr_swl=y&obr_vb=y
> GC> lmi fares worse than the worst phoronix scenarios, i.e.,
> GC> "libgav1" and "function objects".
> GC> 
> GC> Vadim--Does this seem so astonishing that it can't be
> GC> true?
> 
>  I've already seen performance regressions in newer g++ versions, but I
> don't think I've seen anything nearly like 89% slowdown, so it's indeed
> very astonishing. But I have trouble seeing how could it be not true, if
> you consistently obtain such results. And you're not the only one, see e.g.
> this bug report https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96337

Thanks, I hadn't searched bugzilla.

>  Unfortunately there is no clear conclusion there, as gcc developers can't
> reproduce the problem. They do say that -O2 has been changed in 10.x, so it
> could be worth using -O3 with it and see if it helps. Should I/we do it or
> will you test this yourself?

I already did it for 8.x, which needs
  -Wno-vector-operation-performance
(because of -Werror); it didn't make much difference there.
I'll try it with 10.x soon.

>  BTW, while looking at this, I had another question: could we add the
> standard deviation or at least min/max time output to AliquotTimer? It
> might show that the code has indeed become uniformly slower, or that there
> are just more outliers. I don't know which scenario would actually be
> worse, but it looks like it would be useful to know which one are we
> dealing with. If you think this could be useful, I could make a patch
> adding this, of course -- just please let me know.

Sure, go right ahead. I have no objection to showing σ.
I'll probably want to twiddle the output format to make
it really compact, so that we can fit more information
on one line--very roughly, something like this:

-  1.323e-02 s mean;      13009 us least of  76 runs
+  μ=1.32e-02 σ=9.87e-01 min=13009us runs=76

which leaves room if you want to throw in kurtosis or
whatever. But let me worry about formatting, because
it's the math stuff that I think you really want to do
(Welford's algorithm or whatever) and a patch that
just adds some code to do those calculations is easy
to accept because it won't have any perceptible effect
until the format is changed to print it.