Re: Time to merge scratch/correct-warning-pos into master, perhaps?

[Alan Mackenzie]

Hello, Gregory.

On Wed, Jan 26, 2022 at 18:41:57 +0000, Gregory Heytings wrote:


> > I think we need more measurements in scenarios closer to actual Emacs 
> > usage.  One is intensive display operation -- I think Alan posted 
> > something to that effect.  Another could be starting Gnus to read a 
> > large newsgroup.  Yet another could be reindent a large piece of C or 
> > Python or Lisp code.  Perhaps also "M-x occur" through a large buffer. 
> > Stuff like that -- any command that tends to be used frequently and is 
> > known to take a tangible amount of time.

> The problem is that, IME, such benchmarks do not show anything when the 
> slowdown is not significant enough (> 10%), because the measurements vary 
> a lot depending on external factors.  I wonder how Alan could conclude 
> that there is a slowdown of "less than 1%", when I run his benchmark on my 
> (otherwise unloaded) computer, the running times I get are anywhere 
> between 17s and 20s.  Taking the average of such measures does not really 
> make sense.

No, but the first run in an Emacs session tends to be shorter than
subsequent runs.  I don't know why.  I take it you also saw a very small
difference in the timings for this benchmark between the two versions.
Otherwise you would have said, I think.

The four measurements I got from my benchmark in master (from
benchmark-run) were:
      o - (20.798601883 460 7.728701306)
      o - (20.947118356 295 7.1172684969999995)
      o - (20.941589929 293 7.144901186)
      o - (20.917180235 293 7.136285445000002)
The corresponding four from the (then) branch were
      o - (20.854543266 480 7.691123986)
      o - (21.064465459 320 7.189660959000001)
      o - (21.143813105 318 7.287708998000001)
      o - (21.115932422 318 7.266432223999999)
It is fairly clear the second set are ~1% more than the first set, give
or take unimportant measurement errors.

That's precisely what I meant when I said that the branch was about 1%
slower ON THAT BENCHMARK.  Please note what I said, going back to my
original post if needed.  I did not say "less than" 1%.  I was not
referring to the speed in general.  I was very specific about what I
wrote.

> However, I found one measurement that seems to give reasonably stable 
> results.  On my Debian bookworm computer, with a standard build, on 
> src/xdisp.c, (benchmark-run 100 (occur "a.b")) needs on average (with 10 
> runs) 7.42 seconds (standard deviation 0.04) on 3b33a14380 and 7.66 
> seconds (standard deviation 0.05) on 7922131bb2.  That's a >3% slowdown.

Yes, that's around 3.2%.  Scarcely something you'd notice in an
interactive session, where the difference is around the time it takes to
depress a key on the keyboard.

So we now have a real life measurement which is more than 1%.  It's
nowhere near 10%, though.

-- 
Alan Mackenzie (Nuremberg, Germany).