Re: New "make benchmark" target

[Pip Cet]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> Date: Mon, 30 Dec 2024 15:00:29 +0000
>> From: Pip Cet <pipcet@protonmail.com>
>> Cc: acorallo@gnu.org, stefankangas@gmail.com, mattiase@acm.org, 
>> eggert@cs.ucla.edu, emacs-devel@gnu.org, joaotavora@gmail.com
>>
>> "Eli Zaretskii" <eliz@gnu.org> writes:
>>
>> >> Date: Mon, 30 Dec 2024 11:45:36 +0000
>> >> From: Pip Cet <pipcet@protonmail.com>
>> >> Cc: Stefan Kangas <stefankangas@gmail.com>, Eli Zaretskii
>> >> <eliz@gnu.org>, Mattias Engdegård <mattiase@acm.org>, Paul Eggert
>> >> <eggert@cs.ucla.edu>, emacs-devel@gnu.org, João Távora
>> >> <joaotavora@gmail.com>
>> >>
>> >> "Andrea Corallo" <acorallo@gnu.org> writes:
>> >>
>> >> > I'd personally drop the requirement of using ERT as a framework for
>> >> > benchmarks, I'd just move elisp-benchmarks code in emacs core and add
>> >> > the target.
>> >>
>> >> Well, as is obvious from the quoted paragraph, I disagree.  I've stated
>> >> why in the thread; if someone wants a summary, I can provide one
>> >
>> > Can you point to the message where you explained your rationale for
>> > using ERT for this?  I've scanned the discussion, but couldn't find
>> > such a message.
>>
>> The best I can find is this:
>>
>> https://lists.gnu.org/archive/html/emacs-devel/2024-12/msg00595.html
>
> Thanks, but AFAICT this just says that you intended to use/extend ERT
> to run this benchmark suite, but doesn't explain why you think using
> ERT would be an advantage worthy of keeping.

I think some advantages are stated in that email: the ERT tagging
mechanism is more general, works, and can be extended (I describe one
such extension).  All that isn't currently true for elisp-benchmarks.

The other big difference is resource management, which elisp-benchmarks
does via a global variable, reusing one test as data for another.  ERT
has a somewhat better mechanism.

>> I can try to provide a more detailed/structured rationale if that's
>> helpful.  (Is it, though?  Reusing someone's code in a way which reduces
>> their user base and might cause them more work isn't something we should
>> do lightly.)
>
> I'm not sure I follow.  Andrea suggests to move elisp-benchmarks into
> the repository, and add a target to the test/ Makefile to run it.
> AFAIU he suggested that because it should be less work, not more.

I was saying that my proposal would cause Andrea more work, because we'd
reuse his code in a way which reduces (splits) the elisp-benchmarks user
base.  It would also cause me work, which I've done, but that shouldn't
really count against it :-)

> Why do you think it is wrong to do the (AFAIU) simple change that
> Andrea proposed?

Because it's a de facto commitment to not doing it in ERT.  Having two
parallel benchmark suites isn't something I think would happen (and
which one would we use for our make target?)

> The reason I'm asking is because I think we want this suite to be part
> of our test, but don't necessarily want the addition of the benchmarks
> to the test suite be a large job that complicates the benchmarks and
> the test suite alike.

It's not a large job if we make it a clean split.  However, it would be
work for those preferring elisp-benchmarks conventions.

It might be relevant that elisp-benchmarks hasn't seen very active
development lately.  I think switching to ERT might help there, if only
because of the mailing list traffic.

Pip