Re: ert buffer isolation

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: emacs-devel@gnu.org
> Date: Sun, 26 Jan 2025 17:12:24 +0100
> 
> Daniil Iaitskov <dyaitskov@gmail.com> writes:
> 
> > I wrote a few tests with ert package and noticed that buffers opened
> > inside ert-deftest
> >
> > escape the test scope and stay alive.
> >
> > It affects other tests e.g. find-file creates a buffer with a suffixed
> > name if a buffer
> >
> > with name of file is already exist.
> >
> > I spent more time on debugging because of this.
> >
> > Test guidelines recommend tests should not influence each other.
> >
> > So I suggest by default to wrap test body of ert-deftest with:
> >
> > (defun scope-buffers (cb)
> >   "Kill all buffers created by CB."
> >   (let ((existed-buffers (buffer-list)))
> >     (unwind-protect (funcall cb)
> >       (mapc (lambda (b)
> >               (when (not (cl-find b existed-buffers))
> >                 (message "Kill buffer [%s]" b)
> >                 (kill-buffer b)))
> >             (buffer-list)))))
> 
> I disagree. ERT test writers shall care of cleanup, so a recommended
> template is
> 
> --8<---------------cut here---------------start------------->8---
> (ert-deftest name ()
>   "Doc."
>   (let (buffer tmp-file)
>     (unwind-protect
>       (progn BODY)
>       ;; Cleanup.
>       (ignore-errors (kill-buffer buffer))
>       (ignore-errors (delete-file tmp-file)))))
> --8<---------------cut here---------------end--------------->8---

I tend to agree.  It happens quite a lot that several tests reuse the
same buffers, so removing them as part of ert-deftest would be an
annoyance.  Besides, the effect of a test is much more than just
buffers and files it leaves behind: it's modes it activates, features
it loads, variables it changes, etc.  And those are impossible to undo
in general-purpose code.

> Alternatively, we could extend ert-kill-all-test-buffers from ert-x.el
> (or write a new function), which would be called in the Cleanup section.

Ideally, such a function would do more than just kill buffers.