Re: ert buffer isolation

[Daniil Iaitskov]

Emacs predates unit tests practices, but the system should follow trends in the industry.

I anticipate that tests in Emacs might interfere other ways, but I did have a chance and I don't have the context to write shim covering these cases.

I found an article by Kent Beck (https://en.wikipedia.org/wiki/Kent_Beck), who wrote a few popular books about testing software.

https://tidyfirst.substack.com/p/desirable-unit-tests

First 2 bullets on the list:

• Isolated — Unit tests are completely isolated from each other, creating their own test fixtures from scratch each time. (Note that I’m not saying these are the only useful tests, just that if tests aren’t isolated you’re going to have a hard time making the case that they are “unit tests”.)

• Composable — Follows from isolation.

I published ert-scope (https://github.com/yaitskov/ert-scope) package with a shim to handle scope issue for buffers and files in sync and async tests.

(require 'ert)
(require 'ert-scope)

(ert-deftest sync-test ()
  (ert-scope-with-temp-dir
    (ert-scope-buffers
      (should-not (get-buffer "foo.txt"))
      (find-file "foo.txt")
      (should (= 1 (point-max)))
      (insert "foo")
      (save-buffer))))

On 1/26/25 07:26, Eli Zaretskii wrote:

> > From: Michael Albinus <michael.albinus@gmx.de>
> > Cc: emacs-devel@gnu.org
> > Date: Sun, 26 Jan 2025 17:12:24 +0100
> >
> > Daniil Iaitskov <dyaitskov@gmail.com> writes:
> >
> > > I wrote a few tests with ert package and noticed that buffers opened
> > > inside ert-deftest
> > >
> > > escape the test scope and stay alive.
> > >
> > > It affects other tests e.g. find-file creates a buffer with a suffixed
> > > name if a buffer
> > >
> > > with name of file is already exist.
> > >
> > > I spent more time on debugging because of this.
> > >
> > > Test guidelines recommend tests should not influence each other.
> > >
> > > So I suggest by default to wrap test body of ert-deftest with:
> > >
> > > (defun scope-buffers (cb)
> > >   "Kill all buffers created by CB."
> > >   (let ((existed-buffers (buffer-list)))
> > >     (unwind-protect (funcall cb)
> > >       (mapc (lambda (b)
> > >               (when (not (cl-find b existed-buffers))
> > >                 (message "Kill buffer [%s]" b)
> > >                 (kill-buffer b)))
> > >             (buffer-list)))))
> > >         
> > I disagree. ERT test writers shall care of cleanup, so a recommended
> > template is
> >
> > --8<---------------cut here---------------start------------->8---
> > (ert-deftest name ()
> >   "Doc."
> >   (let (buffer tmp-file)
> >     (unwind-protect
> > 	(progn BODY)
> >       ;; Cleanup.
> >       (ignore-errors (kill-buffer buffer))
> >       (ignore-errors (delete-file tmp-file)))))
> > --8<---------------cut here---------------end--------------->8---
> >       
> I tend to agree.  It happens quite a lot that several tests reuse the
> same buffers, so removing them as part of ert-deftest would be an
> annoyance.  Besides, the effect of a test is much more than just
> buffers and files it leaves behind: it's modes it activates, features
> it loads, variables it changes, etc.  And those are impossible to undo
> in general-purpose code.
>
> > Alternatively, we could extend ert-kill-all-test-buffers from ert-x.el
> > (or write a new function), which would be called in the Cleanup section.
> >       
> Ideally, such a function would do more than just kill buffers.
>