>> Hi, I've found I want to run only one or a few of the tests at a
>> time rather than the whole suite.
>
http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00339.html
>
> is supposed to allow this. I don't know if it is waiting on
> something.
Yes indeed, it provides a way to run individual tests:
$ rm cl-lib.log ; make cl-lib.log
Running tests in cl-lib.el... passed all 8 tests
Some comments on the patch follow.
> TEST_LOGS = $(patsubst %.el, %.log, $(wildcard $(test)/*.el))
Other recipes in the same Makefile determine the set of .el files a
different way: they include .el files in subdirectories except data/.
There aren't actually such .el files, but the moment someone adds one
the make code is inconsistent.
> @test -d `dirname "$@"` || mkdir `dirname "$@"`
Why not: mkdir -p `dirname "$@"`
> parallel: $(TEST_LOGS)
> @cd $(test); $(emacs) -f ert-summary-report $(TEST_LOGS)
Instead of creating the new "parallel" target, could we just have the
"check" target run the tests individually?
One argument against might be that a -j1 build would be longer. Here
are some benchmarks (2 CPU cores). 3 different invocations, 2 samples
each:
$ rm *.log ; time make parallel
real 1m21.869s
real 1m21.918s
$ rm *.log ; time make -j4 parallel
real 1m2.816s
real 1m4.667s
$ time make check
real 1m17.989s
real 1m16.836s
(Note: the file-notify-tests alone take about 1min 1sec, which puts a
lower bound on 'time make -j4 parallel'.)
If however we keep the parallel target, it should be renamed. It seems
off to name a target "parallel" just because it is parallelizable. If
the user doesn't pass -j then the target name is technically
incorrect. "summary" would be a good name given what it does.
> (defun ert-run-tests-batch-and-exit-single ()
> [...]
> ;; Load a byte-compiled one or TEST-FILE itself.
> (if (file-newer-than-file-p compiled test-file)
> (progn
> (setq base (file-name-nondirectory compiled))
> (load-file compiled))
> (let ((buf (find-file-noselect test-file)))
> (if (with-current-buffer buf
> (and (boundp 'no-byte-compile) no-byte-compile))
> (with-current-buffer buf
> (eval-buffer))
> (if (byte-compile-file test-file t)
> (setq base (file-name-nondirectory compiled))
> (princ (format "%s failed to compile the file\n" prefix))
> (message "##REPORT:(compile-error \"%s\")##" base)
> (kill-emacs 0))))
Why shouldn't Make have compiled the test-file? Perhaps the log files
should depend on the .elc files instead of the .el files.
> (defun ert-run-tests-batch-and-exit-single ()
> [...]
> (message "##REPORT:(compile-error \"%s\")##" base)
> [...]
> (message "##REPORT:(done %d %d)##" total expected)
> [...]
> (message "##REPORT:(load-error \"%s\")##" base)
It seems the only reason to have ert-run-tests-batch-and-exit-single
is to insert these "##REPORT" tokens. But why can't ert-summary-report
parse:
'^Ran \([0-9]*\) tests, \([0-9]*\) results as expected'
to get the same information? Then could you remove the
ert-run-tests-batch-and-exit-single function and invoke the existing
ert-run-tests-batch-and-exit?
> (defun ert-summary-report ()
> [...]
> (when errors
> (message "\n Following test files have problems:")
When I ran the parallel target, I didn't get this message at all, even
though I have some test failures. eg from my file-notify-tests.log:
1 unexpected results:
FAILED file-notify-test00-availability
My stdout was:
Running tests in add-log-tests.el... passed all 4 tests
[...]
Running tests in vc-bzr.el... passed 0 tests out of 3
## Summary ##
Ran 441 tests, 420 results as expected, 21 unexpected
$