bug#52429: 29.0.50; Flymake tests failing on macOS

[Philipp]

On my macOS Monterey machine, the Flymake tests partially fail, both on
the emacs-28 and the master branch.  Log follows:

Running 9 tests (2021-12-11 14:25:11+0100, selector `(not (or (tag 
:expensive-test) (tag :unstable) (tag :nativecomp)))')
Test different-diagnostic-types backtrace:
  signal(ert-test-failed (((should (eq 'flymake-error (face-at-point))
  ert-fail(((should (eq 'flymake-error (face-at-point))) :form (eq fly
  #f(compiled-function () #<bytecode -0x42652cc1fc0e76>)()
  flymake-tests--call-with-fixture(#f(compiled-function () #<bytecode 
  #f(compiled-function () #<bytecode 0x1be355267f05c7e8>)()
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name different-diagnostic-types :documenta
  ert-run-or-rerun-test(#s(ert--stats :selector (not (or ... ... ...))
  ert-run-tests((not (or (tag :expensive-test) (tag :unstable) (tag :n
  ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable) (
  ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
  eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
  command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/progmodes/flymake-te
  command-line()
  normal-top-level()
Test different-diagnostic-types condition:
    (ert-test-failed
     ((should
       (eq 'flymake-error
            (face-at-point)))
      :form
      (eq flymake-error flymake-warning)
      :value nil))
   FAILED  1/9  different-diagnostic-types (1.273652 sec)
   passed  2/9  dummy-backends (1.009144 sec)
   passed  3/9  eob-region-and-trailing-newline (0.000464 sec)
Test included-c-header-files backtrace:
  signal(ert-test-failed (((should (eq 'flymake-error (face-at-point))
  ert-fail(((should (eq 'flymake-error (face-at-point))) :form (eq fly
  #f(compiled-function () #<bytecode 0x6f0882b3ab2cac0>)()
  flymake-tests--call-with-fixture(#f(compiled-function () #<bytecode 
  #f(compiled-function () #<bytecode 0x9ee9419d23f2474>)()
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name included-c-header-files :documentatio
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :expensive-test) (tag :unstable) (tag :n
  ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable) (
  ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
  eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
  command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/progmodes/flymake-te
  command-line()
  normal-top-level()
Test included-c-header-files condition:
    (ert-test-failed
     ((should
       (eq 'flymake-error
            (face-at-point)))
      :form
      (eq flymake-error flymake-warning)
      :value nil))
   FAILED  4/9  included-c-header-files (1.091255 sec)
   passed  5/9  perl-backend (1.032770 sec)
   passed  6/9  recurrent-backend (1.031638 sec)
   passed  7/9  ruby-backend (1.174518 sec)
   passed  8/9  warning-predicate-function-gcc (1.045177 sec)
   passed  9/9  warning-predicate-rx-gcc (1.089723 sec)

Ran 9 tests, 7 results as expected, 2 unexpected (2021-12-11 14:25:20+0100, 
9.383207 sec)

2 unexpected results:
   FAILED  different-diagnostic-types
   FAILED  included-c-header-files

I haven't dug deeper, but this looks like the compiler returns a warning
where an error is expected.  Typically such tests should mock the
compiler using canned output checked into the repository.


In GNU Emacs 29.0.50 (build 25, aarch64-apple-darwin21.1.0, NS appkit-2113.00 
Version 12.0.1 (Build 21A559))
 of 2021-12-11
Repository revision: 8c50016b100ec2c548ec90131e0f5fb5f4ebb5c1
Repository branch: master
Windowing system distributor 'Apple', version 10.3.2113
System Description:  macOS 12.0.1

Configured using:
 'configure --with-modules --without-xml2 --without-pop --with-mailutils
 --enable-gcc-warnings=warn-only --enable-checking=all
 --enable-check-lisp-object-type '--prefix=/tmp/\7''

Configured features:
ACL GNUTLS JSON LCMS2 MODULES NOTIFY KQUEUE NS PDUMPER PNG SQLITE3
THREADS TOOLKIT_SCROLL_BARS ZLIB

Important settings:
  value of $LANG: de_DE.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message yank-media rmc dired
dired-loaddefs rfc822 mml mml-sec epa epg rfc6068 epg-config gnus-util
rmail rmail-loaddefs time-date mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mail-utils phst skeleton derived pcase
ffap thingatpt url url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util url-parse auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json map url-vars
mailcap rx gnutls puny dbus xml seq gv subr-x byte-opt bytecomp
byte-compile cconv compile text-property-search comint ansi-color ring
cl-loaddefs cl-lib iso-transl tooltip eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/ns-win ns-win
ucs-normalize mule-util term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice
button loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads kqueue cocoa ns lcms2
multi-tty make-network-process emacs)

Memory information:
((conses 16 70005 6853)
 (symbols 48 8477 1)
 (strings 32 23896 1586)
 (string-bytes 1 779222)
 (vectors 16 16571)
 (vector-slots 8 214734 10495)
 (floats 8 27 36)
 (intervals 56 225 0)
 (buffers 992 10))