bug#69718: 29.2; grep japanese-iso-8bit-unix string fails with "grep: illegal byte sequence"

[Akira Shirai]

If LANG=locale grep [-e pattern] [file ...] is specified, macOS grep checks the 
consistency between the locale and the byte sequence of the pattern,
but does not check the consistency between the locale and the byte sequence of 
the file contents.
(Please see *1 below)

This consistency check feature can be disabled if LANG=C is specified or LANG 
is not specified.

emacs-29.1 executes /usr/bin/grep without LANG (*2),
but emacs-29.2 seems to execute /usr/bin/grep with LANG=en_US.UTF-8 (*3).

I wonder that in some cases emacs should invoke subprocesses with LANG=C is 
specified or LANG is not specified,
and this grep issue might be one of the case.

----------------------------------------------------------------------
*1 If LANG=locale grep [-e pattern] [file ...] is specified, macOS grep checks 
the consistency between the locale and the byte sequence of the pattern,
but does not check the consistency between the locale and the byte sequence of 
the file contents.

% cd ~/emacs-29.2/leim/SKK-DIC

/*
 * locale (= $LANG): en_US.UTF-8
 * pattern (= $aaa): UTF-8
 * file:             eucJP
 *   => Though the encodings of locale and file are inconsistent, grep executes 
successfully without any hit
 */
% export aaa=この辞書は; export bbb=`echo $aaa | iconv -f utf-8 -t eucJP`; 
LANG=en_US.UTF-8 grep --color=auto -nH --null -e $aaa SKK-JISYO.L | iconv -f 
eucJP -t utf-8

/*
 * locale (= $LANG): en_US.UTF-8
 * pattern (= $bbb): eucJP
 * file:             eucJP
 *   => Because the encodings of locale and pattern are inconsistent, grep 
fails with illegal byte sequence
 */
% export aaa=この辞書は; export bbb=`echo $aaa | iconv -f utf-8 -t eucJP`; 
LANG=en_US.UTF-8 grep --color=auto -nH --null -e $bbb SKK-JISYO.L | iconv -f 
eucJP -t utf-8
grep: illegal byte sequence

/*
 * locale (= $LANG): C
 * pattern (= $bbb): eucJP
 * file:             eucJP
 *   => Because the encodings of locale is C, grep does not check the 
consistency between the locale and the byte sequence of the pattern and 
executes successfully with one hit
 */
% export aaa=この辞書は; export bbb=`echo $aaa | iconv -f utf-8 -t eucJP`; LANG=C 
grep --color=auto -nH --null -e $bbb SKK-JISYO.L | iconv -f eucJP -t utf-8
SKK-JISYO.L35:;; この辞書は、SKK 原作者の佐藤雅彦先生が、第 1 版作成のために東北大学

/*
 * locale (= $LANG): unspecified
 * pattern (= $bbb): eucJP
 * file:             eucJP
 *   => Because the encodings of locale is not specified, grep does not check 
the consistency between the locale and the byte sequence of the pattern and 
executes successfully with one hit
 */
% export aaa=この辞書は; export bbb=`echo $aaa | iconv -f utf-8 -t eucJP`; LANG= 
grep --color=auto -nH --null -e $bbb SKK-JISYO.L | iconv -f eucJP -t utf-8
SKK-JISYO.L35:;; この辞書は、SKK 原作者の佐藤雅彦先生が、第 1 版作成のために東北大学

----------------------------------------------------------------------
*2 emacs-29.1 executes /usr/bin/grep without LANG

/*
 * emacs-29.1
 * grep executes successfully with one hit
 * printenv shows that the subprocess is invokeed with LANG unspecified
 */

-*- mode: grep; default-directory: "~/emacs-29.1/leim/SKK-DIC/" -*-
Grep started at Wed Mar 13 22:21:06

grep --color=auto -nH --null -e この辞書は SKK-JISYO.L; printenv
SKK-JISYO.L:35:;; この辞書は、SKK 原作者の佐藤雅彦先生が、第 1 版作成のために東北大学
GREP_COLOR=01;31
SHELL=/bin/bash
TERM=emacs-grep
TMPDIR=/var/folders/4l/q0w9w6j914q2n_v7qyysbrhh0000gn/T/
USER=shiraiakira
COMMAND_MODE=unix2003
GREP_COLORS=mt=01;31:fn=:ln=:bn=:se=:sl=:cx=:ne
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.yfsSXeLGOX/Listeners
__CF_USER_TEXT_ENCODING=0x1F5:0x1:0xE
PAGER=
PATH=/usr/bin:/bin:/usr/sbin:/sbin
LaunchInstanceID=8A9BF0A3-3EF0-4A5E-B1A2-E3E4FE729A53
__CFBundleIdentifier=org.gnu.Emacs
PWD=/Users/shiraiakira/emacs-29.1/leim/SKK-DIC
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
SHLVL=1
HOME=/Users/shiraiakira
LOGNAME=shiraiakira
DISPLAY=F9A6231BCF26.local
INSIDE_EMACS=29.1,compile
SECURITYSESSIONID=186a2
_=/usr/bin/printenv

Grep finished with 1 match found at Wed Mar 13 22:21:06

----------------------------------------------------------------------
*3 emacs-29.2 seems to execute /usr/bin/grep with LANG=en_US.UTF-8

/*
 * emacs-29.2
 * grep fails with illegal byte sequence
 * printenv shows that the subprocess is invoked with LANG=en_US.UTF-8
 */

-*- mode: grep; default-directory: "~/emacs-29.2/leim/SKK-DIC/" -*-
Grep started at Wed Mar 13 22:19:25

grep --color=auto -nH --null -e この辞書は SKK-JISYO.L; printenv
grep: illegal byte sequence
GREP_COLOR=01;31
SHELL=/bin/bash
TERM=emacs-grep
TMPDIR=/var/folders/4l/q0w9w6j914q2n_v7qyysbrhh0000gn/T/
USER=shiraiakira
COMMAND_MODE=unix2003
GREP_COLORS=mt=01;31:fn=:ln=:bn=:se=:sl=:cx=:ne
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.yfsSXeLGOX/Listeners
__CF_USER_TEXT_ENCODING=0x1F5:0x1:0xE
PAGER=
PATH=/usr/bin:/bin:/usr/sbin:/sbin
LaunchInstanceID=8A9BF0A3-3EF0-4A5E-B1A2-E3E4FE729A53
__CFBundleIdentifier=org.gnu.Emacs
PWD=/Users/shiraiakira/emacs-29.2/leim/SKK-DIC
LANG=en_US.UTF-8
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
SHLVL=1
HOME=/Users/shiraiakira
LOGNAME=shiraiakira
DISPLAY=F9A6231BCF26.local
INSIDE_EMACS=29.2,compile
SECURITYSESSIONID=186a2
_=/usr/bin/printenv

Grep finished with matches found at Wed Mar 13 22:19:25


> 2024/03/13 4:39、Eli Zaretskii <eliz@gnu.org>のメール:
> 
>> From: Akira Shirai <okshirai@gmail.com>
>> Date: Wed, 13 Mar 2024 00:42:22 +0900
>> Cc: 69718@debbugs.gnu.org,
>> okshirai@joy.ocn.ne.jp
>> 
>> emacs-29.1 executes /usr/bin/grep without LANG,
>> but emacs-29.2 seems to execute /usr/bin/grep with LANG=en_US.UTF-8.
> 
> The fact that LANG could cause this is IMO a bug in macOS's Grep.
> 
> There's no problem for Emacs to put LANG into the environment, but
> Grep can be invoked on several very different files, with no single
> LANG that fits all of them.  Grep should not use LANG at all.
> 
>> I wonder /usr/bin/grep should be invoked in non UTF-8 mode, because emacs 
>> might pass non UTF-8 byte sequence to /usr/bin/grep.
> 
> Illegal byte sequence is not limited to UTF-8.  There really is no
> good solution for this, except in Grep itself.  Which is why I don't
> think this is an Emacs bug.