Re: dired-do-find-regexp failure with latin-1 encoding

[Dmitry Gutov]

On 28.11.2020 22:29, Eli Zaretskii wrote:

> > Ah, so this way the user explicitly searches for a regexp encoded as
> > latin-1?
>
> More accurately, this is how to search in files encoded in Latin-1.
> (The regexp also gets encoded in latin-1, but the important part is
> the files' encoding.)

Right. So when there are files in different encodings, the result will 
be not great, as expected.

> > > Adding -a probably cannot do any harm, but its support should be
> > > detected, since I don't think it's portable enough (it isn't in the
> > > latest Posix spec, at least).
> >
> > Are you sure about that? Are we sure it won't make searching binary
> > files slower, for example?
>
> It will be slower, but more useful: by default Grep just says "Binary
> file foo matches".

Do we want to search the "binary" files at all? Right now we simply 
filter such matches out (see the definition of xref-matches-in-files), 
and I have seen no complaints.

> > Also, the manual has this warning:
> >
> > Warning: The -a option might output  binary  garbage,  which  can  have
> > nasty  side effects if the output is a terminal and if the terminal
> > driver interprets some of it as commands.
> >
> > ...which might conceivably mess up our parsing of Grep output sometimes?
>
> This is not relevant, since we read that output, there's no terminal
> device driver to interpret it and get messed up.

Our interpreter is our regexp with which we parse. But I suppose as long 
as Grep doesn't insert unexpected newlines, the parser will be fine.

> I actually don't think I understand why we need -a in this case, since
> Grep looks for null bytes to decide this is a binary file, and encoded
> non-ASCII characters don't have null bytes 9except if they are in
> UTF-16).

Good question.

> > P.S. Or we can forgo all that and ask the users who want to search for
> > non-ASCII strings to install ripgrep.
>
> We should support Grep regardless, since not everyone will have
> ripgrep.  And in any case, "C-x RET c" will be needed with it as well,
> no?

I'd have to test it explicitly to say for sure, but:

  ripgrep supports searching files in text encodings other than UTF-8,
  such as UTF-16, latin-1, GBK, EUC-JP, Shift_JIS and more. (Some
  support for automatically detecting UTF-16 is provided. Other text
  encodings must be specifically specified with the -E/--encoding flag.)

https://blog.burntsushi.net/ripgrep/#pitch

So if the file encoding is UTF-8, UTF-16, or latin-1 (AND the current 
system locale matches that encoding), the search should work fine across 
such files in different encodings, and without 'C-x RET c'.

Which doesn't cover all situations, of course, but it's about as much as 
can be expected. And more than Grep can.