Re: Incorrect alias expansion within command substitution

[Chet Ramey]

On 2/1/22 1:47 PM, Robert Elz wrote:
>      Date:        Tue, 1 Feb 2022 10:23:53 -0500
>      From:        Chet Ramey <chet.ramey@case.edu>
>      Message-ID:  <1e33f111-b9ff-2d70-adf8-934906321ac6@case.edu>
>
>    | Historically, bash (and ksh93) has favored the former. Just about all the
>    | other shells claiming some sort of POSIX conformance favor the latter (all
>    | the ash-based shells, yash, mksh).
>
> For ash bashed shells, at least, that's because the text in the command
> substitution is not parsed twice - it is parsed (nb: just parsed, not expanded
> or evaluated) once while scanning it to find the closing ')', and the results
> of that parse are simply kept for later when it is to be used.

When you say "just parsed," when are aliases expanded? Are they expanded
while scanning the command substitution to find the closing `)' but not
part of the text that results? What form are the `results' kept in (I
assume a parse tree similar to a shell function)? Do they include expanded
aliases or is that deferred?

> Since it isn't possible for anything to change the command substitution text
> once scanning it has started, the results of a parse of its text done later
> must be the same as that done while scanning (that is, no commands can
> possibly be executed between when that scan starts, and when the command
> substitution is later expanded, assuming it ever is) so doing the parse all
> over again is something of a waste of time. 

That seems like the crux of the issue. If the command substitution is part
of a shell function definition, you only want to expand aliases the `first
time' -- at the time you parse the shell function. You can execute 
arbitrary commands, including alias definitions, between the time the shell
function is defined and the time it's executed. POSIX requires that aliases
in command substitutions be expanded when the function definition is
parsed, not when the command substitution is finally executed, but bash has
not traditionally done it that way. That's where the backwards
compatibility issues come in.


> Since  the command substitution
> might never be expanded however, some bookkeeping is needed to make sure that
> the resulting parse tree is discarded if it isn't used - but compared to the
> work needed for an additional parse, that's trivial.

Yes, you can carry it around as part of a data structure.

> But if you're sure that the double parse method is needed, the simple fix
> would be to save original command substitution text, not the alias expanded
> form of it, in the word being constructed, and simply continue to expand the
> alias both times (as above, since no commands can get executed between the
> first and second parse, assuming the 2nd happens, the alias cannot be altered,
> and it will expand the 2nd time the same way it did the first).

It's not true that `no commands can be executed'. The alias can be altered
by commands between a shell function definition and its execution.

I considered keeping both the original text and the parse tree from the
parsed command substitution (well, a chain of them since you can have an
arbitrary number of command substitutions in a word). It's difficult,
given bash's internal structure, to preserve the original text -- as
opposed to the reconstituted text -- so I didn't do that. Other shells
(bosh, mksh) also recreate the text of a command substitution from the
parsed commands.

> It seems to me that the way you're doing it now would also break:
>
>         alias x=y
>         cat <<$(x)
>         whatever
>         $(x)
>
> which is supposed to work, and indeed, in the bash 5.2 development branch,
> it does (the terminator after the here-doc text needs to be $(y) to work).

This works in bash default mode because aliases aren't expanded while the
command is parsed. The delimiter ends up being `$(x)'. Since you're
required to check the line read for the terminating delimiter before doing
anything else, the delimiter has to be $(x) to make it work.

It works with ksh93 as well, but every other shell produces an error of
some sort (including the NetBSD sh, unless you've changed something in the
couple of months since I last built it).

I can't see it working in any shell's posix mode if posix requires aliases
to be expanded while reading the WORD containing the command substitution
that is the here-doc delimiter. If that's the case, you have an alias
expansion mismatch, since I don't believe you're permitted to perform
alias expansion on the lines of the here-document as you read them, and
the resolution to bug 1036 makes it clear -- to me, at least -- that you
check for the delimiter before doing anything to the line.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/