Re: how much are "historic" issues with test / [ still an issue

[Chet Ramey]

On 5/25/21 12:16 PM, Christoph Anton Mitterer wrote:
> Hey.

Hey.

> On Tue, 2021-05-25 at 11:04 -0400, Chet Ramey wrote:
> > The bash `test' implements the POSIX algorithm, which was first
> > published
> > in the 1003.2 drafts, so bash has provided a way to avoid those
> > problems
> > for at least 30 years.
>
> And AFAIU that's basically the determination based on the number of
> arguments given to test (i.e. without the ] in the [] case), as
> described in the operands section of:
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html
>
>
> That still has a number of "Otherwise, produce unspecified results.":

Two that matter: the four-argument case where the first argument is not `!'
or $1 and $4 aren't `(' and `)', respectively, and five or more arguments.

> I guess for the 2-arguments case, that would be if one writes some
> obvious error like
> test foo bar
> or
> test $var1 $var2
> with $var1 not being ! or any of the unary operators.

Yes, it encompasses error handling -- everyone performs error handling --
as well as providing implementations the flexibility to offer additional
unary operators beyond those listed in the standard. Many shells, including
bash, extend the set of unary operators.

> And the same for 3-arguments case, if one does e.g.:
> test $var1 "" $var2
> test $var1 -xx $var2
> or similar stuff.

As well as offer additional binary operators.

> I would have understood:
> > 3 arguments:
> >      If $2 is a binary primary, perform the binary test of $1 and $3.
> >      If $1 is '!', negate the two-argument test of $2 and $3.
> as:
> First, check whether $2 is any of the binary operators.
> Second, if $1 is !, negate the result of the 2-argument test from $2
> and $3.

That's saying the same thing, except you leave out the actual binary test.
The bash texinfo manual says

"The following conditions are applied in the order listed."

so you can assume things happen in that order. The POSIX text does not make
that guarantee, though it's implied.

> And that's also the reason why test ! = "" works, because it's not the
> negation of a = "" but the comparison of two strings.

Right. There is operator precedence within the specified N-argument cases.

> So what if I have >= 4 arguments... e.g.
> $var1 = $var2 -a $var3 -z $var4
>
> I guess that is then indeed a problem even in modern shells?

I'm not sure why anyone would ignore the guidance that recommends using
expressions composed of individual unary and binary tests and to use the
shell's conditional execution operators to join them. That's the reason
to specify the behavior based on the number of arguments. There is no
good reason -- certainly no portable reason -- to use more than four
arguments to a test command.

One of the reasons POSIX specified test the way it did was to resolve the
differences in the existing test algorithms (or, really, to avoid resolving
them).

> Does it help if one always uses parentheses so that no more than 3
> arguments are in one pair of parentheses?
> Like:
> \( $var1 = $var2 \) -a \( $var3 -z $var4 \)
>
> The algorithm doesn't really say much about these, so I'd guess at
> least per the standard these are prone to be ambiguous?

You run into the same variable quoting problems, at least. My guess is
that most historical test parsing algorithms would produce the same
result, but why bother?


> > > Even more importantly for me, cases like:
> > >      > test "$response" = "expected string"
> > >      shall be written as:
> > >      > test "X$response" = "Xexpected string"
> > >
> > > I guess the leading X is simply to "escape" any possible meta-
> > > character
> > > like ( ! or -, right ?
> >
> > Historically, yes.
>
> So at least in 3 argument case this is safe, and including "X" or
> similar not necessary.

Bash will always give binary primaries precedence over $1 == "!", so this
is safe.


> > You can rewrite as
> >
> >          test -d "$1" || test -d "$2"
> > or
> >          [[ -d "$1" || -d "$2" ]]
> >
> > and have the advantage of short-circuiting after the first test if it
> > ends
up being true.
> test's -a and -o don't short circuit?

They are not guaranteed to, no. You have to consume the arguments, at
least, when you are parsing and evaluating on the fly. Historical versions
of test don't build a parse tree and evaluate it; they parse and evaluate
in one pass.

> So long story short:
> - as long as one uses <= 3 arguments, things always works as expected,
>    even if $1 and $3 are variables with arbitrary values set by the user
>
> - 4 arguments still works if the first is !   or   if $1 and $4 are (
>    and )
>
> - anything else >= 4 might is ambiguous, and one should use the forms
> with multiple test(s) and combine those via ||, &&, and the shell's ( )
> (i.e. the unquoted parentheses that cause subshell evaluation).

That's a pretty decent summary of the POSIX guidance. The subshells can
throw you off, though, since the word expansions happen in the subshell
and some scripts might expect or need the side effects. In most cases,
you can use { and } instead of the subshells.

Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/