Re: smie-next-sexp vs associative operators

[Stephen Leake]

Stefan Monnier <address@hidden> writes:

>> I'm indenting "accept LILLY".  It should be indented relative to
>> "select".
>
> That's odd.  I'd expect it to be indented relative to "or".

The reason is that's how "if/then/else" works:

if A then
   B;

elsif C then
   D;

elsif 
  E then
   F;

else
  G;

end if;

Here the internal keywords "then", "elsif" are not transitive; "else" is
transitive. Here we are assuming "E" is a complex multi-line expression,
so we can't use it as an indentation reference; we have to get back to
"if" or "elsif". We call (smie-backward-sexp "then") to do that; it
stops at "if".

In the analogous situation with "select or", (smie-backward-sexp "or")
stops at the previous "or".

Why should a select statement be different? I guess you'd say "because
it is different" :). Still, I'd like one rule that handles both, and
it's quite easy to achieve. The fewer special cases I have, the easier
it is to maintain in the long run.

>> But this is exactly like the "elsif" case the comments are talking
>> about, so I think the code is broken.
>
> The comment is imprecise, it's meant for "if ... elsif ... end" where
> the BNF said ("if" ... "elsif" ... "end")

That structure matches this comment and code (as does the Ada "if"):

           ;; If the new operator is not the last in the BNF rule,
           ;; and is not associative, it's one of the inner operators
           ;; (like the "in" in "let .. in .. end"), so keep looking.
           ((not (smie--associative-p toklevels))
            (push toklevels levels))

However, what determines whether an operator is "transitive" in this
SMIE sense is more subtle; it depends on what the surrounding
non-terminals are. When the two non-terminals are the same, the operator
is transitive. When they are different, it is not. 

For Ada, we have:

"if" expression "then" statements "elsif" expression "then" statements
"end" "if"

"declare" declarations "begin" statements "end"

These keywords are all non-transitive, because they have different
non-terminals to the right and left; declarations, expressions, and
statements are all distinct. "else" above is transitive because it has
'statements' on both sides.

"select" statements "or" statements "else" statements "end" "select"

Here "or" and "else" are transitive, because they have 'statements' on
both sides. In fact have the same levels; both are (25 25) (illustrating
your earlier point that the operator table is not invertible in general).

Note that the "elsif then" pair also has the same levels on the left and
right - they form a "transitive pair". In this case, smie-next-sexp will
skip as many as appear.

Which suggests another way to state my core complaint; since
smie-next-sexp will skip an arbitrary number of "transitive pairs", it
should also skip an arbitrary number of "transitive singles". Or at
least, provide an option to do so.

> rather than ("if" ... list-of-elseif ... "end") and then
> list-of-elseif defined as (list-of-elseif "elsif" list-of-elseif).

adding the missing nonterminals (and remembering that in C-like
languages there are no "statements", only "expressions"):

("if" expressions "elseif" expressions "elseif" expressions "end")

These "elseif" are transitive. But again, it is the surrounding
non-terminals that determine this. In a language that distinguishes
between expressions and statements, "elsif" will never be transitive,
but "elsif expression then" will be.

> I should fix it, thank you for pointing it out.
>
> I.e. this rule is to avoid stopping at "elsif" when scanning from "if" to
> "end" or from "end" to "if".  It will still stop when scanning from an
> "elsif" and bumping into another "elsif".

Hmm. Calling (smie-forward-sexp "if") on the above "if" statement gives
the following toklevel sequence (obtained by setting a break in
smie-next-sexp):

((184) 3)   if
nil         A
(3 25)      then
nil         B
(37 36)     ;
(25 3)      elsif
nil         C
(3 25)      then
nil         D
(37 36)     ;
(25 3)      elsif
nil         E
(3 25)      then
nil         F
(37 36)     ;
(25 25)     else
nil         G
(37 36)     ;
(25 127)    end
(127 (174)) if

Now I see; the checks for associative in smie-next-sexp handle the case
where there is _one_ associative operator (here "else") in a statement.

And with the equivalent sequence in a C-like language that doesn't
have "then", it would stop at the second elsif. I don't see why Ada and
C-like should be different in this.

Since "else" will be transitive in most languages, and appear only once,
that would be a better keyword to use as an example in the current
smie-next-sexp code comments.

And I would still prefer having more control over this, to unify the
behavior of smie-next-sexp on Ada "if" and "select", and between Ada and
C-like "elsif". Similarly for "+"; if I had kept "+" in the grammar, I
would not want smie-next-sexp to stop at it. That choice should be up to
the smie user, not dictated by smie.

As usual, this conversation has been educational - I learned more about
what "transitive" means in SMIE :).

--
-- Stephe