[PATCH 0/4] Fix caret errors

[Akim Demaille]

Paul, you've invested quite some time in the scanner of Bison, dealing
with width issues.  I would really appreciate your input on the
questions below.

We had a bunch of problems:
- incorrect closing of styling in some corner cases
- incorrect location tracking with multibyte characters

Now that it is fixed, here is what I get (without colors, sorry):

input.y:11.4-17: warning: empty rule without %empty [-Wempty-rule]
 a: {           }
    ^~~~~~~~~~~~~~
input.y:12.4-17: warning: empty rule without %empty [-Wempty-rule]
 b: {            }
    ^~~~~~~~~~~~~~
input.y:13.4-17: warning: empty rule without %empty [-Wempty-rule]
 c: {------------}
    ^~~~~~~~~~~~~~
input.y:14.4-17: warning: empty rule without %empty [-Wempty-rule]
 d: {éééééééééééé}
    ^~~~~~~~~~~~~~
input.y:15.4-17: warning: empty rule without %empty [-Wempty-rule]
 e: {∇⃗×𝐸⃗ = -∂𝐵⃗/∂t}
    ^~~~~~~~~~~~~~
input.y:16.4-17: warning: empty rule without %empty [-Wempty-rule]
 f: {   42      }
    ^~~~~~~~~~~~~~
input.y:17.4-17: warning: empty rule without %empty [-Wempty-rule]
 g: {   "฿¥$€₦" }
    ^~~~~~~~~~~~~~
input.y:18.4-17: warning: empty rule without %empty [-Wempty-rule]
 h: {   🐃      }
    ^~~~~~~~~~~~~~

We have one remaining problem: we indent the quoted code with a single
space, and leave tabs in the output, so there is a "visual off by
one" on the caret-and-tildes:

 a: {           }
    ^~~~~~~~~~~~~~
 f: {   42      }
    ^~~~~~~~~~~~~~

instead of

a: {            }
   ^~~~~~~~~~~~~~
f: {    42      }
   ^~~~~~~~~~~~~~

Here is another example:

$ bison /tmp/foo.y
/tmp/foo.y:3.9: warning: stray '$' [-Wother]
 a: {   $       $       }
         ^
/tmp/foo.y:3.17: warning: stray '$' [-Wother]
 a: {   $       $       }
                 ^
/tmp/foo.y:4.9: warning: stray '$' [-Wother]
 b: {    $       $       }
         ^
/tmp/foo.y:4.17: warning: stray '$' [-Wother]
 b: {    $       $       }
                 ^

The line in "a:" used tabs, the line "b:" used spaces.  Note not only
the off-by-one, but also the fact that columns concur: tabs send to
the next multiple of 8.


I don't know what is the best option here.  I checked how GCC and
Clang deal with this, they have two different approaches.

Clang does not indent the quoted source, so tabs are not a problem.  I
personally don't like the fact that the message is not indented, but
at least it works nicely:

$ clang-mp-7.0 /tmp/foo.c
/tmp/foo.c:3:5: error: cannot combine with previous 'int' declaration specifier
int     int     i;
        ^
1 error generated.

GCC indents with one space, but smashes the tabulation characters into
spaces:

$ gcc-mp-8 /tmp/foo.c
/tmp/foo.c: In function 'main':
/tmp/foo.c:3:5: error: two or more data types in declaration specifiers
 int int i;
     ^~~

Interestingly, both GCC and Clang agree that this is column 5, i.e.,
as if the tabulation was a single space.  This is not was Bison does:
tabs send to the next multiple of 8.

So what shall we do?

1.  Should we stick to treating tabs as magical characters wrt column
    numbers?  Doing so gives consistent locations for people who mix
    tabs and spaces.  But a. they need to also tabs of 8 spaces, and
    b. indenting the way we do is visually wrong.

2.  How should we quote source?
    a. Do like Clang
    b. Do like GCC
    c. Do like Bison, i.e., consider this is an unimportant issue,
       people should not use tabs anyway :)

This is also available here: https://github.com/akimd/bison/pull/9.

Akim Demaille (4):
  diagnostics: check the styling
  diagnostics: fix styling issues
  diagnostics: check the handling of tabulations
  diagnostics: fix the handling of multibyte characters

 gnulib               |   2 +-
 src/complain.c       |  18 +++--
 src/getargs.c        |  11 ++-
 src/getargs.h        |   2 +-
 src/location.c       |  35 ++++++----
 src/location.h       |  18 +++--
 src/parse-gram.c     |   4 +-
 src/scan-gram.l      |  23 +++++--
 tests/diagnostics.at | 156 +++++++++++++++++++++++++++++++++++++++++++
 tests/local.mk       |   1 +
 tests/testsuite.at   |   3 +
 11 files changed, 240 insertions(+), 33 deletions(-)
 create mode 100644 tests/diagnostics.at

-- 
2.21.0