Re: scratch/accurate-warning-pos: next steps.

[Alan Mackenzie]

Hello, Eli.

On Mon, Dec 10, 2018 at 22:06:33 +0200, Eli Zaretskii wrote:
> > Date: Mon, 10 Dec 2018 19:35:57 +0000
> > Cc: address@hidden
> > From: Alan Mackenzie <address@hidden>

> > > Then how about invoking this alternative interpreter only if the prime
> > > interpreter detected a warning or error while byte-compiling?  You
> > > could invoke the alternative interpreter only on the form where the
> > > problem was detected, with the goal of "drilling down" to find the
> > > exact position of the problematic symbol(s).

> > That would mean starting the byte compilation with no position
> > information being gathered, and then when an warning occurs, aborting
> > the compilation and starting again from scratch with the position
> > information being gather and alternative interpreter being used.

> Not necessarily.  It could mean invocation of a special code whose
> goal is to find the position of an error in a given form.  The
> position of the beginning of this form will have been known, as AFAIU
> the existing byte compiler does collect that, or has means to
> determine that.

We know the position of the beginning of the form, yes.  We need some way
of determining the source position of a symbol, cons, or vector on the
inside of this form.

The traditional alist of symbols and positions is one way, and it no
longer works well (if it ever did).  Symbols with position is another
way, which appears to work well, in spite of the complexity.

You seem to be proposing a third way, but without giving away any
details.

> > The problem is, that we cannot use #<symbol nil at 666> in the normal
> > interpreter, since it is not EQ nil there.

> I'm not sure you must use symbols with positions in the above
> arrangement, you could simply invoke special-purpose code that
> analyzed the problematic form.  But if you do need to use symbols with
> positions, you could do this only when looking for error position, so
> other symbol comparisons will not be affected.

I'm not sure how this special-purpose code would work.  Say we find an
error or warning involving symbol foo as the car of some form, I can't
see any way of determining its source position that doesn't involve going
back to the position of the beginning of the form, and slogging through
the form, somehow.

Maybe, rather than reading the form, we could scan it a token at a time,
storing it in, say vectors, rather like a traditional non-lisp compiler
does.  But this is hardly attractive, and would be a LOT of work.

> > I understand the idea, yes.  But given the timings I measured in the
> > existing scratch/accurate-warning-pos (IIRC, around 11% - 12% for an
> > actual compilation) and the fact that in the alternative interpreter,
> > the slowdown will be somewhat less (one fewer flag comparison per EQ,
> > NILP, ...., and we can drop the traditional alist of symbols and
> > positions which is running alongside the new symbols with position) it
> > may not be worth the extra complexity.

> Yes, but what you suggested as the implementation of the alternative
> interpreter includes a heck of complexity of its own, IMO.

It does, yes.  Partly imposed by external circumstances.  ;-)

> The idea I proposed doesn't even require changes in basic types, it
> could hopefully be implemented with "normal" Lisp.

The symbols with positions idea doesn't conceptually change basic types
either - it just adds annotations to an existing type.  Anything you
could do with symbols before, you can still do with
symbols-with-positions and get the same answers now.

If you have come up with a new way of getting a source position for a
symbol or a cons, please detail it here.  I recognise the complexity in
what I'm proposing and what I've already implemented, and I don't think
it's good; it's just less bad than anything else that's come up, so far.

-- 
Alan Mackenzie (Nuremberg, Germany).