Re: Elisp development news

[Marius Vollmer]

Hi,

sorry for the long delay.  Let me try to summarize what we have so
far, in broad strokes.

- Regardless to what conclusion we come here, it will not affect how
  isolated Elisp code runs when being executed by Guile.  What we are
  discussing, only affects how Elisp and Scheme interact when being
  used side-by-side.

- Neither Neil's proposal nor Jim's are perfect.  For both, there is
  probably code out there that will break.  However, with Neil's
  proposal we don't need to change the core of Guile, while with Jim's
  we would need to add a special value and make sure that all
  low-level code respects it.

- There are no drawbacks in Jim's proposal as compared to Neil's.
  Jim's stuff will give us no more problems than Neil's, but probably
  less.

- So the question is, what is the cost of implementing and maintaining
  the core changes for Jim's proposal; and what do we gain compared to
  Neil's proposal.

Is that about right?


My position right now is that I regard Jim's proposal as significantly
more robust, and its cost to be tolerable.

In Elisp, nil is used as false, as the empty list, and as a symbol.
In Scheme we have three distinct objects for these three uses.
Equating nil with any of the three destroys information that might be
needed to work around specific problems in the interaction between
Elisp and Scheme.

I think this destruction of information is an important point.  The
lost information might be reconstructable in many cases, but the need
to do so will be a significant burden.

Neil Jerram <address@hidden> writes:

>     Marius> For example, suppose nil is the same as #f.  This means
>     Marius> that Elisp lists are #f-terminated and you can't pass it
>     Marius> to functions that expect Scheme lists.  This would prevent
>     Marius> you from using (ice-9 common-list), say.
> 
> No; this is not my intention.  I am proposing that nil be the same as
> #f, and that Elisp lists are still terminated by '().  Normal list
> functions and (ice-9 common-list) should work without modification on
> Elisp lists.
> 
> [All we need to achieve this is to handle #f when passed as the CDR
> parameter to cons and setcdr. 

Hmm, sure?  It wouldn't work for the empty list, and cons is not only
used to construct lists.  What about a tree with boolean values at the
leaves?  It seems wrong to treat the cdr different from the car.

>     Marius> If nil would be the same as '(), Elisp lists and Scheme
>     Marius> lists would be the same, but Elisp would use '() to
>     Marius> indicate false.  It would be unnatural to use predicates
>     Marius> written in Elisp with Scheme functions.
> 
> I am not proposing this, but in any case I don't see the unnaturalness
> that you're pointing out.
> 
> Given that Elisp treats nil, #f and '() as the same, I see no problem
> with either
> 
>      (elisp-predicate (scheme-boolean-valued ...))
> 
> or
> 
>      (elisp-predicate (scheme-list-valued ...))
> 
> The other way round is an issue that we'll get to in a moment.

I meant the other way around, like (pick-mappings elisp-predicate ...).

>     Marius> In general, we would need to closely track what values
>     Marius> have been produced by Elisp, and with what meaning.  Doing
>     Marius> this statically, without support from the system, is hard
>     Marius> and will ultimately fail.  Doing it dynamically is
>     Marius> equivalent to not equating nil with either #f or '().
> 
> What do statically and dynamically mean here?

This relates to the `lost information' from above.  With statically,
I'm referring to your proposal where you reconstruct this information
at coding time, by embedding knowledge into the code what the
different slots of a data structure mean with regard to empty
list/false.  With dynamically, I'm referring to Jim's proposal where
this information is not actually lost and can be extracted from the
values itself.  I see this analogous to static/dynamic typing.  Your
proposal amounts to static typing, which I don't think fits well into
either Elisp or Scheme.

> [...]
>
> So, yes, where possible, I agree that detailed understanding should be
> preserved.  And in most cases it is, simply by doing nothing special.
>
> Here are some illustrative Elisp examples, showing how preservation
> would ("YES") or would not ("NO") work under my proposal:

I'd say there are two classes in your examples that probably need to
be separated:

>   x                          ; YES: #f -> #f,  '() -> '()
>   (car (cons x 1))           ; YES
>   (cdr (cons 1 x))           ; NO:  #f -> '(), '() -> '()
>   (aref (vector x) 0)        ; YES

These are functions that normally would return their argument
unchanged.  That some no longer don't (when observed from Scheme),
worries me some.

>   (if x 'true x)             ; YES
>   (if x 'true nil)           ; NO:  #f -> #f,  '() -> #f
>   (if x 'true '())           ; NO:  #f -> '(), '() -> '()

These are function that should preserve the true/false meaning of
their argument.  That some no longer don't (when observed from
Scheme), worries me even more than above.

> With Jim's proposal, I think the only difference is that the
> non-preserving cases would always map to nil rather than #f or '().

Yes, for the second class.  For the first class, x would be returned
unchanged.

>     Marius> But Scheme might want to `do the right thing' without
>     Marius> knowing that the value came from Elisp, and in what
>     Marius> context.  It might want to do this because tracking this
>     Marius> information might be too hard.
> 
> I think this might be the key issue.
> 
> In concrete terms, if I understand you correctly, you mean that it
> would be convenient to write (in Scheme):
> 
>     (if (elisp-defined-boolean-valued-function ...)
>         ...)
> 
> and
> 
>     (if (null? (elisp-defined-list-valued-function ...))
>         ...)
> 
> whereas my current proposal requires something like
> 
>     (if (not (elisp-nil? (elisp-defined-boolean-valued-function ...)))
>         ...)
> 
> and
> 
>     (if (elisp-nil? (elisp-defined-list-valued-function ...))
>         ...)
> 
> In other words, you (and Jim) would like Elisp boolean false values to
> work "natively" with `if', and Elisp empty list values to work
> natively with `null?'.

Exactly.  But also note that this goes beyond return values from
function calls.  All values, whether returned by some function or
found deep in data structures need to behave this way.  For function
calls it might work to put in `elisp-nil?' in the right places, but
for general data structures that can't be clearly associated with one
of Elisp or Scheme, it will not work.

> I agree that it would be convenient, but I think it's impossible in
> general to achieve this.

But I hope and think that we can get very, very close.

> The problem is preservation, as discussed just above.  Although the
> code for an Elisp boolean (or list) valued function will most usually
> generate and return a new `nil' (or `'()') value, it's also possible
> for it to return a false (or empty) value that comes by preservation
> from an input parameter that was a different kind of null value.
> 
> For example,
> 
> (defun frobbable-p (x)
>   (if x
>       (and (consp x)
>            (eq (car x 'frobbable)))
>     x))
> 
> Then we import frobbable-p into Scheme and try to use it in Scheme
> code like this:
> 
> (if (frobbable-p my-x)
>     (frob my-x))
> 
> According to Jim's proposal, I also assume for this example that `nil'
> is a distinct value and that Scheme's `if' has been modified to treat
> both #f and nil as false.
> 
> So,
> 
> - if my-x is true as seen by Elisp, we evaluate the `and' expression,
>   and the return value is either `t' or `nil'
> 
> - if my-x is nil or #f, Elisp sees it as false and returns the same
>   value, which the modified Scheme `if' treats as we intend
> 
> - if my-x is '(), Elisp sees it as false and returns the same value,
>   which the modified Scheme `if' treats as _true_; so the code does
>   not work as intended.

Yes.

I think that this kind of code is sufficiently rare, tho.  It should
really be written as

    (defun frobbable-p (x)
      (and (consp x)
           (eq (car x) 'frobbable)))

[ or even as (defun frobbable-p (x) (eq (car x) 'frobbable)), given
  that (car nil) is nil ]

> Conclusion: the kind of transparency that you would like to see, when
> dealing in Scheme with the return value from an Elisp function, is not
> achievable in conjunction with the value preservation behaviour
> described above.

No perfect transparency is achievable, but I think Jim's proposal is
significantly more transparent than yours.

> We could achieve the desired transparency by retreating a bit on value
> preservation.  Two options spring to mind:
> 
> 1. Whenever Elisp sees a #f or '() value (at "top level", not within a
>    data structure), it converts it to the distinct value nil.

What is the "top level"?  Whenever evaluating a variable reference?
This might work, but I think we can get away without it.

>     Marius> So, I think Jim's proposal gives us the same things as
>     Marius> your proposal, and more.  What is the advantage of your
>     Marius> proposal over Jim's?
> 
> If all else is equal, I would say that my proposal is better than
> Jim's because it is simpler.  In particular, because it does not
> require us to change Guile Scheme such that there are 2 possible false
> values and 2 possible end-of-list values.
> 
> (Take the documentation, for example.  Currently we can have nice
> unequivocal statements like these:
> 
>   "In test condition contexts like `if' and `cond' (*note if cond
> case::), where a group of subexpressions will be evaluated only if a
> CONDITION expression evaluates to "true", "true" means any value at all
> except `#f'."
> 
>   "It is important to note that `#f' is *not* equivalent to any other
> Scheme value.  In particular, `#f' is not the same as the number 0
> (like in C and C++), and not the same as the "empty list" (like in some
> Lisp dialects).")
> 
> But is all else equal?
> 
> The only advantage of Jim's proposal (AFAICS) is that, by introducing
> and using the distinct value `nil', it tries to provide more
> seamlessness in the handling in Scheme of null-like values returned
> from Elisp, specifically by trying to arrange that these null-like
> values work "natively" with Scheme's if, cond and null?.  However, I
> think I have demonstrated above that this does not work in general,

We should analyze the existing Elisp code to see how often the pattern
(if x ... x) appears.

> given our other desideratum ("value preservation") that Elisp should
> not unnecessarily modify data that simply passes through Elisp code.
> IMO, if we can't make this work in all cases, it is better to avoid
> the confusion of it working in just some cases.

I would characterize this as "working in the vast majority of cases".

> So I think all else _is_ equal, because the advantage of Jim's
> proposal turns out AFAICS to be a false promise; therefore I
> recommend my own proposal on the basis of its relative simplicity.

I don't agree, yet.  However, since adopting your proposal means doing
nothing, we have in effect adopted it already until it is clear that
it falls short of our goals (if at all).  Please try to make sure that
the Elisp translator can be switched between both proposals.  I think
this will not be too difficult.