Re: Elisp development news

[Neil Jerram]

>>>>> "Marius" == Marius Vollmer <address@hidden> writes:

    Marius> Neil Jerram <address@hidden> writes:
    >> Now here's Jim's proposal and my comments on it...
    >> 
    >> From: Jim Blandy (address@hidden)
    >> 
    >> [...]
    >> 
    >> None of the obvious compromises work nicely. Modifying either
    >> language to match the other breaks existing code in ways that
    >> are impossible to fix mechanically. Making Emacs Lisp's nil
    >> identical to either of Scheme's #f or '() will cause
    >> misinterpretations when Scheme receives either Boolean values
    >> or lists from Emacs Lisp.
    >> 
    >> How?  Jim isn't clear about these misinterpretations.
    >> 
    >> Given <the set of objects that Elisp code treats as false>, and
    >> an Elisp function that may return any of these to indicate
    >> false, a Scheme function examining this return code would have
    >> to check for all possible Elisp false values.  I don't see how
    >> this interacts with the question of whether nil can be
    >> identical to either #f or '().

    Marius> I think Jim refers to the goal to be `nearly transparent':
    Marius> there might be cases where explicit conversion between
    Marius> Scheme and Elisp data structures need to take place, but
    Marius> they should be very sparse and for the unusual situation.

    Marius> When nil is equated with one of '() or #f, only the
    Marius> context determines whether it should be treated as false
    Marius> or as null in Scheme.  You would have to know in what
    Marius> context you are to use the right predicate.  It is easy to
    Marius> write Scheme that knows the context, but if we can arrange
    Marius> things so that one can write code that works regardless of
    Marius> context, it would be better.

OK, thanks; I think I understand what you are driving at here.  See
detailed discussion further down.  ("I think this might be the key
issue.")

    Marius> For example, suppose nil is the same as #f.  This means
    Marius> that Elisp lists are #f-terminated and you can't pass it
    Marius> to functions that expect Scheme lists.  This would prevent
    Marius> you from using (ice-9 common-list), say.

No; this is not my intention.  I am proposing that nil be the same as
#f, and that Elisp lists are still terminated by '().  Normal list
functions and (ice-9 common-list) should work without modification on
Elisp lists.

[All we need to achieve this is to handle #f when passed as the CDR
parameter to cons and setcdr.  For example, my current definitions of
cons and setcdr are:

(fset 'cons
      (lambda (x y)
        (cons x (or y '()))))

(fset 'setcdr
      (lambda (cell newcdr)
        (set-cdr! cell (or newcdr '())))) ]

    Marius> If nil would be the same as '(), Elisp lists and Scheme
    Marius> lists would be the same, but Elisp would use '() to
    Marius> indicate false.  It would be unnatural to use predicates
    Marius> written in Elisp with Scheme functions.

I am not proposing this, but in any case I don't see the unnaturalness
that you're pointing out.

Given that Elisp treats nil, #f and '() as the same, I see no problem
with either

     (elisp-predicate (scheme-boolean-valued ...))

or

     (elisp-predicate (scheme-list-valued ...))

The other way round is an issue that we'll get to in a moment.

    Marius> In general, we would need to closely track what values
    Marius> have been produced by Elisp, and with what meaning.  Doing
    Marius> this statically, without support from the system, is hard
    Marius> and will ultimately fail.  Doing it dynamically is
    Marius> equivalent to not equating nil with either #f or '().

What do statically and dynamically mean here?

    >> We want list-valued Scheme and Emacs Lisp functions to
    >> interoperate.
    >> 
    >> Yes, but ...
    >> 
    >> So Scheme must be able to return '() to Emacs Lisp, and have
    >> Emacs Lisp recognize it as the empty list. Emacs Lisp must be
    >> able to return nil to Scheme, and have Scheme recognize it as
    >> the empty list.
    >> 
    >> ... not quite.
    >> 
    >> For any value X that language A passes to language B, the
    >> requirement is that B understands what A means by X, to the
    >> extent possible given B's own linguistic limitations.

    Marius> Right.  But this understanding should persevere thru
    Marius> layers of ignorant code and should be extractable from
    Marius> data structures.

    Marius> This is possible to do statically, but will look unnatural
    Marius> and needlessly hard for a dynamically typed language, in
    Marius> my opinion.

I have previously suggested that Elisp code should convert '() on
first contact to #f.  I don't now think this is a good idea - for
exactly this reason - and wasn't suggesting this as part of the
current proposal.

So, yes, where possible, I agree that detailed understanding should be
preserved.  And in most cases it is, simply by doing nothing special.

Here are some illustrative Elisp examples, showing how preservation
would ("YES") or would not ("NO") work under my proposal:

  x                          ; YES: #f -> #f,  '() -> '()
  (if x 'true x)             ; YES
  (if x 'true nil)           ; NO:  #f -> #f,  '() -> #f
  (if x 'true '())           ; NO:  #f -> '(), '() -> '()
  (car (cons x 1))           ; YES
  (cdr (cons 1 x))           ; NO:  #f -> '(), '() -> '()
  (aref (vector x) 0)        ; YES

With Jim's proposal, I think the only difference is that the
non-preserving cases would always map to nil rather than #f or '().

    >> Where null-like values are concerned,
    >> 
    >> - When A is Elisp and B is Scheme, "what A means by X" can only
    >> mean whether X is false/empty in the Elisp sense.  If the
    >> concept of Elisp false/empty maps to more than one
    >> distinguishable value in Scheme, note that
    >> 
    >> - it is possible and straightforward to write a Scheme
    >> predicate `elisp-false/empty?' that returns #t if its argument
    >> is any of those values
    >> 
    >> - it doesn't make sense for the Scheme side to try to draw any
    >> conclusion by examining which of those values is the one
    >> actually returned, since the distinction was not observable in
    >> Elisp.

    Marius> But Scheme might want to `do the right thing' without
    Marius> knowing that the value came from Elisp, and in what
    Marius> context.  It might want to do this because tracking this
    Marius> information might be too hard.

I think this might be the key issue.

In concrete terms, if I understand you correctly, you mean that it
would be convenient to write (in Scheme):

    (if (elisp-defined-boolean-valued-function ...)
        ...)

and

    (if (null? (elisp-defined-list-valued-function ...))
        ...)

whereas my current proposal requires something like

    (if (not (elisp-nil? (elisp-defined-boolean-valued-function ...)))
        ...)

and

    (if (elisp-nil? (elisp-defined-list-valued-function ...))
        ...)

In other words, you (and Jim) would like Elisp boolean false values to
work "natively" with `if', and Elisp empty list values to work
natively with `null?'.

I agree that it would be convenient, but I think it's impossible in
general to achieve this.

The problem is preservation, as discussed just above.  Although the
code for an Elisp boolean (or list) valued function will most usually
generate and return a new `nil' (or `'()') value, it's also possible
for it to return a false (or empty) value that comes by preservation
from an input parameter that was a different kind of null value.

For example,

(defun frobbable-p (x)
  (if x
      (and (consp x)
           (eq (car x 'frobbable)))
    x))

Then we import frobbable-p into Scheme and try to use it in Scheme
code like this:

(if (frobbable-p my-x)
    (frob my-x))

According to Jim's proposal, I also assume for this example that `nil'
is a distinct value and that Scheme's `if' has been modified to treat
both #f and nil as false.

So,

- if my-x is true as seen by Elisp, we evaluate the `and' expression,
  and the return value is either `t' or `nil'

- if my-x is nil or #f, Elisp sees it as false and returns the same
  value, which the modified Scheme `if' treats as we intend

- if my-x is '(), Elisp sees it as false and returns the same value,
  which the modified Scheme `if' treats as _true_; so the code does
  not work as intended.

Conclusion: the kind of transparency that you would like to see, when
dealing in Scheme with the return value from an Elisp function, is not
achievable in conjunction with the value preservation behaviour
described above.

We could achieve the desired transparency by retreating a bit on value
preservation.  Two options spring to mind:

1. Whenever Elisp sees a #f or '() value (at "top level", not within a
   data structure), it converts it to the distinct value nil.

2. We use some heuristic to guess when a function is intended to be
   boolean- or list- valued, and apply the conversion to nil only for
   those functions.

I don't like either of these.  In particular, if we ever think along
the lines suggested by (2), I think a far preferable solution would be
to provide `to-scheme-boolean' and `to-scheme-list' functions in Elisp
that the developer can use to force conversions when he/she wants.

    >> My statements are weaker than Jim's.  For example, his
    >> 
    >> "have Scheme recognize it as the empty list"
    >> 
    >> becomes my
    >> 
    >> "have Scheme tell the difference between a non-empty list and a
    >> false/empty value".
    >> 
    >> But I think that my statements are all that is necessary in
    >> practice.

    Marius> Maybe, but I don't think it Jim's proposal is difficult to
    Marius> implement or necessarily inefficient.

Nor do I.  But comparative attributes are more relevant I think than
absolute ones.  I'll summarize my view of the comparison between Jim's
and my proposal at the end of this email.

    >> The crucial consequence of this weakening is that the set of
    >> Guile values that describe end-of-list in Elisp do not all have
    >> to satisfy (null? x) in Scheme, and the set of Guile values
    >> that describe false in Elisp do not all have to satisfy (not x)
    >> in Scheme.
    >> 
    >> This opens the door for proposing that `nil' be made identical
    >> to #f and Elisp's '() the same as Scheme's '().

    Marius> But in Elisp nil and '() are the same.  You can rig Elisp
    Marius> `eq' to recognize this, but it is legal in Elisp to
    Marius> terminate lists with '() or nil, so I don't think it makes
    Marius> sense to have Eliusp '() and Elisp nil not be equal.  They
    Marius> would have to be treated identically in all contexts.

Yes, they are certainly equal, so far as Elisp can tell, and are
treated identically in all contexts.  This is easily achieved by
rigging eq, null (aka not), cons, setcdr, if, while, cond (and perhaps
a few that I've forgotten).  This is the same for both my and Jim's
proposals, I think.  (Except that my rigging only has to cope with 2
different underlying values; Jim's has to cope with 3.)

    >> I don't see any harm in telling Emacs Lisp that #f is an empty
    >> list.  That merely equates two objects that Emacs Lisp never
    >> distinguished anyway.
    >> 
    >> I'm not clear what Jim means here.  If that (if (null X) "yes"
    >> "no") => "yes", where X is the Scheme #f value, fine.

    Marius> Yes, that's how I understand it as well.
Right- thanks.

    Marius> So, I think Jim's proposal gives us the same things as
    Marius> your proposal, and more.  What is the advantage of your
    Marius> proposal over Jim's?

If all else is equal, I would say that my proposal is better than
Jim's because it is simpler.  In particular, because it does not
require us to change Guile Scheme such that there are 2 possible false
values and 2 possible end-of-list values.

(Take the documentation, for example.  Currently we can have nice
unequivocal statements like these:

  "In test condition contexts like `if' and `cond' (*note if cond
case::), where a group of subexpressions will be evaluated only if a
CONDITION expression evaluates to "true", "true" means any value at all
except `#f'."

  "It is important to note that `#f' is *not* equivalent to any other
Scheme value.  In particular, `#f' is not the same as the number 0
(like in C and C++), and not the same as the "empty list" (like in some
Lisp dialects).")

But is all else equal?

The only advantage of Jim's proposal (AFAICS) is that, by introducing
and using the distinct value `nil', it tries to provide more
seamlessness in the handling in Scheme of null-like values returned
from Elisp, specifically by trying to arrange that these null-like
values work "natively" with Scheme's if, cond and null?.  However, I
think I have demonstrated above that this does not work in general,
given our other desideratum ("value preservation") that Elisp should
not unnecessarily modify data that simply passes through Elisp code.
IMO, if we can't make this work in all cases, it is better to avoid
the confusion of it working in just some cases.  A preferable
solution, which is also compatible with my proposal, is to provide
Elisp functions that the developer can use to force conversion
explicitly.

So I think all else _is_ equal, because the advantage of Jim's
proposal turns out AFAICS to be a false promise; therefore I
recommend my own proposal on the basis of its relative simplicity.

Thanks for reading!  If nothing else, I think we are at least building
a much clearer picture of the issues involved here.

        Neil