Need Advice: Rebinding in the Face of Buffer-Local Variables; `let-default'

[hokomo]

Hello,

I'm looking for a bit of Emacs Lisp advice regarding the behavior of 
buffer-local variables. Assume `lexical-binding' is t in all of the examples 
that follow.

First, some context. Buffer-local variables have the curious property that their let-bindings 
temporarily "disappear" when the buffer is changed. This is indeed documented in the 
manual [1] (see "Warning: ..."), but here's a slightly bigger example:

 (defvar local 'default)

 (let ((a (generate-new-buffer "a"))
       (b (generate-new-buffer "b")))
   (unwind-protect
       (progn
         ;; Make `local' buffer-local in `a'.
         (with-current-buffer a
           (make-local-variable 'local)
           (setq local 123))
         (message "(1) `local': %s" local)
         (with-current-buffer a
           (message "(2) `local': %s" local)
           ;; Bind `local' within `a'.
           (let ((local 456))
             (message "(3) `local': %s" local)
             ;; The binding of `local' "disappears".
             (with-current-buffer b
               (message "(4) `local': %s" local)
               ;; The binding of `local' is back.
               (with-current-buffer a
                 (message "(5) `local': %s" local))))))
     (kill-buffer a)
     (kill-buffer b)))

 ;; (1) ‘local’: default
 ;; (2) ‘local’: 123
 ;; (3) ‘local’: 456
 ;; (4) ‘local’: default
 ;; (5) ‘local’: 456

Note the outputs of (4) and (5). Conceptually, it's as if a buffer-local variable has a separate "stack" of 
dynamic bindings within each buffer where it is marked buffer-local. This is in addition to the variable's 
"default stack" of dynamic bindings that make up its "default" and "top-level default" 
values. The default value of a variable is therefore the value of its innermost non-buffer-local binding, or the 
global/top-level value if it is not let-bound. [2]

Now, here's a problem I ran into. Assume there's some library that provides a 
variable `var' and a function `f', and that `f' uses `var' somehow. I want to 
temporarily rebind `var' and invoke the function `f' to get some desired effect:

 (let ((var 123))
   (f))

However, assume also that (a) `var' just happens to be buffer-local within the 
current buffer B1 (e.g. because the user made it so), and (b) before doing the 
work that depends on `var', `f' first switches to buffer B2, does some 
bookkeeping, then does whatever depends on `var', and finally switches back to 
B1.

This unfortunate combination of events causes my binding of `var' to never be 
seen, as shown in the example at the beginning. Essentially, I was expecting to 
rebind the value of `var' so that it would be seen for the duration of the call 
to `f', but because `var' happened to be buffer-local (perhaps by user's choice 
or accident) and `f' changed the buffer, the binding vanished during the part 
of `f' where it mattered.

Note that it doesn't really matter whether `var' is buffer-local within B2 or 
not, as the effect will be the same. If it is, then B2's buffer-local binding 
of `var' will be seen. If it isn't, then `var's default value will be seen. 
Both of these are different from the binding that was established just before 
invoking `f' (which was conceptually added to the stack of B1's buffer-local 
bindings). For the purposes of my problem here, let's say that we are 
guaranteed `var' won't be buffer-local within B2 (e.g. because it's just a 
temporary one-off buffer).

Does this mean that the "simple task" of let-binding a variable around a call is inherently 
"unreliable" in Emacs Lisp due to the fact `var' might be buffer-local in B1? Is there a way out of 
this dilemma or is this "just the way it is"?

With the above guarantee about B2, one solution could be to explicitly rebind 
the *default* value of `var'. This can be done by temporarily switching from B1 
to some buffer where `var' is not buffer-local, creating the binding, and 
switching back before invoking `f'. For example, here's how a `let-default' 
might look like:

 (defmacro let-default (bindings &rest body)
   (declare (indent 1))
   (macroexp-let2 nil buffer '(current-buffer)
     `(with-current-buffer (generate-new-buffer " *let-default*" t)
        (let ,bindings
          (with-current-buffer ,buffer
            ,@body)))))

The usage would then be:

 (let-default ((var 123))
   (f))

A small wrinkle is the existence of "built-in buffer-local variables" [3] (see 
`buffer-local-variables') such as `default-directory'. These variables are buffer-local 
within all newly created buffers so rebinding their default value would make no sense. 
The implementation of `let-default' could check for these and throw an error at 
macroexpansion-time to avoid programmer suprises. However, these variables don't fit the 
use case anyway, as they automatically violate our guarantee about B2.

While `let-default' helps, it feels very overkill, as does having to worry 
about this issue in the first place. But clearly situations like these do pop 
up so it seems worth talking about. Is there any other way out?

Perhaps it's not so much about binding but about switching buffers? Maybe the lesson here 
is that switching the current buffer should be done carefully and `f' should restore the 
original buffer before proceeding with the work that depends on `var', in order to 
anticipate `var' possibly being buffer-local and not disrupt any bindings established by 
the user or code? This pushes the responsibility onto library authors and sadly makes 
buffer-local variables "less transparent" to work with than one would hope for 
(for the programmer at least).

Or maybe if B2 is a temporary buffer it should first inherit all of B1's 
buffer-locals and their current bindings? Is there an idiomatic way to do this?

Thoughts?

[1] 
<https://www.gnu.org/software/emacs/manual/html_node/elisp/Intro-to-Buffer_002dLocal.html>
[2] 
<https://www.gnu.org/software/emacs/manual/html_node/elisp/Default-Value.html>
[3] 
<https://www.gnu.org/software/emacs/manual/html_node/elisp/Creating-Buffer_002dLocal.html#index-buffer_002dlocal_002dvariables>

Thanks,
hokomo