bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Drew Adams]

> > > > Hope this clarifies the request/bug report.
> > >
> > > It doesn't.  Sorry, I guess I'm too stupid
> > > to understand what you are asking.
> >
> > Don't be silly or sarcastic, please.
> 
> This isn't sarcasm, this is frustration.
> And please avoid ad-hominem if you can, especially when you yourself
> use language that can easily be interpreted as sarcasm:
> 
> > A user cares what's available from `M-n'.
> > (I don't expect another "Why?" for that,
> > but it comes, I'll try to answer that too.)
> 
> Isn't that sarcasm?

No, it's not.  At all.  It's legitimate for
someone to ask "Why?" for that too.  I was
guessing that I probably didn't need to
explain that, but I wanted to make clear that
I could, and would, if you wanted a reason
for that too.

> So let's try to drop the attitude and discuss the real problems, okay?

Agreed.
(But I'm not showing any negative "attitude".
Please don't look for that a priori.)

> (Adding Stefan, because I think he could have insights in this area
> that is nowadays complicated enough to blow my mind.)

Good.  I'm like you in that.

> > Why does it matter that all initial (i.e.,
> > prior to any user input in the minibuffer)
> > completions are put into the `M-n' queue?
> >
> > Because that set of candidates is often
> > huge.  And because its order isn't designed
> > for `M-n' or (especially) for the particular
> > act of input reading.  Often its order has
> > no special reason.
> >
> > And the case given as an example, `C-h v',
> > illustrates that well: (1) zillions of vars,
> > (2) in no user-expectable/understandable/
> > useful order - the order of obarray!
> 
> This part of your report seems to be a separate issue -- you seem to
> be saying that "C-h v" and similar commands should not add all the
> variables to the "future history".  It's possible that you are right,
> although it could be useful if M-s and M-r in the minibuffer would
> actually search that list -- which they don't currently, due to how
> this "add to future history" feature is implemented to add elements
> lazily (see goto-history-element).

It may seem to be a separate issue.  Actually,
it's just an extreme case, to make clear what
the general problem is.  `C-h v' is as bad as
it can be.

Yes, such extreme cases could be handled
separately. It would be easier, and saner, to
do that if the default behavior for automatic
addition were OFF, not ON.

But I argue that there isn't any good reason,
even in the case where the search space is
small (small completion domain).  Or at least
I'm asking for a rationale behind such default
behavior.  So far, I think there's no good
reason, but I might just be missing something.
 
> But that is a separate issue, almost unrelated to the Subject of your
> report, which is about documentation.

I don't think so (see above).  Cases such as
`C-h v' just point out the problem starkly.

Yes, large completion domains are especially
problematic for this default behavior, because
of the size of the `M-n' queue.  Yes, obarray
is especially problematic for this, because of
its useless order of symbols.

But I'm asking about the general idea behind
this default behavior: What use case(s) does
it really help with?  Even with a small list
of (empty-input) completions, and even when
those are in some meaningful/useful order,
what's the use case for adding them to `M-n',
which is a carefully designed default or list
of defaults?  Why use the completion domain
as a set of defaults - at all?

> Whatever problems we have in this area with
> "C-h v", they cannot be solved by documentation
> in the ELisp manual.

I spoke to this.  With the assumption that the
behavior won't be changed, the doc can at least
help by pointing out that for such cases (large
completion domain, no useful order), you can at
least mitigate the problem:

* You can bind the variable to nil - no addition.
* You can give the variable a custom function
  value that limits the size or sorts to a useful
  order.

Elisp programmers can do that in code they write
that reads minibuffer input (in particular, with
completion).  Emacs users can do that with a
defvar that defines the default value of
`minibuffer-default-add-function' (e.g. as nil).

Currently the doc doesn't point out the problem
or these possible mitigations.  If the problem
isn't fixed then the doc can avert users to it.
 
> So what is the documentation issue?

See above.  Make users aware of the problem
and how to mitigate it a bit - as long as
it exists.

> You say:
> 
> > What an Elisp user (not an end user of a
> > command) really needs to care about is var
> > `minibuffer-default-add-function', not the
> > particular function that's its default value.
> >
> > Forget for a moment about what various
> > function values for that variable might do.
> > The most important thing about that var is
> > that if nil then the domain of completions
> > isn't added to the `M-n' queue at all.  IOW,
> > that _turns off_ the automatic filling of
> > the `M-n' queue.
> >
> > An Elisp user needs to know that fact, if
> > s?he uses `completing-read' and s?he wants
> > to prevent the kind of confusing overkill
> > exhibited by `C-h v'.  (She then needs to
> > bind the var to nil around the call to
> > `completing-read').
> > [...]
> > Elisp users thus need to know that to define
> > the subset and its order for `M-n' they can
> > bind var `minibuffer-default-add-function'
> > to a function that returns such a list.
> > This isn't obvious.  You won't find it by
> > reading the `completing-read' doc, at least,
> > though it's just as important to controlling
> > the behavior as the args to that function.
> 
> First, M-n is not about completion, it is about minibuffer history.
> Completion functions use the minibuffer, so the minibuffer history
> affects them, but they are not the only ones affected.

That's true.  My concern is mainly with the
completion case.  After all, the default value
of `minibuffer-default-add-function' is a
function that only applies to minibuffer
reading with completion.

And I spoke of `completing-read', as a shortcut
to talking about reading with completion more
generally (`read-extended-command-1',
`read-shell-command', `read-file-name-default',
`minibuffer-complete-defaults',...).

> The documentation of completing-read and of read-from-minibuffer already
> state that DEFAULT is added to the "future history":
> 
>      The argument DEFAULT specifies default values to make available
>      through the history commands.  It should be a string, a list of
>      strings, or ‘nil’.  The string or strings become the minibuffer’s
>      “future history”, available to the user with ‘M-n’.

1. It isn't just added to the "future history".
   It's the base (prefix) of the future history.

2. The problem identified has nothing to do with
   putting DEFAULT (single or list) into the
   future history.  That's by definition always
   useful, because it assumes that the programmer
   provides a reasonable set of such defaults.
   Control by human design.

> What is missing here, it seems, is the hint that this addition can be
> controlled, 

Not if by "this addition" you mean the addition
described in that doc you quoted, which is just
addition of DEFAULT.

But if by "this addition" you also mean the
addition that's not mentioned there - automatic
addition of the completion domain, then OK -
THAT addition can (only?) be controlled by var
`minibuffer-default-add-function'.  That var
has no effect on the addition of DEFAULT.

> among other measures, via minibuffer-default-add-function,
> and the documentation of that variable where the minibuffer history is
> documented.  Is that what you are asking for, or is there anything
> else?

See above.

1. I'm asking whether this feature (addition
   of completion domain automatically)
   shouldn't be revisited, maybe even removed,
   and at least default to OFF.

2. If that revisit is NOT to be, then I'm
   asking that the doc at least (a) point out
   that this automatic behavior can be
   problematic, and (b) tell users how to
   (i) turn it off and (ii) control it a bit
   if not turned off.  That control can include
   limiting the size and sorting the elements
   to be added.

> > I mentioned that I think it would help to
> > make some changes to both the Elisp doc and
> > the user doc.
> 
> Why would Emacs users need to know about this?  The mechanism to
> control what and how is added to minibuffer history is not user-level
> information; users cannot use it to their benefit.

Users can defvar the variable value to nil (or
a different function).  That won't help with
some minibuffer reading that binds its own
function, but it can help in many cases.

Users can at least be aware of what's happening,
so they won't wonder "WTF?" when they see odd
`M-n' insertions.  Knowing that, they'll know
enough to stop repeating `M-n' when they get a
few such odd insertions, instead of maybe
thinking that they were one-offs.  IOW, by
being aware of what's happening they can maybe
give up on meaningless `M-n' input sooner.

> > In effect, the heads-up tells a user that
> > when prompted for input with completion,
> > in some cases the "future history" of
> > defaults is effectively useless.  And it
> > doesn't hurt to tell users why: _all_
> > possible domain completions are included,
> > possibly in a meaningless order.
> 
> If we think that future history in some case is useless, TRT is to
> change the code so that it ceases to be useless,

100% agreement.  That's why that was my first
priority request.  Fix this and there likely
will be no, or little, need to change the doc.

I didn't assume that others would agree that
the behavior is harmful.

I thought too that I might just be missing
something.  The behavior seems so bizarre to
me that I don't understand why it would have
been adopted.  Benefit of the doubt made me
guess I'm maybe just misunderstanding.

> not to document that it is useless.  IOW, we
> don't document our own bugs, we prefer to fix
> them.

Agree fully.  Let's see what Stefan, Juri,
etc. think about the actual problem, leaving
doc out of it till the behavior question is
settled.

> So no, we won't be telling this in user documentation.  If we decide
> that this behavior of "C-h v" and similar commands is not useful, we
> should change it to be more useful.

Great.  SGTM.