bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 21 Apr 2023 13:19:39 +0300
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, 62720@debbugs.gnu.org,
>  philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 21/04/2023 09:37, Eli Zaretskii wrote:
> 
> >> Why list the commands people use to install packages if we're talking
> >> about upgrading them?
> > 
> > I said "to install and upgrade", not just "install".
> 
> I listed the upgrade commands.

Yes, and I think we should consider both install and upgrade commands.

> >> Hopefully we'll decide that 'package-install'
> >> won't upgrade packages because it hasn't done that in the past.
> > 
> > But it does upgrade non-built-in packages, doesn't it?
> 
> Apparently not. I didn't remember whether it does, and I deduced that it 
> does just from reading this discussion previously, but it does not.
> 
> E.g. just now pressing 'U' after 'M-x list-packages' showed me that I 
> have available upgrades for a lot of packages. But still if I evaluate
> 
>    (package-install 'sml-mode)
> 
> where sml-mode is one of said packages, I just get the message:
> 
>    ‘sml-mode’ is already installed
> 
> > And at least
> > João (and I think others as well) expected it to upgrade Eglot even
> > though it is now built in.
> 
> I think he wants that because this way (package-install 'eglot) and 
> (use-package eglot :ensure t) could match the behavior of Emacs 28 with 
> an empty init directory. Backward compatibility and all that.

But if, with older Emacsen, package-install would refuse to update to
a newer version of Eglot if _some_ older version of Eglot is already
installed, then where's the problem with the default behavior of
package-install? it behaves exactly like in previous versions of
Emacs.  And why is this a problem for users of Eglot, if they couldn't
use package-install more than once for Eglot anyway?

Something is amiss here.

> But I think that's questionable, semantically. Given that Eglot is 
> already "installed". Though, of course, one could argue that a bundled 
> package is not exactly installed, but then we should change what 
> 'package-installed-p' does as well. And think hard before doing that.

I'd question why we have two commands instead of just one, but that's
probably water under the bridge at this point.

> >>>> - package-upgrade (nee package-update) doesn't upgrade builtin packages
> >>>> that never been upgraded before. It's a bug. Hopefully not too hard to 
> >>>> fix.
> >>>
> >>> I'm okay with adding the same prefix argument to package-upgrade,
> >>> which would then allow upgrading a built-in package.  IOW, a change
> >>> similar to what we did in package-install -- provided that the change
> >>> is safe enough to go into Emacs 29.
> >>
> >> If we agree it's a bug, why don't we just fix it?
> > 
> > Precisely because, as with package-install, this is a bug for some and
> > a feature for others, depending on whether people do or don't want the
> > built-in packages to be upgraded by default.
> 
> I'm having a hard time imagining someone evaluating (package-upgrade 
> 'eglot) without intention to upgrade it to the latest version. Or 
> invoking it interactively with same argument, expecting a different result.

In interactive invocation, package-upgrade calls completing-read with
its 4th argument non-nil, so you cannot select a package which is not
in the collection returned by package--updateable-packages.  What I
meant above is to allow that collection to include built-in packages
as optional behavior.  I just tried invoking package-update for ElDoc,
and I get "No match" after typing "eldoc" to its prompt, although
eldoc version 1.14.0 is in the list presented by list-packages as
"available".

> >> 'package-update' is an
> >> interactive function which itself it called from only one place:
> >> 'package-update-all', and since the plan is to improve both, we can make
> >> sure they only do what we ask of them: package-update will upgrade
> >> builtins when invoked directly, and package-update-all will upgrade them
> >> only when the builtin has been upgraded before (making it not a builtin
> >> anymore), or a new user option is set.
> > 
> > This is one possibility, and it might make sense to some users.  But I
> > don't think we can be sure it will make sense to an overwhelming
> > majority of the users.
> 
> Hence the user option?

Which one?  Are you suggesting to add a new one?  If so, why not use
the one we already added, package-install-upgrade-built-in?

> But okay, this particular addition, though trivial, we could probably 
> postpone until Emacs 30, or even avoid adding at all. It is indeed not 
> obvious that people will really need it, although

If by "this particular addition" you mean to allow package-update to
update built-in packages, then I thought adding that for consistency
with package-install was one of your main bothers?  Or what am I
missing?

>    (setq package-upgradable-builtins '(eglot use-package))
>    (package-upgrade-all)
>    ;; or M-x package-upgrade-all
>    ;; or 'U' in the list-packages menu
> 
> seems like a plausible scenario for a certain kind of user.

Why not treat the fact that some version was already installed from
ELPA as an indication that the user wants this?

> Because 
> package-upgrade does not have a menu entry, or a button anywhere, 
> whereas package-upgradable-builtins can be altered from the Customize UI.

Maybe marking a package in the list for update could be interpreted as
"upgrade that, no questions asked", and we will need no user options?

> >>> We could make it do that if
> >>> package-install-upgrade-built-in is non-nil -- again, if such a change
> >>> could be safe enough.  If not, then the same workaround as for
> >>> package-upgrade would do here, I suppose?
> >>
> >> What workaround would that be? use-package is not invoked interactively
> >> -- there is no prefix argument to pass.
> > 
> > The workaround is to manually install the package from ELPA, once,
> > using "C-u M-x package-install RET".
> 
> That's not the use-package workflow.

The use-package workflow should perhaps get a separate and different
solution.

> >>>> Whereas I think we originally only wanted that for Eglot and maybe
> >>>> for use-package.
> >>>
> >>> "We" never did want that.  João did, for obvious reasons, but that was
> >>> never my intent.  The issue is indeed more general: what should
> >>> package-install and package.el in general do with built-in packages
> >>> for which a newer version is on ELPA?
> >>
> >> It could continue doing what it's done before: when a package is already
> >> installed, abort. For upgrading, we should recommend commands with
> >> "upgrade" in their names.
> > 
> > If people agree with that, I don't think I'll object.  But this is in
> > a sense a breaking change: package-install will only install, and
> > thereafter users will need to use package-upgrade.  Some might dislike
> > such behavior changes.  And we will need to make sure that all the
> > available methods of "installing" do not "upgrade", for consistency.
> 
> Yeah, apparently it won't be a breaking change, or a change at all.

I'm not sure, see above.

Also, when you mark packages for update from the list presented by
list-packages, the menu entry says

  i    Mark for Install

and its help-echo says "Mark a package for installation and move to
the next line", so we already confuse "install" and "upgrade".

> >>>> For this and other minor reasons I would suggest reverting
> >>>> 580d8278c5f48.
> >>>
> >>> Not going to happen, not unless someone comes up with a better
> >>> solution that is much better and still safe enough.  Personally, I
> >>> don't believe such a solution exists, since we don't really know the
> >>> answer to the above question.
> >>
> >> Could you specify which problem it's currently solving? Some particular
> >> scenario.
> > 
> > The scenario which started this bug report, see the message whose URL
> > I mentioned above.  IOW, we now allow users to explicitly request that
> > package-install includes built-in packages in the list of candidates,
> > and will therefore allow to upgrade them.
> 
> After we fix 'package-upgrade', users will be able to 'M-x 
> package-upgrade RET eglot RET'.

This goes back to the issue of having two confusingly-similar but
different commands, as mentioned above.  I guess we should first make
up our minds what, if anything, we want to do about this.