bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 21 Apr 2023 03:50:52 +0300
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, 62720@debbugs.gnu.org,
>  philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 20/04/2023 17:25, Eli Zaretskii wrote:
> 
> > What I miss in this description is something a bit higher-level: the
> > list of all the ways/commands/methods people use to install and
> > upgrade the packages.  We must have this list before our eyes to make
> > sure we review all the behaviors related to these activities, if we
> > want them all to eventually behave consistently.
> 
> Why list the commands people use to install packages if we're talking 
> about upgrading them?

I said "to install and upgrade", not just "install".

And I do think we should also consider commands that install packages
because they are part of this issue, and in particular, should be
consistent with the upgrade commands.  Also, package-install was the
original cause of this thread.

> Hopefully we'll decide that 'package-install' 
> won't upgrade packages because it hasn't done that in the past.

But it does upgrade non-built-in packages, doesn't it?  And at least
João (and I think others as well) expected it to upgrade Eglot even
though it is now built in.

> > E.g., you don't mention any commands invoked from the package menu.
> 
> The command which people use to upgrade packages from the package menu 
> is called package-menu-mark-upgrades (bound to 'U'), mentioned here a 
> few times. I also said that I hope it's *the* main way people use to 
> upgrade packages.
> 
> The others would be package-update and package-update-all (can we please 
> do the rename now? I know they've been in for a year, but just as this 
> very discussion has shown, most people weren't even aware of their 
> existence).

If the list you provided covers all of the relevant commands and
functions, fine.  Still, for completeness' sake, we should have the
higher-level description as well, as a framework against which to
examine the proposed solutions.  And I'm still not convinced that we
covered all the relevant package-menu commands.

> >> - package-upgrade (nee package-update) doesn't upgrade builtin packages
> >> that never been upgraded before. It's a bug. Hopefully not too hard to fix.
> > 
> > I'm okay with adding the same prefix argument to package-upgrade,
> > which would then allow upgrading a built-in package.  IOW, a change
> > similar to what we did in package-install -- provided that the change
> > is safe enough to go into Emacs 29.
> 
> If we agree it's a bug, why don't we just fix it?

Precisely because, as with package-install, this is a bug for some and
a feature for others, depending on whether people do or don't want the
built-in packages to be upgraded by default.

> 'package-update' is an 
> interactive function which itself it called from only one place: 
> 'package-update-all', and since the plan is to improve both, we can make 
> sure they only do what we ask of them: package-update will upgrade 
> builtins when invoked directly, and package-update-all will upgrade them 
> only when the builtin has been upgraded before (making it not a builtin 
> anymore), or a new user option is set.

This is one possibility, and it might make sense to some users.  But I
don't think we can be sure it will make sense to an overwhelming
majority of the users.

> > But note that AFAIU there's a relatively easy workaround for this:
> > install the later version of the package manually, just once.
> 
> It's indeed a workaround that exists, and if we were satisfied with it, 
> we would have closed this bug as "wontfix". But it's not a very friendly 
> way to do that (the user must hunt for the appropriate version in the 
> packages menu), and it's very poorly scriptable (you won't be able to 
> provide an easy and readable snippet for the user to evaluate or put in 
> their init script).

For Emacs 29, we don't have the luxury of rejecting slightly awkward
workarounds.

> >> - package-menu-mark-upgrades and package-update-all don't upgrade them
> >> either. That's not necessarily a bug, but a problem nevertheless. A new
> >> user option could help.
> > 
> > A very relevant question is: can this wait till Emacs 30?
> 
> If you ask my opinion, then I already wrote it.

I'm asking everyone who reads this.

> Should we leave bugfixing and easy upgrading of builtin packages to 
> Emacs 30? I suppose we could. Let me know if it's your foregone 
> conclusion, then I'll just walk away now.

It isn't a forgone conclusion, because no one has yet shown the code
to fix the issues you mention.  If the code is safe enough and doesn't
risk breaking some reasonable workflows, it could be considered for
Emacs 29.

> >> - The current fix (commit 580d8278c5f48) is not comprehensive WRT to
> >> Joao's scenario because use-package-ensure-elpa short-circuits when it
> >> find that the package is installed ('package-installed-p' returns t). So
> >> (use-package eglot :ensure t) does not upgrade Eglot even when
> >> package-install-upgrade-built-in is t.
> > 
> > I don't think (use-package eglot :ensure t) should automatically
> > upgrade built-in packages.
> 
> I don't think it should, either.
> 
> But IIUC that's the scenario 580d8278c5f48 was supposed to make possible.

No, it was supposed to allow the user to invoke package-install in a
way that upgrades built-in packages, something that doesn't happen by
default.  IOW, it was a partial solution to the original problem which
started this discussion, see

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#5

> > We could make it do that if
> > package-install-upgrade-built-in is non-nil -- again, if such a change
> > could be safe enough.  If not, then the same workaround as for
> > package-upgrade would do here, I suppose?
> 
> What workaround would that be? use-package is not invoked interactively 
> -- there is no prefix argument to pass.

The workaround is to manually install the package from ELPA, once,
using "C-u M-x package-install RET".

> >> - package-install-upgrade-built-in is not nuanced: if we suggest the
> >> users to set it to t, that can result in making _all_ builtin packages
> >> upgradable with 'package-install'.
> > 
> > It should be set to t only by users who indeed want all built-in
> > packages to be updated.  That's why the default is nil.
> 
> Sounds unnecessarily dangerous.

Which is why it shouldn't be the default.

> >> Whereas I think we originally only wanted that for Eglot and maybe
> >> for use-package.
> > 
> > "We" never did want that.  João did, for obvious reasons, but that was
> > never my intent.  The issue is indeed more general: what should
> > package-install and package.el in general do with built-in packages
> > for which a newer version is on ELPA?
> 
> It could continue doing what it's done before: when a package is already 
> installed, abort. For upgrading, we should recommend commands with 
> "upgrade" in their names.

If people agree with that, I don't think I'll object.  But this is in
a sense a breaking change: package-install will only install, and
thereafter users will need to use package-upgrade.  Some might dislike
such behavior changes.  And we will need to make sure that all the
available methods of "installing" do not "upgrade", for consistency.

> > We don't yet know the answer to
> > that, and no hope of obtaining one before Emacs 29.1 is released
> > (barring any calamities).
> 
> I'm not sure how we're going to get the answer to that question after 
> 29.1 is released, too. Similar to that thing with treesit modes and 
> auto-mode-alist.

Time will tell.  If we have no significant new information and data
points by the time the release of Emacs 30 is on the table, we will
have to use our best judgment at that time.

> >> For this and other minor reasons I would suggest reverting
> >> 580d8278c5f48.
> > 
> > Not going to happen, not unless someone comes up with a better
> > solution that is much better and still safe enough.  Personally, I
> > don't believe such a solution exists, since we don't really know the
> > answer to the above question.
> 
> Could you specify which problem it's currently solving? Some particular 
> scenario.

The scenario which started this bug report, see the message whose URL
I mentioned above.  IOW, we now allow users to explicitly request that
package-install includes built-in packages in the list of candidates,
and will therefore allow to upgrade them.

> >> But I suppose we could also try to make it more granular (e.g. turn
> >> the boolean option into a list). I'm not sure it's a good direction
> >> overall, however.
> > 
> > I don't think we should go in this direction, precisely because we
> > don't know it's a good one.
> 
> If we made it more granular, its use could become a reasonable option, 
> then we'd be able to get some feedback from people who end up using it. 
> I don't know what kind of people that will be, though.

It could be a reasonable option, or it could be a nuisance because of
a gratuitous proliferation of too-granular options.  To make a good
decision, we need data and experience we don't have yet.  So let's not
rush into solutions about which we are uncertain.  We can always make
a boolean option have more values in the future, this kind of change
is usually easy and backward-compatible.

> >> - The functions are propose to fix are "leaves": those that are supposed
> >> to be used interactively, without (many) known callers in-tree. E.g.
> >> fixing package-update-all shouldn't affect some other part of Emacs as
> >> much as changing package-install could.
> > 
> > Experience until now in this thread indicates otherwise: almost every
> > suggested change touched low-level code with many callers.  It isn't
> > an accident that arriving at what was finally installed took so many
> > iterations for such a simple change.
> 
> You might be correct at that, but note that I'm suggesting a slightly 
> different set of goals than what had been discussed here before. So the 
> changes will likewise need to be different.

It's possible.  I said what I said not as an up-front decision, but as
a warning: we should not raise our expectations too high.