Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 18 Apr 2023 16:45:33 +0100
> Cc: dmitry@gutov.dev, emacs-devel@gnu.org
> 
> > > It's not a "must".  Eglot can work without it.  The problem happens in
> > > ElDoc and doesn't affect its interface.
> >
> > Then what Dmitry said about Eglot 1.15 being dependent on that change
> > in ElDoc is not relevant to the issue at hand, which is whether Eglot
> > 1.15 could be bundled with Emacs 29.1.
> 
> It is quite relevant.
> 
> Eglot 1.15 depends on many other things in ElDoc.

It isn't like ElDoc is not available in Emacs 29.

> That particular bugfix might not -- or might indeed -- included,
> depending on what other non-bugfix things Eglot will require of
> ElDoc at the time.  ElDoc is now 1.14 but it could be 1.15 at the
> time motivated by Eglot 1.15/16/17.

My point is that Eglot and any other core package will do its users a
favor if it deliberately and consistently makes a point to depend on
as few _new_ features of other core packages as possible.  If you
disagree with that, then we will have to agree to disagree, because
for me this is a very basic issue with core packages and the future of
Emacs.

> And even in ElDoc 1.14 there are already things (the :echo display
> option) that are not in Emacs 29. And Eglot 1.14 directly depends
> on those things.  It relies on them to do a good job.

Which IMO is not a good thing, not unless Eglot 1.14 has fallbacks for
when these features are not available.

But if that is impossible or impractical for some reason in this
particular case, then it simply means that users of Emacs 29 will be
unable to upgrade Eglot without also risking less stability due to
upgrading the dependencies.  Again, if you don't think this
"dependencies hell" is a Bad Thing in general, we will have to agree
to disagree.

> I would think that if you oppose a bugfix backport you would also
> oppose a _feature_ backport, which usually is (and is indeed in
> this case) a much more complicated, "scary", bug-prone development.

I consider each case of a request to backport on its own merit.  So
conclusions such as the above, which don't consider the specifics of
each change, but instead go by some abstract principles, are not what
I usually make.

> > Understood.  My point is that if you want Eglot users to be able to
> > upgrade to a newer versions, you need to have compatibility layers, to
> > avoid the need to upgrade too many other packages, which might hamper
> > stability.  Otherwise, we cannot in good faith recommend that users of
> > stable Emacs update their core packages without a second thought.
> 
> Yes, and this is why each released version of Eglot specifies exactly
> the _released_ versions of its dependencies that it depends on.

You are missing my point.  I'm not talking about dependencies with
other packages.  My point is not about other core packages, it's about
Emacs itself and its stability as a whole.  Users of a stable release
of Emacs can, and usually do, expect their entire Emacs configurations
to be stable, and telling them to upgrade packages without any clear
indication of their stability goes against that.  Yes, requiring
specific versions of the other N packages will minimize breakage due
to incompatibilities between those packages, but what about unintended
consequences, regressions, etc.?  Suppose Eglot 1.14 brings with it
some package whose updated version has a bug -- why should a user who
wants to have a stable Emacs environment to risk this?

> > Updating a package P1 should require update of as few packages P2...Pn
> > as possible.  Ideally, none at all.
> 
> And very often that does happen, I suppose.  Not every Eglot release
> _requires_ installation of new versions of its dependencies.  But some
> do.

I'm asking whether you make an extra effort to avoid such requirements
whenever possible, even if that would call for extra work on your
part?  I hope you and other package developers do, because otherwise
proliferation of core packages would be a very bad deal for the future
of Emacs, which traditionally is perceived as an extremely stable
platform.

> > Users should be able to decide
> > whether they want or don't want to update any single package without
> > also needing to decide whether they are okay with updating half a
> > dozen of others.  This should be our goal, because otherwise updating
> > a package will be unsafe if you use any Emacs except master (and thus
> > don't care much about stability anyway).
> 
> I don't know how you can meet that goal in general.

If you at least think we should try, then we have something to work
with.  Even if the ideal of having to upgrade no package is
unattainable, bringing the number close to zero is a worthy goal.

> We should indeed work to minimize dependencies and do things that
> don't affect interfaces and don't require changes other's
> interfaces.  As much as possible, I agree.  But in general programs
> rely on other programs.  Dependencies exist.  Like in many other
> package managers, users should be presented with the consequences of
> their wishes, when that is feasible and when we can do so without
> breaking their configs.

I'm saying that we should make an extra effort to avoid that, not
accept dependencies without a "fight".

> I propose two main sets of :core packages to start with.
> 
> Set 1 - :core packages that have always been core, i.e. they started
> their life in the code
> 
> Set 2 - :core packages that started their life somewhere else
> (GNU ELPA, Github, etc), were installable through ELPA interfaces
> and now are :core (which means Git-versioned in Emacs.git but
> still installable via ELPA).  Two known such packages are Eglot
> and Use-Package.  Both depend on _other_ :core packages.  Eglot
> on many of these, Use-Package only on "bind-key", which is
> also new, but didn't seem to be installable independently
> before Use-Package appeared.
> 
> This isn't the end of the analysis, of course.

I'm not sure history is the aspect that distinguishes them.  An
important aspect is how "low' is the functionality provided by the
package.  Some packages provide infrastructure -- those affect many
other places by definition.  Others are applications on which nothing
else depends.  Perhaps these are the important factors, I don't know.

> - That members of set 1 shouldn't be upgradable "willy nilly" to
> maintain exact backward-compatibility.
> 
> - That members of set 2 should be upgraded in much easier fashion
> because that's what guarantees that people's configs already
> doing so won't break.

That is completely irrelevant for this discussion, IMO.  It is up to
the users whether to upgrade and how aggressively.  We don't know
enough about the configurations, the environment, and the usage
patterns of the users, we can only give them information -- in this
case regarding stability of each available version -- and let them
make the conclusions as they see fit.  We will never be able to second
guess them well enough.