Re: Request to add Package to GNU ELPA

[Jonas Bernoulli]

Philip Kaludercic <philipk@posteo.net> writes:

> Jonas Bernoulli <jonas@bernoul.li> writes:
>
>> Hello,
>
> Hi, hope I can add a few useful comments here.
>
>> I would like to request that "package.el" be added to GNU Elpa as a
>> "core" package.  This would make new features available to users who are
>> stuck on older Emacs releases and it would even bring us one step closer
>> to being able to make backward incompatible changes in the future.
>>
>> Some examples for why that would be desirable:
>>
>> - A recent addition that could be useful is the new "package-vc.el"
>>   (which I think should be distributed as part of the `package' package,
>>   but it could also be distributed as a separate core package).
>
> This will be difficult, since package-vc depends on the new "vc-clone"
> function, which would also have to be provided on ELPA to work.

The simplest way to deal with that, while still making the latest
Package available to users of older Emacs releases, would be to *not*
include package-vc.el in the Package package.

Going one step further we could make package-vc available as a separate
package.  That would not be of much use *now*, but future improvements
would then be available to users of Emacs 29.

I had a look at vc-clone and a few vc-*-clone.  They seem trivial
enough to backport, either as part of Compat or in package-vc.el.

> This
> seems to me to be a general issue with proposing to create new core
> packages.  When working in emacs.git it is easy and usually
> uncontroversial to make use of whatever is the current state-of-the-art
> when it comes to Elisp.  The Compat library on ELPA can help here, as it
> already does in a few cases (ERC and Python, and it has been discussed
> for project.el).

Thanks for creating Compat!  It certainly makes things easier for
maintainers of external packages.

> All of this touches on the general topic of how Emacs should be
> developed: Small and to be extended, therefore of little use OOTB or big
> and feature-full, but slow to be updated.  IIUC a compromise could be
> achieved by having two versions of Emacs that would respectively attempt
> to satisfy the two aspirations.

This doesn't sound like a good idea to me because I doubt that it would
have the benefits you seem to expect.  A sizable portion of an external
package would choose to use the slow-moving Emacs.  So the maintainer of
the package would have to continue to support that, similar to how they
now have to support older releases.  So even though there would be a
fast-moving Emacs, they still would not be able to take full advantage
of the new/improved features it provides.  The situation would be
basically the same as now; but not supporting the slow-moving Emacs
would be even harder to justify than dropping support for Emacs 25.1.

> There was some work in that direction a
> few years ago[0], but I don't know if anything has happened since.
> Perhaps pushing for this more general solution would help solve this
> particular problem?
>
> [0] https://git.savannah.gnu.org/cgit/emacs.git/log/?h=elparized-core

I remember that there was some talk about moving in that direction, and
I am in favor and have been patiently waiting for someone to make
another push in that direction, but things indeed seem to have fizzled
out.

Meanwhile making certain packages available on a case by case basis does
not seem like a bad idea to me.  It doesn't mean that we *commit* to
making "most" packages available like that eventually, so even those you
are opposed to that idea might be able to agree to doing it selectively.
Doing it for a select few packages, allows us to gain some experience,
whether that be bad or good.

>>   That isn't just useful for users who like to live on the edge.  For
>>   example, if a package drops support for an old Emacs release, then
>>   a user who is stuck on that release, may wish to keep using the last
>>   release of the package that still supports that Emacs release, and
>>   package-vc would allow them to do just that.
>>
>> - I don't use Package myself, so I am not fully aware of all the quality
>>   of live enhancements that surely have accumulated over the years.  But
>>   I am aware of some small missing features, that would be beneficial to
>>   users of older Emacs releases.
>>
>> - For example, I think it would be nice if the columns displayed by
>>   `list-packages' were customizable.  A user might want to increase the
>>   width of the "Version" column, so that not every GNU-devel ELPA version
>>   is truncated, 
>
> A more general solution to this problem would be if tabulated-list-mode
> could dynamically resize a column, or at least shrink it possible.

That would indeed be nice ... and would make tabulated-list-mode another
candidate for GNU ELPA. ;P

> Also on another topic, I am still uncertain as to the status of the
> -devel archives.  I see them primarily as a means for developers to
> check the status of a package pre-release or to make sure that the ELPA
> build-system is working the way they want it to (e.g. when building
> manuals, news or README files).  They are not advertised anywhere, and I
> don't think they are made for general consumption -- or rather I don't
> think they should be, especially with package-vc there is a much better
> way to track development and make it possible to contribute changes
> upstream.

Most packages still get installed through Melpa these days and that
means installing the latest commit from the development branch.  (The
[Non]GNU-devel ELPA variants probably exist in part to satisfy users who
are used to having access to the development version through "regular"
package i.e., without using package-vc.)

In my experience that seems to work surprisingly well; I can't really
remember any regressions that were not fixed almost instantly, but that
does of course depend on what packages one has installed.  I am sure
little regressions happen all the time, but they get fixed so quickly,
that any one user usually doesn't notice.  In my experience, using the
development versions of some quality packages, is no different from
using the development version of Emacs itself.

But I agree that it would be better to release more often (and yes, I
am guilty of not doing that myself), and tend to agree that it would be
better if, in the long run, the community moved towards relying more on
released versions.

At the same time I don't think that the fact that we are not there yet
is due to "mistakes".  Emacs/Elisp is different from other "languages",
in that everything runs in the same process, and that is at least one
of the causes of the status quo.

We cannot really use multiple versions of the same library in different
parts of Emacs, in the same way one Python or Rust application could
use xyz-v1 and another, related but independent, application could use
xyz-v2.  If one package depends on xyz-v2 but there were incompatible
changes in that release and, another package has not been updated yet to
support xyz-v2, then those two packages cannot be used at the same time.

Again, that does not seem to happen very often, and I think that is
partially the case because the hold out package can be quickly updated
to support v2 on its development branch, *without* also having to do a
release, which the maintainers might not wish to do due to where in its
own release-cycle the package happens to be.

>> - I should also mention that my main motivation for pushing for this
>>   now, is that I would like to improve version handling.  That is a
>>   whole other can of worms, so I do not wish to discuss it just yet,
>>   to avoid distracting from the topic at hand, but I thought I should
>>   at least mention it.  You might very well end up being against my
>>   suggestions regarding versions strings, once I present them, but that
>>   should not be cause to oppose the change requested here as well.  Even
>>   if my suggestions regarding version strings are ultimately rejected, I
>>   still think it would be a good idea to add `package' to GNU ELPA, for
>>   the other reasons presented here.
>
> This implies that packages might themselves depend on newer versions of
> package.el -- which I think one should put some thought into before
> anything is done.

How so? (Further down I suggest making some popular packages temporarily
depend on Package from GNU Elpa as a means to get that installed as
widely as possible, but that doesn't mean that those packages actually
*need* a newer Package.)

> How can you e.g. change the way versions are handled
> in a way that people with older versions of package.el could still
> handle the change without confusion?

You can't, and that is exactly the point I was trying to make.  Certain
things (including, but not limited to how version strings are handled)
are effectively set in stone, unless certain packages/features are made
available to users who, for whatever reason, stick to an old Emacs
release.

> (This will be an issue for at
> least a decade to come, knowing the pace at which users get access to
> newer versions of Emacs).  You mentioned updating the ELPA protocol a
> few months back, and I believe a few points of interest were collected
> in that thread.  It seems to me these issues are all intertwined.

Yes indeed, they are all intertwined, and I am trying to make the point
that we have to stop muddling along once in a while, and make a breaking
change, even if the transition is rough and inconveniences users for a
while.

> With Emacs 29, `package-refresh-contents' now calls a Hook called
> `package-refresh-contents-hook'.  This was initially introduced so that
> package-vc can download the "elpa-packages.eld" file, but it could also
> be of use here as well -- but it wouldn't help anyone with Emacs 28 or
> older...  I could imagine a more hacky approach based on carefully
> placed advice, but only as a last resort.

We cannot travel back in time to fix past mistakes/omissions that are
holding us back in the presence.  All we can do now is to pay the price
and try harder to be forward compatible this time around.  Or live with
the limitations forever.  Or wait some more until they become unbearable
and a fix more urgent; but that would increase the odds of messing it up
again.

>> Addressing (1) is harder, and I don't think it possible to do so in a
>> way that guarantees that not a single user would end up seeing an error
>> due to an incompatible change.  On the other hand "archive-contents"
>> comes with a version field, and if we bump that, we at least get a
>> meaningful warning: "Package archive version 2 is higher than 1".  This
>> doesn't say that the solution is to install `package' from GNU ELPA, but
>> it should be enough to feed into a search engine to get to that answer.
>
> The upgrading procedure for archive-contents has never appeared to me to
> be very robust.  Perhaps it would be better to introduce a second file
> (archive-contents.eld) with a more flexible format?

An effort was made to provide an upgrade procedure -- the
"archive-contents" were prefixed with a version number.  But then
package.el was added to Emacs and we stopped to distribute it
separately.  Apparently we didn't realize at the time that this only
allowed us to tell the user "the archive and tool are not compatible,
deal with it".

We could continue to distribute "archive-contents", which continues to
use version 1, and add "archive-contents.eld" which uses version 2.
But that would do nothing to get users to install a forward-compatible
version of Package (in the sense that once it becomes incompatible, it
will provide a smooth upgrade path to the new version).

>> Additionally, we could get many users to install `package' from GNU
>> ELPA, by making a few popular packages explicitly depend on a `package'
>> version that is available from GNU ELPA for a few months.
>
> Again, a few months wouldn't be enough to solve the issue.  But again as
> well, Compat could help by adding package as a noop dependency.

Much like your "archive-contents.eld" suggestion, doing this would not
*solve* the issue.  Unlike that, it would however significantly reduce
the number of users who would be negatively affected by the upgrade.
(We could do it for a few years instead of months to further decrease
the number of affected users.)

I love Compat, but I don't think it is the right hammer here (except
maybe for the part where it could provide vc-clone et al.).

Maybe it helps to make it explicit what is going to happen to users
if/when all the *ELPA start to expect that package.el support
"archive-contents" v2.

- The user runs "M-x package-refresh-contents" and is told "Package
  archive version 2 is higher than 1".

- They ask a search engine what that means.  We could create a few blog
  and forum posts before hand and give the search engines a few days to
  index them, providing the following instructions:

  (We should invest some time to investigate how to make this as smooth
  as possible, but basically:)

- 1. wget https://...package-2.0.0.tar
  2. tar -xf package-2.2.0.tar -C ~/.config/emacs/elpa/
  3. Restart Emacs and run package-refresh-contents again.

     Cheers, 
     Jonas