Re: ELPA submission: plz-see

[Adam Porter]

Hello Eli,
> > Date: Sat, 4 Nov 2023 08:30:47 -0500
> > Cc: emacs-devel@gnu.org, tomas@tuxteam.de
> > From: Adam Porter <adam@alphapapa.net>
> >
> > > The claim is that fetching a URL with our built-in primitives is
> > > too slow.
> >
> > Forgive me, but that is not the claim.  There are numerous problems 
> > experienced with using url.el "in anger," as well as with other HTTP 
> > client libraries for Emacs.  Stefan K. was kind enough to link one of my 
> > posts about this issue a few years ago, when plz.el was earlier in its 
> > development (the issues mentioned having been solved now)[0].  Briefly, 
> > the problems include unreliability (e.g. callbacks being called multiple 
> > times or not at all, timeouts that don't happen), very awkward API (e.g. 
> > having to let-bind often-undocumented variables to do basic HTTP 
> > operations)
>
> These all sound like bugs, and I'd be surprised if bugs were cited as
> a reason to go to a completely different implementation, and one that
> is based on an external program on top of that.  Bugs should be fixed;
> they should not cause us to throw away our code just because it has
> bugs.  Especially since those bugs cannot be so bad, as our
> implementation of network communications does work: I'm sending this
> mail using it, for example.

Ideally, of course, I would agree with you.  In practice, experience has 
shown that fixing these bugs is not so simple.  As even Lars said years 
ago[0]:

    "And the more I looked at this, the more apparently it became 
[clear] that Emacs' network stuff should just be totally reimplemented, 
because it's so full of...  oddnesses geared toward processes.  And 
processes and network connections are very different things.

    "So it's a Big Project.  :-/"

The file process.c is known for being hard to debug, especially with 
regard to these kinds of problems.  I feel fortunate that I was able to, 
with the help of a few other people, work around them in plz.el to 
resolve issues with reliability.

> > and, perhaps also, inferior performance compared to a 
> > curl-based implementation (curl being implemented in a separate process, 
> > in C).
>
> So is performance an issue or isn't it?

As others have noted here and on the bug tracker, Emacs's built-in 
network processes seem to have hangs with regard to DNS, TLS, etc, and 
having the external curl process handle those issues avoids those 
problems.  Some of this discussion happened three years ago, where even 
Richard suggested that using an external process like curl to solve it 
seemed reasonable.[1]

> > I confess to feeling a bit annoyed at having to re-justify the inclusion 
> > of plz.el in GNU ELPA (or, at least, I feel like I'm having to do so). 
>
> This is a misunderstanding.  I never said anything about inclusion of
> the package in ELPA or the need to justify it.  I responded to the
> suggestion to integrate curl with the Emacs core:

> > > > > As has been said on emacs-devel several times over the years, the next 
> > > > > significant step in this field would most likely be for Emacs to 
> > > > > integrate libcurl directly.  Maybe someday that will happen...
>
> Please don't misrepresent what I said and why, because the above
> sounds like you are saying I questioned your reasons for developing
> the package or asked you to explain why it was needed.  I did not.

Forgive me, then, for misunderstanding you.  It's never my intention to 
misrepresent.

> > Having said all that, of course it would be best if Emacs's built-in 
> > library were the best one.  But as the author of url.el himself has 
> > admitted, it has some fundamental issues which are not easy to solve, 
> > and his own branch related to improving it hasn't been touched in 6 
> > years[3].  And so far, no one has stepped up to integrate libcurl or 
> > similar.
>
> If url.el has fundamental problems, they can be fixed by thorough
> modifications or even with a completely new line of APIs which don't
> have those problems.  That is not the issue that bothers me here; the
> issue that bothers me is the idea that instead of improving our own
> implementation, we should use an external program.  It should be clear
> to anyone who understands how this stuff is integrated into the core
> that basing this on external programs has serious issues of its own,
> because Emacs is not a garden-variety program, it needs to support
> many low-level hooks into the communications level which are hard or
> even impossible with external programs and libraries.  I very much
> doubt that going this way will ever be able to support everything we
> support now with network "processes" and applications based on them.

Of course, I don't disagree with you.  The fallback to an external 
program is a pragmatic decision, one which plz.el is in a long line of 
similar packages to make.

> It is indeed unfortunate that we don't have anyone working on these
> issues.  I still hope that someone will step forward soon enough.
>
> > But the name is of very little importance, and surely no more
> > difficult to remember than an API, of which there are innumerable
> > ones in Emacs.
>
> My original request to find a better name was for plz-see, not for
> plz.  So once again, please don't misrepresent what I said and why.

Indeed, I misunderstood.  Thanks for patiently clarifying what you meant.

--Adam

0: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22921#19
1: https://lists.gnu.org/r/emacs-devel/2020-12/msg01456.html