bug#64202: [PATCH] Gnus: Add back end for Atom feeds (nnatom)

[Eric Abrahamsen]

Daniel Semyonov via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

>>>>>> Eric Abrahamsen writes:
>
>     > Daniel Semyonov via "Bug reports for GNU Emacs, the Swiss army
>     > knife of text editors" <bug-gnu-emacs@gnu.org> writes:
>
>     >> AFAIK it should work for any entry point which allows you to
>     >> define a new server or select method (Atom feeds are represented
>     >> as servers with a single group, so there is no way to create a
>     >> group without a pre-existing corresponding server).  I'll be
>     >> honest though, I only ever use this method and
>     >> 'gnus-secondary-select-methods', so I'm not really sure which
>     >> other entry points there are.
>
>     > What I usually start with is "G m", for
>     > `gnus-group-make-group'. That prompts for a group name, and also a
>     > backend. The viable list of backends includes all your
>     > currently-defined servers, plus the generic symbols for all
>     > defined backends, in this case a plain 'nnatom.
>
>     > As you point out, that doesn't work for nnatom because each group
>     > will need its own server. What *does* work is to create a group
>     > this way, then put point on the new group and hit "M-e", for
>     > `gnus-group-edit-group-method', put in the actual URL, then "C-c
>     > C-c".
>
>     > Obviously this is horribly baroque and it would be better if
>     > `gnus-group-make-group' could prompt for the URL. No matter what,
>     > it would be good to document the various ways to do this.
>
> Weirdly enough, 'gnus-group-make-group' accepts an ADDRESS argument, and
> the docstring even claims it prompts for it, but the function fails if
> you pass it an address it seems.
> However, since Gnus knows which backends require an address, and adds
> it to the name of the method when present, it shouldn't be hard to fix
> this command to actually prompt for an address when needed.
> I haven't looked deeply enough into how that function works to know
> whether it would actually work as expected in that case though.

Where do you see that Gnus knows which backends require an address? That
would be very handy. But my reading of this function is that we don't
even known that the backend is loaded until the end of the group
creation process. You could probably mildly abuse the
'request-create-group server function to further prompt the user for the
URL.

It is a bit annoying that the ADDRESS argument is in there, but not
prompted for. It looks like that's been the case for at least two
decades, though.

>     >> > Regarding your earlier question about having this backend
>     >> handle > RSS too, I'm not aware of any significant difference
>     >> between the > two beyond the format of the XML. Is that true?
>     >> 
>     >> Yes, even the XML format is very similar.
>     >> 
>     >> > If so, it seems like it would make most sense to merge the >
>     >> code. Have you looked at nnrss? It would be good to know if there
>     >> > was anything in there worth stealing for nnatom -- if one of
>     >> them > has a faster parser than the other, for instance, or
>     >> better logic > for keeping updates efficient.
>     >> 
>     >> The issue with merging the two is that nnrss saves feed data
>     >> differently (both on disk and in memory), and also represents
>     >> each feed as a group, with a virtual "server" holding all groups.
>     >> I'm not sure if it's possible/a good idea to migrate feed data
>     >> from nnrss to a hypothetical merged backend, at least not
>     >> automatically.
>     >> 
>     >> Stefan floated the idea of adding RSS support, deprecating nnrss
>     >> and creating an interactive migration command - so users who wish
>     >> to migrate will have to do so manually (which should also
>     >> potentially allow asking the user some questions if the migration
>     >> includes some non-trivial steps).
>     >> 
>     >> nnrss does do some cool stuff that nnatom doesn't, though (for
>     >> example, it tries very hard to find an RSS feed when you provide
>     >> it with a link to a website, while nnatom currently requires a
>     >> direct link to a feed).
>
>     > Right, I should have been more explicit here -- what I was
>     > thinking is what Stefan suggests: just deprecate nnrss
>     > altogether. If it does anything cool like feed discovery, just
>     > steal that code. If you're inclined to be nice enough to provide a
>     > migration function, that's a bonus.
>
> I guess I'll see how hard it would be.  First I have to actually
> implement RSS support though.

I also don't think it would hurt to start with what we've got, and
incrementally add to it after an initial deployment. The important thing
would be factoring out a nnfeed.el backend, then providing nnatom.el on
top of that. The rest could come as you get to it.

>     >> > I just subscribed to a feed with nnrss, and noticed that after
>     >> I > marked all the items in the feed as read, I couldn't re-enter
>     >> the > group and see the old items. It gave me "Can't select
>     >> group". So > that's not very encouraging.
>     >> 
>     >> Honestly, from my experience nnrss has many small issues
>     >> (although I never encountered this exact issue).  It is partly
>     >> why I developed nnatom (previously I used a hack documented on
>     >> the Emacs wiki which converted Atom feeds to RSS feeds on the
>     >> fly).
>     >> 
>     >> > If you do want to expand this to be a general "feed" backend,
>     >> we > might want to do some boring things like rename it
>     >> nnfeed.el,
>     >> 
>     >> This is the name I thought of too, and I guess if two people
>     >> thought of it independently it's probably fine.
>
>     > Good!
>   
>     >> > and add support for ridiculous things like JSON feed[0]
>     >> (why?!?). I > assume a derived backend could handle JSON feeds by
>     >> setting the > appropriate values for the `nnatom-read-*-function'
>     >> deffoos?
>     >> 
>     >> I actually attached a derivative "nnjsonfeed" backend I made as
>     >> an experiment to one of my previous messages (it doesn't work
>     >> with the current version of the patch, but it won't be hard to
>     >> fix).
>     >> 
>     >> It wasn't 100% conforming to the standard (JSON feeds support
>     >> some weird features like pagination, which can actually be
>     >> supported very well in theory by nnatom, but I didn't feel like
>     >> doing that), but it worked and it was very easy to make.
>
>     > I missed nnjsonfeed, sorry. That's great you're already working in
>     > this direction. I don't think we need to support absolutely
>     > everything it does (what would pagination look like in Gnus?),
>     > just the basics to get started with.
>
> The feed itself is paginated, meaning that the entries are spread across
> multiple files, with each one pointing to the next with a special tag.
> The design of nnatom allows for this since it doesn't necessarily
> require parsing the entire feed ahead of time, and also notifies the
> backend when a stale (not new or updated) article is reached, so the
> backend can intelligently stop (instead of trying to parse thousands of
> pages every time you update a group, for example).  These capabilities
> were actually added in anticipation of a JSON feed backend, the Atom
> parser doesn't use them at all.
>
> Gnus itself won't show pages of course, just a list of articles as
> normal, but they will come from different JSON feed pages.

I see! Sounds like a reasonable optimization for making updates faster.

>     >> > One of the awkward things about nnrss is that it's never really
>     >> > fit well into Gnus' one-server-many-groups paradigm, which you
>     >> > allude to in the nnatom Info section. Do you have any further >
>     >> ideas in that direction?
>     >> 
>     >> Well, nnatom theoretically supports this paradigm, but it doesn't
>     >> do this with Atom feeds, since it doesn't really make sense IMO.
>     >> However, there is a standardized way to include links to Atom
>     >> feeds in HTML documents, so it might be a good idea to support
>     >> adding them as servers which show any linked Atom feeds as
>     >> groups.
>
>     > It's also perfectly possible that a single website would publish
>     > multiple Atom feeds, right? Like weather.gov, for instance. That
>     > would have an added benefit of letting us simplify the server name
>     > from the full URL (https://alerts.weather.gov/cap/wa.php?x=0)
>     > which looks ugly in the Group buffer, to just eg
>     > alerts.weather.gov.
>
> Yes, that's actually what I meant, although looking at real world
> examples, it seems most websites either only link to one Atom feed in
> the "correct" way, or don't link to any at all in that way.
>
> <https://alerts.weather.gov/cap> doesn't link to any feed in the
> standard way at all, for example.
>
>     > But I guess I don't know why it would matter, really. The only
>     > practical use for wanting multiple feeds under a single server
>     > would be to set some configuration at the server level, that would
>     > apply to all feeds from that server. And at the moment I don't
>     > think there are many such settings (right?).
>
> Actually, since the various parsing functions are stored in server
> variables, all of them can be changed on a per-server basis, which is
> pretty cool.
>
> As an example, after I add support for fetching feeds contained in HTML,
> I can imagine a scenario where a user adds a website which publishes
> multiple, non-standard feeds, and fixes the parsing of all of them at
> once by providing some customized parsing functions for that specific
> server.
> This is a contrived example, but I know at least of youtube, which
> publishes slightly customized Atom feeds for each channel, so this is
> already useful.

Thanks for the background. So it does sound like one Gnus server per
remote server is the way to go.