bug#64202: [PATCH] Gnus: Add back end for Atom feeds (nnatom)

[Daniel Semyonov]

>>>>> Eric Abrahamsen writes:

    > Daniel Semyonov via "Bug reports for GNU Emacs, the Swiss army
    > knife of text editors" <bug-gnu-emacs@gnu.org> writes:

    >> AFAIK it should work for any entry point which allows you to
    >> define a new server or select method (Atom feeds are represented
    >> as servers with a single group, so there is no way to create a
    >> group without a pre-existing corresponding server).  I'll be
    >> honest though, I only ever use this method and
    >> 'gnus-secondary-select-methods', so I'm not really sure which
    >> other entry points there are.

    > What I usually start with is "G m", for
    > `gnus-group-make-group'. That prompts for a group name, and also a
    > backend. The viable list of backends includes all your
    > currently-defined servers, plus the generic symbols for all
    > defined backends, in this case a plain 'nnatom.

    > As you point out, that doesn't work for nnatom because each group
    > will need its own server. What *does* work is to create a group
    > this way, then put point on the new group and hit "M-e", for
    > `gnus-group-edit-group-method', put in the actual URL, then "C-c
    > C-c".

    > Obviously this is horribly baroque and it would be better if
    > `gnus-group-make-group' could prompt for the URL. No matter what,
    > it would be good to document the various ways to do this.

Weirdly enough, 'gnus-group-make-group' accepts an ADDRESS argument, and
the docstring even claims it prompts for it, but the function fails if
you pass it an address it seems.
However, since Gnus knows which backends require an address, and adds
it to the name of the method when present, it shouldn't be hard to fix
this command to actually prompt for an address when needed.
I haven't looked deeply enough into how that function works to know
whether it would actually work as expected in that case though.

    >> > Regarding your earlier question about having this backend
    >> handle > RSS too, I'm not aware of any significant difference
    >> between the > two beyond the format of the XML. Is that true?
    >> 
    >> Yes, even the XML format is very similar.
    >> 
    >> > If so, it seems like it would make most sense to merge the >
    >> code. Have you looked at nnrss? It would be good to know if there
    >> > was anything in there worth stealing for nnatom -- if one of
    >> them > has a faster parser than the other, for instance, or
    >> better logic > for keeping updates efficient.
    >> 
    >> The issue with merging the two is that nnrss saves feed data
    >> differently (both on disk and in memory), and also represents
    >> each feed as a group, with a virtual "server" holding all groups.
    >> I'm not sure if it's possible/a good idea to migrate feed data
    >> from nnrss to a hypothetical merged backend, at least not
    >> automatically.
    >> 
    >> Stefan floated the idea of adding RSS support, deprecating nnrss
    >> and creating an interactive migration command - so users who wish
    >> to migrate will have to do so manually (which should also
    >> potentially allow asking the user some questions if the migration
    >> includes some non-trivial steps).
    >> 
    >> nnrss does do some cool stuff that nnatom doesn't, though (for
    >> example, it tries very hard to find an RSS feed when you provide
    >> it with a link to a website, while nnatom currently requires a
    >> direct link to a feed).

    > Right, I should have been more explicit here -- what I was
    > thinking is what Stefan suggests: just deprecate nnrss
    > altogether. If it does anything cool like feed discovery, just
    > steal that code. If you're inclined to be nice enough to provide a
    > migration function, that's a bonus.

I guess I'll see how hard it would be.  First I have to actually
implement RSS support though.

    >> > I just subscribed to a feed with nnrss, and noticed that after
    >> I > marked all the items in the feed as read, I couldn't re-enter
    >> the > group and see the old items. It gave me "Can't select
    >> group". So > that's not very encouraging.
    >> 
    >> Honestly, from my experience nnrss has many small issues
    >> (although I never encountered this exact issue).  It is partly
    >> why I developed nnatom (previously I used a hack documented on
    >> the Emacs wiki which converted Atom feeds to RSS feeds on the
    >> fly).
    >> 
    >> > If you do want to expand this to be a general "feed" backend,
    >> we > might want to do some boring things like rename it
    >> nnfeed.el,
    >> 
    >> This is the name I thought of too, and I guess if two people
    >> thought of it independently it's probably fine.

    > Good!
  
    >> > and add support for ridiculous things like JSON feed[0]
    >> (why?!?). I > assume a derived backend could handle JSON feeds by
    >> setting the > appropriate values for the `nnatom-read-*-function'
    >> deffoos?
    >> 
    >> I actually attached a derivative "nnjsonfeed" backend I made as
    >> an experiment to one of my previous messages (it doesn't work
    >> with the current version of the patch, but it won't be hard to
    >> fix).
    >> 
    >> It wasn't 100% conforming to the standard (JSON feeds support
    >> some weird features like pagination, which can actually be
    >> supported very well in theory by nnatom, but I didn't feel like
    >> doing that), but it worked and it was very easy to make.

    > I missed nnjsonfeed, sorry. That's great you're already working in
    > this direction. I don't think we need to support absolutely
    > everything it does (what would pagination look like in Gnus?),
    > just the basics to get started with.

The feed itself is paginated, meaning that the entries are spread across
multiple files, with each one pointing to the next with a special tag.
The design of nnatom allows for this since it doesn't necessarily
require parsing the entire feed ahead of time, and also notifies the
backend when a stale (not new or updated) article is reached, so the
backend can intelligently stop (instead of trying to parse thousands of
pages every time you update a group, for example).  These capabilities
were actually added in anticipation of a JSON feed backend, the Atom
parser doesn't use them at all.

Gnus itself won't show pages of course, just a list of articles as
normal, but they will come from different JSON feed pages.

    >> > One of the awkward things about nnrss is that it's never really
    >> > fit well into Gnus' one-server-many-groups paradigm, which you
    >> > allude to in the nnatom Info section. Do you have any further >
    >> ideas in that direction?
    >> 
    >> Well, nnatom theoretically supports this paradigm, but it doesn't
    >> do this with Atom feeds, since it doesn't really make sense IMO.
    >> However, there is a standardized way to include links to Atom
    >> feeds in HTML documents, so it might be a good idea to support
    >> adding them as servers which show any linked Atom feeds as
    >> groups.

    > It's also perfectly possible that a single website would publish
    > multiple Atom feeds, right? Like weather.gov, for instance. That
    > would have an added benefit of letting us simplify the server name
    > from the full URL (https://alerts.weather.gov/cap/wa.php?x=0)
    > which looks ugly in the Group buffer, to just eg
    > alerts.weather.gov.

Yes, that's actually what I meant, although looking at real world
examples, it seems most websites either only link to one Atom feed in
the "correct" way, or don't link to any at all in that way.

<https://alerts.weather.gov/cap> doesn't link to any feed in the
standard way at all, for example.

    > But I guess I don't know why it would matter, really. The only
    > practical use for wanting multiple feeds under a single server
    > would be to set some configuration at the server level, that would
    > apply to all feeds from that server. And at the moment I don't
    > think there are many such settings (right?).

Actually, since the various parsing functions are stored in server
variables, all of them can be changed on a per-server basis, which is
pretty cool.

As an example, after I add support for fetching feeds contained in HTML,
I can imagine a scenario where a user adds a website which publishes
multiple, non-standard feeds, and fixes the parsing of all of them at
once by providing some customized parsing functions for that specific
server.
This is a contrived example, but I know at least of youtube, which
publishes slightly customized Atom feeds for each channel, so this is
already useful.

Daniel