Re: Rethinking the design of xwidgets

[Akira Kyle]

On Tue, Oct 13, 2020 at 08:16 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

> Can you tell more about the dynamic content you had in mind, and 
> why
> you thought to use xwidgets as the basis for that?  There's IMO 
> a leap
> of logic here that I cannot follow, perhaps because I don't have 
> the
> necessary background information.

Sure, so to elaborate on my motivation: I spend a lot of my time 
in emacs using org-mode's babel support for python, specifically 
through emacs-jupyter [1]. This gives me much of the power of 
Jupyter notebooks, however not all. For example when generating 
plots using matplotlib, they are simply embedded in the buffer as 
an image, however matplotlib also has support for interactive 
backends [2] which allow one to zoom and pan the plot and see it 
update in real time as each plot command is issued. It would be 
convenient to have such interactivity directly inside the buffer 
rather than having to use an external window, similar to the 
interactivity offered when using matplotlib in a jupyter 
notebook. The bokeh plotting [3] library provides even more 
interactivity, allowing one to create complex and dynamic and 
interactive visualizations. When used inside a jupyter notebook, 
the plot is embedded inline, however when used with emacs-jupyter, 
it launches an external browser to display the interactive plot 
and it's controls. It would be fantastic if I could have such 
interactivity inside Emacs.

Another use case I had in mind for my proposed improved xwidgets 
is better pdf rendering. I use pdf-tools constantly and it's one 
of the best pdf viewers out there (not just because it's in 
emacs). I'll sometimes even use it to give presentations so I 
never have to leave Emacs. However it's design inherently limits 
its rendering performance as it must rasterize each pdf page in a 
separate process then pipe that data to emacs to display as an 
image. Granted this could be avoided by turning the epdfinfo 
server process into a dynamic module, however it will still be a 
bit roundabout as the pdf page would have to first be rendered 
into a cairo context then rasterized then Emacs will draw that 
rasterized image onto another cairo context to get it 
onscreen. Having direct access to an onscreen cairo context would 
eliminate this and make it easier to support crisp, performant 
rendering onto HiDPI displays. Maybe I could even throw slide 
animations into my presentations this way!

> From my POV, xwidgets have several problems which don't allow me 
> to
> easily see them as the basis for some much more general set of
> features or infrastructure:
>
>   . they depend on GTK and webkit, whose development leaves a 
>   lot to
>     be desired, and some say parts of that have no future; they 
>     are
>     definitely less portable than I'd like to see in Emacs

My proposal should actually make it easier for xwidgets to be 
decoupled from a specific GUI toolkit. The recent addition for a 
cocoa webkit xwidget has forced me to think a bit about this as I 
modify the xwidget code, and I'm fairly convinced that xwidgets 
can be designed is such as way that any object-based and 
event-driven toolkit could fit in its abstraction.

>   . their integration in the Emacs display code "needs work", 
>   there's
>     at least one or two places where the code which handles them 
>     is
>     clearly wrong -- this is semi-okay for a minor niche 
>     feature, but
>     not for something on which we want to build our future

Could you point me to some of those places. Maybe they are issues 
I've already taken a look at.

> So before we decide that xwidgets is the way to go, I'd like to 
> make a
> step back and see what other alternatives we can think of, if 
> any, and
> carefully consider which alternative is best in the long run.

I'd be interested if you can think of any alternatives since, at 
least to me, this seems like the natural way to implement such 
features.

> Here, too, I'd like first to consider a design that doesn't 
> require
> dynamic modules as a prerequisite for the feature.  If we can 
> come up
> with something that requires only Lisp (like image display does, 
> for
> example), then I think we all win, because it is much easier to 
> write
> a Lisp package than a package that requires a module in C or 
> similar
> language.

I think my pdf use case illustrates the issue with Emacs trying to 
support every such type of content as a high level "image-like" 
feature since you either force the application to convert 
everything into a sub-optimal format that Emacs understands for 
display or Emacs' must add native for such format, increasing the 
number of external dependencies to build Emacs.

For example, consider svg support, which is my preferred image 
format. As far is I can tell, currently Emacs rasterizes svgs 
itself before ultimately drawing them on a cairo context. Since 
librsvg supports drawing directly on a cairo context, it would be 
much more efficient to do so, however adding support for svg could 
be easily moved out of emacs into a dynamic module and drawn 
directly onto it's own widget. This would allow users of the NS 
toolkit to not have to rely on librsvg if they didn't want to as I 
believe NS can render svgs itself. 

> Bottom line, I'd like to understand the problem you are trying 
> to
> solve better before we conclude that xwidgets and dynamic 
> modules are
> the way to go.

Let me know if there's anything more I can clarify about my 
motivation and proposal to improve xwidgets.

[1] https://github.com/nnicandro/emacs-jupyter
[2] https://matplotlib.org/faq/usage_faq.html#what-is-a-backend
[3] https://bokeh.org/