Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Tue 01 Dec 2020 at 21:41, Eli Zaretskii <eliz@gnu.org> wrote:
> How would Emacs communicate with w3mimgdisplay? via some pipe?

Yes, pipe.

At the moment, in emacs-framebuffer, I just call it to display an image
using call-process-region.  It could be optimized with pipe, but this
has low priority for emacs-framebuffer as it is fast enough for my
needs.

If it was properly integrated with emacs at the level of image.el, then
pipe would be more optimized implementation, the way w3mimgdisplay is
actually meant to be used.

It is meant to be used this way: w3m browser spawns w3mimgdisplay and
talks to it via pipe.  w3m tells w3mimgdisplay to cache some images.
w3mimgdisplay tells w3m sizes of the images, w3m leaves space for the
images on the screen, draws text and then tells w3mimgdisplay to draw
the cached images in the right places on the screen.

> is that fast enough to be done in the middle of redisplay?

I am not exatly sure, what the concern is here.

The images do not travel via pipe.  Only meta-data like filenames,
coordinates and handles.

What exactly needs to be done in the middle of redisplay that is
relevant to drawing images?  Determining image width and height?
Something else like actually drawing the image?

The actual drawing could be postponed after Emacs has drawn its text.

>> I think that the only issue for me is that there is no way to
>> implement an alternative "system API" and "call some image library"
>> in pure elisp.
>
> Why is that an issue?

Because doing it in C requires significantly more effort and my free
time is limited.

> And what does an implementation in pure Lisp have to do with
> w3mimgdisplay?

Nothing.

Except the point above.

>> How much work would it be to enable alternative "system API" and
>> "call some image library" in pure elisp?
>
> If they support the same interfaces, very little.  But I'm not sure we
> are on the same page yet, so we might be miscommunicating.

What are the current interfaces?

>> >> It would be more convenient, if an image was represented as elisp
>> >> data instead of C data.
>> >
>> > create-image already knows how to accept image data.
>> 
>> But those must be defined in C, see lookup_image_type.
>
> You are supposed to submit data in one of the known formats.  Why is
> that a problem?

Because they do not work on the console.

And the reason they do not work on the console is because there is
hardcoded dependency on image drawing libraries which do not work on the
console.

I want Emacs to understand images even on the console.  But I have no
enough free time to do it in C.  If there was elisp level boundary
between image object and something that draws that image object, it
would be much easier.  I could just specify my own thing that draws that
image object without going so low level.  And it would work with any
existing elisp code like image.el transparently.

> why do you care about the exact format of the image data, and why do
> you insist to introduce a new type instead of using existing types?

I do not care about the exact format of the image data.

I do not care about the image data at all.

I care about extending Emacs' understanding of images in elisp at
meta-data level, not image data level.  For example, an image has kind
of type or format like png or jpeg, it has width and height, maybe some
other properties and some kind of opaque data blob.  I would like to be
able to specify and/or override how to draw images in elisp and not
having that hardcoded in C.

Additionally there is no reason why documents like pdf, epub, odt etc
could not be an image (which they basically are).  They have pages but
tiff also has more than one image.  There is also no reason why image
modes could not have similar capabilities to doc-view or pdf-tools.  All
this could be eventually unified.

>> Or is there a way to define custom image format in pure elisp?
>
> No.

That is a shame.

>> It would be actually enough for me if my custom image was opaque to
>> emacs (except probably some attributes like width and height) and
>> w3mimgdisplay handled the rest.
>
> That's okay, but I still don't see how all that will fit into the
> current framework of displaying images.

Maybe I do not need custom image type.

But I need to change the way images are drawn on the console, i.e. not
at all vs draw the image.

But the way image types are implemented currently does not allow me to
do that in elisp.

>> >> iirc there is no way to add new image types without touching C.
>> >
>> > No, because you need to know how to convert that into the internal
>> > format used to draw the image, which is platform dependent.
>> 
>> I don't need to know that.  w3mimgdisplay program does that.
>
> How can w3mimgdisplay know the internal format used by Emacs for
> storing images?

Why would w3mimgdisplay need to know that?

w3mimgdisplay just needs filename and coordinates.

>> As far as emacs is concerned, it needs to give to w3mimgdisplay
>> filename to draw and then use a handle to cache to ask w3mimgdisplay
>> for redraw.
>
> I suggest to take a look at x_draw_image_foreground and its
> subroutines, this is where we actually put the image on the glass.
> Then tell in a bit more detail how to integrate an external program
> into that, keeping all the features we support in this regard, like
> masks, clipping, slices, etc.

Thanks, I'll have a look.

>> > Why is it a problem to use one of the known image types?  E.g., SVG
>> > (via the svg.el library).
>> 
>> Because those those are not universally available, e.g. on the
>> console.
>
> On the console you currently cannot have images in Emacs, period.

Exactly.  And I want this to change.  There is no reason for such
restriction.

> This is not because of image libraries, this is because the text-mode
> display simply cannot currently support such display elements.

This is not an issue at all.  Capabilities of text-mode display are
irrelevant.  w3mimgdisplay is using framebuffer.  I do not know how to
better communicate that.

Try:

   $ w3m https://gnu.org

You will see text-mode HTML browser layout with images on the console.

>> >> Is there a way to turn off cursor for a buffer?
>> >
>> > There's the cursor-type variable.
>> 
>> Thanks.
>> 
>> This does not work on the console.
>
> (You never said you were talking about a console.)  We currently don't
> expose console cursor control to Lisp.  Should be easy to add that, if
> needed.

I see.

cursor-type does not say it does not work on the console.

I think not all values are doable and relevant.  t and nil should be
enough.

Shell I open a bug report for this?

>> Is there a way to hook a function when a buffer looses and gains
>> focus?
>
> Buffer cannot have focus.  Do you mean when the buffer's window is the
> selected window?  There's window-selection-change-functions.

Thanks, I'll look into this.

>> >> Is there a way run a function after emacs changes something on the
>> >> screen in a buffer?
>> >
>> > That depends by what you mean by "after emacs changes something on
>> > the screen", since in Emacs redisplay is independent/separate of
>> > data changes.  Please elaborate.
>> 
>> Example: I am on the console.  I am in dired buffer.  I press my key.
>> w3mimgdisplay program draws my image on the screen on the specified
>> coordinates "over" emacs.  All is well (except the blinking cursor
>> which we can ignore for now).  Now I press another key to display the
>> next image in the dired buffer.  Emacs redraws some characters, that
>> have changed.  This disrupts the image drawn earlier and I need to
>> detect, that I need to tell w3mimgdisplay to redraw the image.  I
>> need to hook my redraw image function after emacs is done with its
>> drawing.
>
> I don't think we have such a hook, no.  But if we had it, using it
> like you want would cause annoying flickering, because you'd
> constantly redraw the image at very high frequency.

I do not think it would cause flickering on the console.  The only thing
that normally "flickers" is the blinking cursor.  Emacs does not draw
anything if nothing changed.

>> Is there such hook?  If not, would it be difficult to add it?
>
> It won't be hard to add it, but I'm not sure we want to add such a
> hook, see above.

That is a shame.  If there was a hook to detect when Emacs has drawn
something on the console, I could plug image redrawing there.

If I wanted to add it, where in the code should I look?

Thanks for your help!