Re: [PATCH] Add current-suspendable-io-status parameter

[Mark H Weaver]

Hi Nala,

Nala Ginrut <address@hidden> writes:

> On Tue, May 14, 2019 at 7:01 AM Mark H Weaver <address@hidden> wrote:
>> I guess what you want is the ability to see incremental reports on the
>> progress of your large I/O operations.  Is that right?  If we are going
>> to add an API for this, it needs to be reliable, and always give reports
>> in terms of the high-level requests that the user gave.
>
> Yes, that's exactly what I want. We need to get the progress of I/O
> operation when it's blocking
> so that we can compute a fair priority for the tasks.
>
>> My preferred approach would be something like this: we could add a
>> 'put-bytevector-some' I/O primitive which writes some bytes from a
>> bytevector, blocking only as needed to write at least one byte.  It
>> would return the number of bytes written.  This can be used to implement
>> an efficient variant of 'put-bytevector' that gives you access to the
>> real-time progress information.
>
> I'm not sure if put-bytevector-some does the work, I'll list my concerns:
>
> 1. All I/O will be managed by Guile when we enabled suspendable-port.
> That is to say, from the users side, users never know their I/O
> operations are blocking or not. It's transparent to users.
> Guile will guarantee the I/O operations to be finished by managing all
> the blocking I/O mechanisms.
> Users can only interact with the task with read or write waiter, which
> are registered by users themselves.
> In this scenario, users are out of control of I/O operations. And they
> have no way to get the progress of I/O, since there's
> no way to pass this status to the waiter function except for
> parameters in my patch.
>
> 2. suspendable-port module has already provided a bunch of overridden
> bytevector-* functions.
> However, they're hidden from users. I think it's good since the
> purpose of suspendable-port is to abstract all these details from
> users. Users only consider the read-waiter and write-waiter for scheduling.
> If we provide the low-level bytevector functions to users to let them
> do the non-blocking I/O by themselves, just like most C framework
> does. Then Guile suspendable-port will lose a critical feature,
> although users can still implement asynchronous non-blocking I/O by
> themselves with a non-managed approach. Say, do the I/O, check result
> by themselves, and do the scheduling.
> Personally, I'm fine with this way, since I'm familiar with both ways.
> But managed I/O of suspendable-port is a good selling point for many
> inexperienced server-side developers, they can use it in Scheme just
> like IOCP or AIO.
>
> Of course, I may misunderstand your mind.
> Could you elaborate more about your approach?

Here's what I had in mind.  I'm not suggesting that you use asynchronous
non-blocking I/O.  'put-bytevector-some' would be like 'put-bytevector'
except that it can write less than you asked it to.  Ideally it would
block only as needed to write at least one byte.  It would be analogous
to POSIX write(2).

Here's an untested draft implementation of 'put-bytevector*' which sets
the parameter 'current-io-status' to provide the information you wanted,
except that here the information will reliably refer to your own buffer.
The primitive it is built on 'put-bytevector-some' does not yet exist,
but we could add it.  Note that 'put-bytevector-some' would still block,
and it would be suspendable.

--8<---------------cut here---------------start------------->8---
(define* (put-bytevector* port bv
                          #:optional
                          (start 0)
                          (count (bytevector-length bv)))
  (let loop ((start start) (count count))
    (unless (zero? count)
      (let ((written (parameterize ((current-io-status (cons start count)))
                       (put-bytevector-some port bv start count))))
        (loop (+ start written) (- count written))))))
--8<---------------cut here---------------end--------------->8---

We can already do something analogous on the reader side, using either
'get-bytevector-some' or the already-implemented-but-not-yet-pushed
'get-bytevector-some!'.

What do you think?

      Mark