--- Begin Message ---
|
Subject: |
[PATCH 0/2] Multiplexed build output from the daemon |
|
Date: |
Mon, 8 Oct 2018 00:29:45 +0200 |
Hello Guix!
This patch set is partly a response to the problem Ricardo raised at:
https://lists.gnu.org/archive/html/guix-devel/2018-09/msg00322.html
and partly a natural followup to the addition of (guix status):
https://issues.guix.info/issue/32837
So far the daemon would send all its output (messages it writes as well as
messages any of its build processes writes) directly as a single stream
to its clients. This had several shortcomings:
1. Clients could not distinguish messages coming from the daemon (such
as build traces or ‘guix substitute’ messages) from messages coming
from build processes. (Notably build processes could “forge” build
traces such as “@ build-started”.)
2. When max-jobs > 1, clients would receive intermingled output from all
the build processes without any way to disentangle it.
3. Build traces written by the daemon were expected to start on a new
line but a builder could write output not terminated by a newline
(e.g., “checking for …” messages from ./configure) and consequently
build traces would not start on a new line and would go unnoticed
by (guix status) and co.
With this change clients can optionally ask for “multiplexed build
output”. When it’s enabled, build output is prefixed by a special
trace, like this:
@ build-output 1234 21
checking for fcntl...
where 1234 is the PID of the build process speaking and 21 is the number
of bytes in the following build output fragment. The PID is first given
in the corresponding “@ build-started” trace.
On the client side, (guix status) is adjusted to produce events like:
(build-log #f MESSAGE)
for a message coming from the daemon, and:
(build-log PID MESSAGE)
for a message coming from PID.
The downside of the protocol is that it creates quite some overhead. For
example, when extracting a tarball, we see things like:
read(13, "gmlo\0\0\0\0", 8) = 8
read(13, "5\0\0\0\0\0\0\0", 8) = 8
read(13, "@ build-output 25935 29\ncoreutils-8.29/m4/fseterr.m4\n", 53) = 53
read(13, "\0\0\0", 3) = 3
That is, a 29-byte message with a 24-byte header (plus the
8 + 8 + 3 = 19 bytes of the underlying protocol; see ‘process-stderr’.)
Another option would be to incorporate multiplexing in the lower-level
binary protocol. However, the client side would need potentially bigger
changes: the single ‘current-build-output-port’ sink would no longer
be a good match.
Thoughts?
Ludo’.
Ludovic Courtès (2):
daemon: Support multiplexed build output.
status: Build upon multiplexed build output.
guix/scripts/build.scm | 3 +
guix/scripts/environment.scm | 1 +
guix/scripts/pack.scm | 1 +
guix/scripts/package.scm | 3 +-
guix/scripts/pull.scm | 1 +
guix/scripts/system.scm | 1 +
guix/status.scm | 165 ++++++++++++++++++++++++--------
guix/store.scm | 15 ++-
nix/libstore/build.cc | 25 +++--
nix/libstore/globals.cc | 2 +
nix/libstore/globals.hh | 9 +-
nix/libstore/worker-protocol.hh | 2 +-
nix/nix-daemon/nix-daemon.cc | 2 +-
tests/status.scm | 47 ++++++++-
tests/store.scm | 63 ++++++++++++
15 files changed, 289 insertions(+), 51 deletions(-)
--
2.19.0
--- End Message ---
--- Begin Message ---
|
Subject: |
Re: [bug#32980] [PATCH 0/2] Multiplexed build output from the daemon |
|
Date: |
Mon, 15 Oct 2018 23:27:55 +0200 |
|
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) |
Hello,
Ludovic Courtès <address@hidden> skribis:
> The downside of the protocol is that it creates quite some overhead. For
> example, when extracting a tarball, we see things like:
>
> read(13, "gmlo\0\0\0\0", 8) = 8
> read(13, "5\0\0\0\0\0\0\0", 8) = 8
> read(13, "@ build-output 25935 29\ncoreutils-8.29/m4/fseterr.m4\n", 53) = 53
> read(13, "\0\0\0", 3) = 3
>
> That is, a 29-byte message with a 24-byte header (plus the
> 8 + 8 + 3 = 19 bytes of the underlying protocol; see ‘process-stderr’.)
I timed “guix build -S binutils --check”, which includes the output of
“tar xvf” followed by “tar cvf”, and the timing difference is not
noticeable (of course the peak of CPU activity is caused by xz.)
Anyway I went ahead and pushed this as
f9a8fce10f2d99efec7cb1dd0f6c5f0df9d1b2df.
I changed “@ build-output” to “@ build-log”, and also adjusted
tests/status.scm since new tests had been added to that file in the
meantime.
Ludo’.
--- End Message ---