Re: Workflow management with GNU Guix

[Ludovic Courtès]

Hi!

Roel Janssen <address@hidden> skribis:

> Ludovic Courtès writes:

[...]

>> IIUC, (guix workflows) from the tarball you sent executes workflows in
>> the current environment, as opposed to creating a derivation that would
>> actually perform the workflow.  What motivated this approach?
>
> The short answer:
> Lack of time to implement it properly ;).
>
> The slightly longer answer:
> I want to avoid storing results in the store, because we could be
> analyzing files of 100GB or more that we do not want to copy into the
> store, neither do we want to store the results of the run in the store.

Good point!

> I now realize we could only put the derivation in the store, and not the
> build output itself..

A derivation has to get its inputs from the store, and to write its
output to the store.  There’s no other option.

So I guess that’s an argument in favor of the approach you chose.

>> Workflows could compiled to derivations, which in turn could be “built”,
>> and their build result would be the workflow’s output file.
>>
>> I guess in practice it only works if users of the cluster can build
>> derivations on the cluster and have them scheduled on compute nodes.
>>
>> Thoughts?
>
> For building derivations, I think we need super user privileges, right?

Well guix-daemon needs to run as root, unless --disable-chroot is used.

> Why can't the scripts "just" output the environment variables required as
> @code{guix package --search-paths} provides, and then run the commands
> with the newly set environment?

Fundamentally, a derivation just describes a command, its arguments, its
dependencies, its outputs, and its environment variables.

So you’re right: you can very much run a derivation “by hand” instead of
letting the daemon do it on your behalf.  The only difference is that
you won’t have write access to the store.

Here’s an example:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix)
scheme@(guile-user)> ,use(gnu packages base)
scheme@(guile-user)> (define s (open-connection))
scheme@(guile-user)> (package-derivation s coreutils)
$4 = #<derivation 
/gnu/store/rmnb2x5vh9d9gdn1zb8q83hpyfnici18-coreutils-8.25.drv => 
/gnu/store/81pkzgzjwbnxfd5izgmgam8hfmjn20v8-coreutils-8.25-debug 
/gnu/store/apx87qb8g3f6x0gbx555qpnfm1wkdv4v-coreutils-8.25 5baea00>
scheme@(guile-user)> (derivation-builder $4)
$5 = "/gnu/store/ik15p8lrbk6jfa3fs3x34m78lj2c0ix1-guile-2.0.11/bin/guile"
scheme@(guile-user)> (derivation-builder-arguments $4)
$6 = ("--no-auto-compile" "-L" 
"/gnu/store/mn706n39l8z37w8wdqcm9v8pg6zcn33v-module-import" 
"/gnu/store/1a559p1yki9x1g676r8z0p3cf1f3pq7l-coreutils-8.25-guile-builder")
scheme@(guile-user)> (apply system* $5 $6)
[ Well, chdir to /tmp or something before trying it at home… ]
--8<---------------cut here---------------end--------------->8---

Ludo’.