Re: Workflow management with GNU Guix

[Roel Janssen]

Ludovic Courtès writes:

> Hi!
>
> Roel Janssen <address@hidden> skribis:
>
>> Ludovic Courtès writes:
>
> [...]
>
>>> IIUC, (guix workflows) from the tarball you sent executes workflows in
>>> the current environment, as opposed to creating a derivation that would
>>> actually perform the workflow.  What motivated this approach?
>>
>> The short answer:
>> Lack of time to implement it properly ;).
>>
>> The slightly longer answer:
>> I want to avoid storing results in the store, because we could be
>> analyzing files of 100GB or more that we do not want to copy into the
>> store, neither do we want to store the results of the run in the store.
>
> Good point!
>
>> I now realize we could only put the derivation in the store, and not the
>> build output itself..
>
> A derivation has to get its inputs from the store, and to write its
> output to the store.  There’s no other option.
>
> So I guess that’s an argument in favor of the approach you chose.

Can't a derivation write its output to some other place than the store?
Maybe by running it "by hand"?

>>> Workflows could compiled to derivations, which in turn could be “built”,
>>> and their build result would be the workflow’s output file.
>>>
>>> I guess in practice it only works if users of the cluster can build
>>> derivations on the cluster and have them scheduled on compute nodes.
>>>
>>> Thoughts?
>>
>> For building derivations, I think we need super user privileges, right?
>
> Well guix-daemon needs to run as root, unless --disable-chroot is used.

Yeah ok..  But as long as the guix-daemon doesn't build any derivation it
doesn't need super user privileges ;).

>> Why can't the scripts "just" output the environment variables required as
>> @code{guix package --search-paths} provides, and then run the commands
>> with the newly set environment?
>
> Fundamentally, a derivation just describes a command, its arguments, its
> dependencies, its outputs, and its environment variables.
>
> So you’re right: you can very much run a derivation “by hand” instead of
> letting the daemon do it on your behalf.  The only difference is that
> you won’t have write access to the store.

And that's fine, because we don't want to write the output to the store :).

So, the workflow language should create a derivation, but then
guix-daemon should not execute the derivation, but instead, the workflow
execution engine can do it so it can avoid writing the output to the
store.. right?

Kind regards,
Roel Janssen