Re: Viewing derivation output in the store

[Brian Cully]

Josselin Poiret <dev@jpoiret.xyz> writes:

> Just for completeness' sake, here's my take on it from yesterday while
> we were talking about it with Brian:

        Thank you for that, it was very enlightening.

> (define-module (test)
>   #:use-module (guix gexp)
>   #:use-module (guix monads)
>   #:use-module (guix derivations)
>   #:use-module (guix store))
>
> (define test-gexp
>   #~(begin
>       (copy-file #$(plain-file "helloes.txt" "contents") #$output)
>       (format #t "Helloes~%")))
>
> (with-store store
>   (run-with-store store
>     (mlet* %store-monad
>         ((drv (gexp->derivation "myderivation" test-gexp))
>          (output -> (derivation->output-path drv)))
>       (mbegin %store-monad
>         (built-derivations (list drv))
>         (return (format #t "~a~%" output))))))

        I tried playing with this for a while and ran into an issue when
actually executing it in a REPL: if the derivation hasn’t yet been
built, then during execution of of the ‘built-derivations’, an error is
thrown:
---[snip]---
ice-9/boot-9.scm:1685:16: In procedure raise-exception:
In procedure display: Wrong type argument in position 2: #<closed: string 
7fb2850f3bd0>

Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.
scheme@(zgrub) [1]> ,bt
In ice-9/boot-9.scm:
  1752:10  9 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
In /home/bjc/guix/guix/store.scm:
   658:37  8 (thunk)
  2129:24  7 (run-with-store #<store-connection 256.99 7fb2707783c0> _ 
#:guile-for-build _ #:system _ #:target _)
In current input:
   6373:8  6 (_ _)
In /home/bjc/guix/guix/store.scm:
  2001:38  5 (_ #<store-connection 256.99 7fb2748609b0>)
  1421:15  4 (_ #<store-connection 256.99 7fb2748609b0> _ _)
   759:13  3 (process-stderr _ _)
In unknown file:
           2 (display "building path(s) 
`/gnu/store/5jdlbc8vf5d0hf4inzj9k7gq6r9k6xsy-grub.cfg'\n" #<closed: string 
7fb2850f3bd0>)
In ice-9/boot-9.scm:
  1685:16  1 (raise-exception _ #:continuable? _)
  1685:16  0 (raise-exception _ #:continuable? _)
---[snip]---

        I’m way too new to all this to say for sure, but it seems to me
that the build log output is being sent to a string, which, at least in
the REPL, is closed prematurely (if it’s ever even opened).

        If I wait a bit for the derivation to build, executing the above
code works fine and produces output in the output path.

        Is this something that’s worthy of a bug report, or am I doing
things I’m not supposed to be doing by building derivations in a REPL?

> I tried to use the most monadic approach here to demonstrate its
> usefulness, however, it seems to me that derivation->output-path is not
> documented, along with built-derivations (which is just (store-lift
> build-derivations)).

        Yes, thankfully the code was either easy to infer from the name
(derivation->output-path) or easy to read (built-derivations).

> Lesson learned: don't shadow gexp (hence `test-gexp`)!
>
> As an aside: is there anything preventing us from having do notation à
> la Haskell?  This could help bridge mlet and mbegin with >>=, which in
> its current form is impractical.  Here's what it could look like:

        Because of the above error, and general cluelessness, I tried
removing the ‘mbegin’ and found that the error went away, but the
derivation wasn’t built. My (wrong) assumption was that the ‘mbegin’ was
redundant with the ‘mlet’ since I figured the body of the mlet was in
the monad context. The documentation also describes mbegin as akin to
mlet, although at least the argument list implies a difference with
careful reading: mlet has ‘body’ while mbegin uses ‘mexp’.

> (mdo %store-monad
>   (drv <- (gexp-derivation "myderivation" test-gexp))
>   (output <- (return (derivation->output-path drv)))
>   (built-derivations (list drv))
>   (return (format #t "~a~%" output)))
>
> We could even have some more sugar for (x <- (return y)), Haskell has
> `let x = y`, but we could instead have something like `(x := y)`?

        This would definitely be more intelligible to me, at least, and
I know precious little Haskell.

        One final note that’s worth covering: while I was digging
through code I noticed that file-like objects have an embedded gexp
derivation. In the documentation they are described as the declarative
counterpart of ‘gexp->derivation’, but since there is no documented way
of turning them directly into a derivation, I was at a loss for what to
do with them until someone in IRC suggested using the #~(copy-file …)
gexp above.

        It turns out that there are ‘gexp-compiler’ instructions for
these things, which will turn them back into derivations. However, there
doesn’t seem to be a way to call them directly, and the only way I could
find to invoke them was by using ‘lower-object’ (again found by
grep-find through the codebase). This makes an intuitive sense, but the
naming implies I’m missing some larger architectural picture here where
‘raising’ and ‘lowering’ are more precisely defined.

        So the copy-file gexp can be substituted by ‘(lower-object
test-gexp)’:
---[snip]---
(mlet* %store-monad ((drv (lower-object (plain-file …)))) …)
---[snip]---

        I don’t know how useful these observations are, but hopefully at
least a little. I know I’m not alone in trying to understand how to make
the leap from file-likes to actual files; it seems to come up on IRC
with some regularity. At a minimum, this email will get indexed by
search engines and hopefully answer some future questions from there.

-bjc